To deliver reliable, fast, and responsive services, providers demand real-time automation informed by actionable insights gleaned from the torrents of data and device telemetry events streaming through their networks.
For ages, it seemed that one could have the speed or data granularity in network analytics, but not both, or that one could enjoy both speed and granularity, but not the long-term data retention necessary for divining trends, planning infrastructure, and so on. But now, that’s changed.
Here, I will discuss how the Volt Active Data Platform works in conjunction with Vertica, currently powering post-event analytics in the leading Network Data Analytics Function (NWDAF) and network intelligence solutions worldwide, to provide comprehensive insights and immediate actionability.
Table Of Contents
Why databases struggle with network intelligence
Deep packet inspection, network intelligence, intrusion detection, and intrusion prevention generate large volumes of granular data about the data packets flowing through an operator’s network with user and/or equipment identification; IP addresses and/or hostnames; protocols; service names and categories; and vital stats such as internal and external latencies and the number of packets and bytes that were transmitted, dropped, or delayed. Systems can emit these packet flow statistics many times per second, which could translate into several hundred million records and gigabytes per second.
Unfortunately, most databases simply aren’t capable of drinking from such a massive firehose and still perform well. Of the handful that can handle the volume, just about all of them sacrifice immediate consistency and/or query execution times.
Network intelligence vendors have tried scaling their underlying databases to dozens of nodes—a prohibitively expensive solution—to handle the volume, but they have had to dial back the update intervals to five or even fifteen minutes and sacrifice 99% of the most valuable data so that queries don’t require hours or even days to return answers.
Can network intelligence and network automation succeed without granular data and responsiveness?
Regrettably, most network automation vendors have discarded the very granular detail that makes their solutions so useful: ie, factors that would have presaged traffic surges and informed appropriate adjustments to avert outages, lag, buffering, and everything else that infuriate millions of subscribers; data points that would have heralded ingenious network penetrations and halted thefts of hundreds of millions of financial account details; and essential details that could have helped operators mitigate billions in losses due to piracy, interconnect bypass (SIM box) fraud, and other schemes. They’ve thrown away essential detail by lengthening the interval between updates.
Very short-duration events that come and go in-between update intervals just don’t show up at all. For example, when tens of thousands of subscribers in a certain area experience recurring service degradation or outages of durations less than the update interval but long enough to seem like a veritable eternity to many of them, they’re going to tie up customer care with calls and messages or leave for another provider. Automation that could have easily averted this by making policy adjustments, rerouting traffic, or dispatching a maintenance crew just wasn’t triggered.
Trading granularity for better query responsiveness hasn’t paid off, either, because they’re still taking too long to execute––often several minutes, but sometimes hours. Queries may complete in mere seconds at the smallest MVNOs and ISPs with only a few thousand subscribers, but most operators discovered that many routine queries, especially those involving aggregate functions, were yielding insights too late for them to be of any practical use in their automation schemes. For sure, they could use some of the information returned for post-event actions like network capacity planning, but they couldn’t inform automations that would have an immediate effect on the quality of experience and profitability.
Today, many network operators are still unable to seize prime opportunities that can pay off only by processing data and acting on insights within single-digit milliseconds. For instance, many MNOs/MVNOs have been missing out on doubling revenues from prepaid mobile top-ups and keeping heavy users from switching to other providers.
However, operators deploying one vendor’s Volt Active Data-powered solution have been making personalized airtime advance and data bundle offers by text at just the right time to encourage top-ups and loyalty. Before they run out of airtime or data—or even after they’ve run out of airtime but in the precious milliseconds before customers put their phones in their pockets or swap out their SIM cards—the subscribers receive a customized airtime advance or data bundle offer that a machine learning algorithm predicted they’re very likely to accept and, of course, pay off within a reasonable time. As a result, some operators have seen offer uptake surge by 150% or more, such that airtime advances now account for about half of total prepaid recharge revenue.
Delivering both speed and deep insights
One thing that network intelligence solutions are always doing is aggregating statistics by various categories such as subscribers, subscriber category, service/application, service category, protocol, and subscription plan. They’re even aggregating by timestamps and time periods. These systems constantly transform the raw statistics into key performance indicators and interpret them in the contexts of service/application needs, subscription plans, and so on.
Applications depend on these contextualized KPIs, these measurements of quality of service and quality of experience, to trigger appropriate actions. These aggregate operations are where every other database stumbles performance-wise and/or validity-wise.
But we built Volt Active Data specifically for fast aggregate operations with instantaneous materialized views. Volt Active Data’s materialized views provide instant visibility into aggregate data, are always up to date with each insert and update, are transactionally consistent, take mere microseconds rather than minutes or even hours to refresh, and are readily accessible via both standard SQL and stored procedures, usually in less than a millisecond.
Providers expect such instantaneous answers to inform automations that keep their network running optimally, securely, and better-equipped to engage and serve their customers.
Delivering actionable intelligence and long-term data retention
Of course, one can’t keep all of that data around in Volt Active Data forever. Someday, we’ll all have access to systems with multiple petabytes of cheap RAM that would allow anyone to keep that level of data online indefinitely. But that’s still a long way off. And of course, Volt Active Data wasn’t built for big data. Rather, it was optimized for fast data, retaining only the contextual data it needs to process millions of simultaneous streaming events that require actions in a handful of milliseconds.
Still, many regulatory authorities demand providers retain certain records somewhere for months, or even years. That’s why we furnished Volt Active Data with the ability to automatically migrate old data that is no longer needed, aggregating it if necessary before exporting to other databases such as Vertica or a data lake. The vast majority of regulators require retaining only summarized data. Regardless of the granularity or retention periods, using Volt Active Data and Vertica together provides exactly what is needed to deliver actionable intelligence in real time while serving up everything needed for post-event processes.
Conclusion
As a scalable, in-memory, ACID-compliant data platform supporting real-time aggregation and real-time SQL analytics against high-volume, high-velocity data streams, Volt Active Data perfectly complements the Vertica database embedded in the leading network intelligence solutions. Volt Active Data provides the consistent, real-time answers needed for effective network automation, freeing Vertica to concentrate on providing long-term storage, efficient post-event analytics, and the machine learning algorithms that run on longer time scales.
