THIS BLOG IS PART 2 OF A 3-PART SERIES. PART 1 DELVED INTO BENCHMARK DATA. PART 3 WILL DISCUSS SPECIFIC VOLT + REDPANDA USE CASES.
Needing to act quickly on data is not a new imperative. What is new is the need to act super quickly on massive amounts of highly varied and volatile data coming in through data streaming platforms.
Think of data as a residential building with hundreds of new additions adding tremendous stress to that building’s infrastructure, including its electricity, water, and plumbing systems. You need to update the systems to allow them to handle this new stress, or they will break; thus, so will your development.
Recent benchmark data showed that Redpanda and Volt provide a powerful combination to help companies more easily manage their streaming data via real-time decisioning. Redpanda and Volt pair very well together to enable organizations to take real-time actions on streaming data, at a much lower total cost of ownership vs other solutions.
This is the second of a three-part series on why Volt + Redpanda is such an impactful combination that allows companies with latency-dependent applications to more fully capitalize on their streaming data.
Table Of Contents
The Challenges of “Real Time”
The desire and need to use data quickly (ie, in “real time”) is driving an evolution from slow, batch-oriented processing to streaming, Event-Driven Architectures (EDA).
Fully capitalizing on real-time data means identifying and acting on the moment of “significance”—i.e., the moment of customer engagement or threat prevention—before the chance to monetize it or prevent it from doing damage passes.
To accomplish this, your data platform needs to be able to perform the entire ingest-decide-act sequence with low latency. If not, the chance to act on the data is lost, and so is the revenue (or maybe even the customer).
Acting on complex data in this moment of value is very hard to do with a composite, multi-layer topology in which the data being enriched, transformed, joined, aggregated, etc., moves through many different parts. Using a composite architecture with many technologies for each stage of the decision-making process brings many challenges, including:
- Inability to meet latency SLAs
- Missed opportunities for customer engagement and fraud prevention
- Infrastructure sprawl due to resiliency management of each layer
- Having to move large amounts of data to and from the storage and processing layers
- Eventual consistency, which leads to data loss
- Multiple failure domains making debugging errors or latency vectors extremely difficult
The combination of Volt and Redpanda empowers companies to take action on real-time data by empowering them to efficiently handle both streaming and processing.
Here’s how each part of the equation works.
How Redpanda Supports Actions on Real-Time Data
Streaming data from the source to a fast storage and pub/sub layer in the stack requires efficient, low-latency data movement at scale. In recent years, Apache Kafka (Kafka) has established itself as the open-source technology of choice to manage this streaming data.
Kafka is a distributed log management system written in Java to connect data publishers with data subscribers through a topic system. Data creators will either create a new topic or use an existing topic to publish the data. Consumers subscribe to these topics either as an individual client or as part of a client group for rapid consumption of messages. Kafka has a rich connector ecosystem to connect to preexisting systems ranging from databases to prepackaged applications such as ERP or CRM.
Kafka is an excellent tool for capitalizing on streaming data through messaging; However, as companies scale, their use of Kafka tends to run into serious challenges, including:
- Java garbage collection unceremoniously slowing things down.
- The complexity of managing large clusters.
- The complexity of Zookeeper (or kRaft running in a separate cluster) and the need for a separate schema registry.
- High latency for delivering messages.
Redpanda built its kafka-oriented streaming data platform in c++ with embedded Raft to address the aforementioned challenges, creating a streaming data platform for developers that is API-compatible with Kafka but 10x faster, 6X more cost-efficient, much easier to use, and safer.
Also, by frequently syncing data to the disk with every batch/message, Redpanda is able to achieve greater data safety and low latency for message delivery.
How Volt Supports Actions on Real-Time Data
The streaming is one part. The processing—or what we call “decision-making”—is the other.
Real-time applications require real-time decision-making to work well. This decision-making part requires applying codified rules to event data in a way that takes both recent and historical events into account in a quick, intelligent manner.
This is where Volt excels.
Volt Active Data is a distributed-by-core in-memory data platform that addresses the entire spectrum of activities needed for making real-time decisions, including:
- Ingestion via Volt’s client drivers or integration with streaming systems via our built-in or custom-made importers.
- Storage via a storage engine and SQL execution engine written in C++ to avoid Java Garbage Collection.
- Aggregation using Volt’s materialized views (real-time transactional SQL that can join multiple tables that can be streamed out as a continuous view to streaming systems.
- Real-time decisioning (that can be further enhanced with ML) based on KPI deviations via Volt Stored Procedures written in Java and accommodating third-party libraries.
- Alerts around activity in downstream or streaming systems using our exporter framework.
Some of the results Volt customers have achieved from using Volt to move from near real-time to event-driven real-time processing are:
- 253% increase in offer acceptance
- 83% reduction in fraudulent credit card transaction completion
- 100% prevention of DDoS attacks
- 100% detection of Ad bots
- 90% reduction in Infrastructure cost
Not too shabby, and I think any company would consider any of the above a major success and a major return on investment for their data platform.
How Volt and Redpanda Combine to Form a Whole Greater than the Sum of the Parts
Leading enterprises use the combination of streaming data and contextual real-time decision intelligence to get ahead of competitors by improving their customers’ real-time engagement models. These customers have recognized that to effectively capitalize on significant customer moments, they need to use their data much more efficiently and contextually than just simple aggregation, filtering, and enrichment.
The partnership between Volt Active Data and Redpanda enables low-latency data movement and decision-making so enterprises can monetize important events as they’re happening to intelligently engage customers and recognize anomalies before they wreak havoc.
Redpanda addresses the low-latency data movement needs while Volt helps customers make real-time decisions and actions.
Ultimately, the Volt + Redpanda partnership takes event-driven architectures beyond just systems simplification to the larger business benefits of increased revenue and reduced revenue loss.
What about just using ksqlDB or a NoSQL database?
NoSQL databases and ksqlDB typically don’t work well for customers needing to take real-time actions on streaming data.
Ksql works for simple filtering, aggregation, and stream enrichment, especially for use cases that are just about preparing the data for consumption by latency-insensitive applications and batch analytics systems. However, when the application requires contextual decisions to take real-time actions in milliseconds, ksqlDB is only a small part of the equation for data aggregation and enrichment. Even for that, it needs to be set up DAG (Distributed Acyclic Graph)-style since it can only execute one SQL statement at a time before it needs to emit the output to a topic. This will cause many hops from Kafka to ksqlDB to Kafka and so on.
Simple NoSQL or Key/Value data stores can serve or write data fast, but to do anything with that data, it (the data) needs to be moved from the store to a processing or an application layer, which creates both network traffic and latency.
Volt + Redpanda: A Recipe for Real-Time Success
Combining Redpanda and Volt makes sense not only for companies that focus on building latency-sensitive (and low-latency dependent) applications but for any organization that needs to exploit its data as effectively and efficiently as possible.
In the next blog, we’ll explain how to connect Redpanda with Volt.
In the meantime, if you would like to learn more or explore how Volt and Redpanda can help you get your event-driven architecture optimized for low latency and high-efficiency usage of resources to gain benefits like those stated above, feel free to reach out to us.
