Processing data streams in real time has become essential for modern businesses. The rapid increase in data generation and the demand for immediate decision-making have rendered traditional batch processing inadequate. Simply collecting and analyzing data after the fact is no longer enough—companies now need to handle data as it flows, applying filtering, transformation, validation, and analytics instantly to respond to events effectively.
For basic tasks, this real-time approach works well. Operations like applying static rules to filter data or converting formats can be performed efficiently. However, a key challenge in stream processing is the need for up-to-date and reliable reference data—such as account balances, stock levels, or system availability. If these stateful operations rely on a conventional SQL database, they suffer from latency issues similar to traditional centralized processing. This is where VoltSP offers a solution.
VoltSP seamlessly integrates with Volt Active Data, an ACID-compliant database built for high-speed transactions while ensuring consistency, durability, and availability. This combination allows both stateless and stateful processing to be executed in real time, without compromising performance.
Table Of Contents
Key Benefits of VoltSP
- Cloud-native – VoltSP is built for cloud environments, requiring no extra infrastructure (such as schedulers or resource managers), making deployment and scaling straightforward.
- Seamless integration with Kafka – Out-of-the-box support for Kafka as both a source and a sink allows for quick pipeline setup.
- Support for complex business logic – Stateful processing is enhanced with partitioned procedures in Volt Active Data, ensuring high-speed execution without increasing latency.
- Flexible configuration – Pipeline templates use placeholders for key elements like server addresses and topic names, enabling easy customization at runtime.
- Scalability – Pipelines can be scaled independently from other components, such as Kafka servers or Volt Active Data nodes, allowing resources to be optimized based on demand.
The VoltSP Architecture
VoltSP’s architecture is structured around three core elements: sources, sinks, and processors. This structure is mirrored in its Domain Specific Language (DSL), enabling users to define stream processing pipelines by specifying the data source, one or more processing steps, and a final output destination.
Processors can be either stateless or stateful, with Volt Active Data providing immediate access to reference information for authentication, validation, or enrichment as data moves through the pipeline.
Circuit breaker
A circuit breaker temporarily halts processing with a remote system, which is important for when a remote system experiences major problems and sending more requests will not help. A circuit breaker will retry sending requests to complete any pending processing.
public interface CircuitBreaker {
State getState();
/**
* Checks state of circuit breaker.
* @return true if circuit breaker is closed
*/
default boolean isClosed() {
return getState().isClosed();
}
default boolean isOpen() {
return !isClosed();
}
/**
* Updates the monitored value and/or opens circuit breaker.
*/
void markFailure();
/**
* Updates the monitored value and/or closes circuit breaker.
*/
void markSuccess();
}
During the configuration phase, an operator can request its own circuit breaker.
interface StreamExecutionContext {
/**
* Use CircuitBreaker to control the data flow of each StreamStage.
* If a sequence of failures occurs, StreamStage will not execute for a certain period.
*
* @param operator to own a circuit breaker
* @return CircuitBreaker assigned to operator within StreamStage
*/
CircuitBreaker getCircuitBreakerFor(Operator operator);
}The circuit breaker counts how many failures or successes were received from the remote When the number of failures goes beyond the configured threshold, the circuit breaker opens. One successful response is enough to close an open circuit breaker.
If the remote system fails to return to a stable state before the configured commitResultTimeout, the system will fail the commit operation and VoltSp will schedule another batch processing (likely with the same data).
Global circuit breaker
The global circuit breaker halts current execution and blocks any further processing until the circuit breaker is closed again. When the pipeline’s logic realizes the remote system can’t process requests, the pipeline may open a global circuit breaker until the current batch is successfully processed.
interface StreamExecutionContext {
/**
* Gives a way to temporarily pause the event processing because of expected of unexpected remote system unavailability.
* Once a circuit breaker is opened voltsp system will wait maximum of {@link VoltEnvironment#CIRCUIT_BREAKER_TIMEOUT_MS_PROPERTY} for remote systems to be back online.
* After configured time voltsp will crash as it cannot make a progress.
*
* @param batchId value of the current batch
* @param dataToRetry non-empty list of events to be reprocessed by failed operator
* @param initialCause of the failure that caused the circuit breaker to be opened
*/
void openCircuitBreaker(long batchId, List<?> dataToRetry, Throwable initialCause);
/**
* Similar to {@link #openCircuitBreaker(long, List, Throwable)} but it will use all events emitted to main sink.
* @param batchId value of the current batch
* @param initialCause of the failure that caused the circuit breaker to be opened
*/
void openCircuitBreaker(long batchId, Throwable initialCause);
}Committer
A committer counts how many requests were sent remotely and how many responses were received. If those numbers are the same, the committer will successfully complete the commit result.
The committer also creates and tracks a CommitHandle created for the given batchId. A commitHandle is used to distinguish commit phases. VoltSp creates a unique batchId for new processing phase and a commit handle is created for that batchId at each commit phase. Batch processing can be retried and for each retry VoltSp will create a new, unique commit handle.
Note that responses related to some old handle are ignored and will not impact the current commit phase.
During the configuration phase, an operator can request its own committer.
interface StreamExecutionContext {
/**
* Use committer to control batch commit for asynchronous processing.
* @param operator to own a committer
* @return a committer
* @param <T> an event type
*/
<T> Committer<T> getCommitterFor(Operator operator);
}Conclusion
As businesses continue to rely on real-time data to drive decision-making, the need for efficient, scalable, and low-latency stream processing solutions has never been greater. VoltSP revolutionizes stream data processing by seamlessly combining both stateless and stateful operations, ensuring that businesses can act on live data without the bottlenecks of traditional databases.
With its cloud-native design, deep integration with Apache Kafka, and ability to handle complex business logic at scale, VoltSP provides a future-proof solution for organizations looking to stay ahead in a data-driven world.
By leveraging VoltSP, companies can move beyond reactive decision-making and embrace a proactive approach—where insights are not just collected, but instantly transformed into action. Whether optimizing financial transactions, monitoring IoT networks, or improving customer experiences, VoltSP delivers the speed, consistency, and flexibility needed to turn streaming data into a strategic advantage.
Now is the time to rethink how your organization processes real-time streaming data. With VoltSP, real-time intelligence isn’t just possible—it’s effortless. Get started now.
