Stateless Stream Processing: Decoupling Compute and State with Table-First Storage

A widespread assumption across data engineering teams is that operational instability in stream processing is an unavoidable cost of real-time computing. Teams accept ballooning local RocksDB state backends, stop-the-world checkpoint barriers, multi-hour recovery times after node failures, and duplicated state across consumer jobs as permanent realities of streaming engines.

This assumption is incorrect. These operational liabilities are not caused by continuous computation itself. They are the direct result of an architectural flaw in traditional log-first streaming systems: embedding mutable state inside the compute worker. Shifting to a table-first streaming storage model separates state from compute, transforming heavy stateful processors into lightweight, virtually stateless workers.

The Log-First Bottleneck: Dumb Storage and Heavy Compute

Apache Kafka established the log-first streaming model based on an append-only, opaque byte-log abstraction. In this pattern of dumb storage and smart compute, brokers treat record payloads as raw byte buffers without awareness of schemas or record structures.

Because storage is opaque, downstream consumers such as Apache Flink jobs or ksqlDB instances must independently deserialize data payloads, manage schemas, and maintain state locally. When multiple consumer pipelines perform stateful operations such as stream-stream joins or data enrichment, every consumer job is forced to duplicate and materialize the identical upstream dataset within its own local RocksDB state backend. This duplication multiplies memory consumption, disk storage, and CPU overhead across the cluster.

The Operational Cost of Embedded Worker State

Embedding mutable state inside stream processors introduces severe operational liabilities:

• Checkpoint Stalls and Snapshot Bloat: Streaming compute engines must continuously snapshot multi-gigabyte or terabyte local RocksDB state instances to remote storage, creating stop-the-world checkpointing barriers and operational instability.
• Hours-Long Recovery Times: During worker failover or cluster autoscaling events, replacement compute workers must pull massive state files across the network before resuming processing. This network transfer drives recovery time objectives from minutes into hours.
• Dual-Write Anomalies: When streaming pipelines sync intermediate state to external key-value caches to serve point lookups, dual-write synchronization creates race conditions, cache-invalidation failures, and state drift.

Table-First Storage Architecture: Apache Fluss

To address these fundamental limitations, emerging table-first storage engines like Apache Fluss establish mutable, structured tables as first-class storage primitives. Within each partitioned bucket, Apache Fluss couples a streaming write-ahead log (LogTablet) with an embedded key-value store (KvTablet, backed by RocksDB).

This co-located log and key-value architecture enables Apache Fluss to natively serve sub-millisecond primary-key point lookups and streaming changelog updates directly from the storage layer. Engineering teams no longer need to pair Kafka with external key-value databases like Redis or Cassandra. While Apache Kafka remains an engine-agnostic platform for general event messaging, Apache Fluss is tightly integrated with the Apache Flink ecosystem and specifically targets analytical and streaming-table workloads.

Eliminating Consumer State with Flink 2.1 Delta Joins

The combination of table-first storage and modern stream compute engines redefines distributed joins. Apache Flink 2.1 introduces the Delta Join framework (FLIP-486), designed specifically for table-first streaming storage like Apache Fluss.

In a traditional log-first stream-stream join, Flink operators must retain both sides of the stream in local RocksDB state buffers indefinitely. Under the Delta Join framework, Flink operators execute bidirectional, lookup-based joins directly against Apache Fluss primary-key tables. Because state lives in the storage tier, Flink does not need to buffer entire data streams in local operator state. Externalizing state into Apache Fluss eliminates terabytes of redundant state across consumer jobs and can reduce Flink compute resource consumption in CPU and memory by up to 85%. FLIP-486 Delta Joins require the join tables to be backed by streaming storage that supports bidirectional primary-key lookups (like Apache Fluss) and are optimized in Apache Flink 2.1 and later.

Columnar Streaming with Apache Arrow IPC

Traditional row-oriented byte streaming forces downstream consumers to transmit full message payloads over the network and execute CPU-intensive client-side deserialization, even when queries access only a subset of fields from a wide event schema.

Apache Fluss solves this efficiency bottleneck by integrating the Apache Arrow IPC Streaming Format directly on disk and on the wire within its LogTablet storage layer. This integration enables server-side column projection and predicate pushdown at the TabletServer level before bytes are transmitted across the network. For wide-table streaming analytics where consumers select specific column subsets, server-side projection achieves up to 10x read throughput improvements while significantly lowering network bandwidth and serialization CPU overhead. Columnar streaming format benefits are highest for wide tables where consumers selectively access specific columns; narrow or single-field streams see less substantial deserialization and bandwidth gains.

Shrinking Recovery Objectives from Hours to Seconds

Decoupling mutable state from the compute engine fundamentally alters pipeline reliability and operations. Moving mutable state management from downstream stream compute engines into the replicated Apache Fluss storage tier makes Apache Flink streaming pipelines lightweight and virtually stateless.

When a compute worker fails or an autoscaling event occurs, replacement workers do not need to download multi-gigabyte state snapshots over the network. Workers immediately resume processing by querying the table storage tier. This shift shrinks checkpoint snapshot sizes, lowers checkpoint latency, and reduces failure recovery times from tens of minutes or hours to seconds. It also eliminates the cache-invalidation race conditions and state drift common in dual-write cache architectures. Moving to stateless stream processors shifts I/O and query pressure onto storage TabletServers, requiring proper provisioning and low-latency storage tiering.

Automated Lakehouse Tiering into Iceberg and Paimon

In conventional streaming architectures, connecting real-time event streams to historical data lakes like Apache Iceberg, Apache Paimon, or Delta Lake requires building and maintaining separate, brittle change data capture (CDC) and ETL pipelines. This dual architecture introduces data duplication and synchronization lag.

Apache Fluss incorporates an automated, policy-driven Tiering Service that continuously migrates older data segments from its hot real-time log tier directly into open columnar lakehouse formats like Apache Iceberg and Apache Paimon on object storage. This architecture delivers a unified Streamhouse model. Analytical teams can execute sub-second real-time streaming reads and writes on recent data while querying petabyte-scale historical data without maintaining separate ETL synchronization jobs. Near-real-time streaming analytics query the hot Fluss tier with sub-second latency, while historical queries reflect asynchronous lakehouse tiering cadences.

Aeolus Data Solutions view. Decoupling state from stream compute workers resolves the core operational fragility that has burdened real-time platforms for over a decade. By adopting table-first storage with Apache Fluss, Flink 2.1 Delta Joins, and automated lakehouse tiering, data engineering teams eliminate redundant state backends, cut cluster compute consumption by up to 85%, and reduce recovery times from hours to seconds. As data architectures evolve toward unified streaming and AI data readiness, moving mutable state into storage makes streaming pipelines simpler, faster, and far more reliable.

Building Resilient Real-Time Data Architectures

Transitioning from log-first message brokers to table-first streaming storage represents a structural simplification for real-time platforms. Eliminating local worker state removes the primary cause of checkpoint failures, cluster sizing instability, and recovery delays.

If your engineering organization is evaluating table-first streaming architectures, testing Apache Fluss with Flink Delta Joins, or modernizing real-time pipeline state management, Aeolus Data Solutions welcomes the opportunity to discuss your data platform design and technical requirements.