Batch or streaming is becoming a table property

Most teams still treat batch versus streaming as an architecture decision: you pick one at design time, you build around it, and changing your mind later means rewriting the pipeline. That framing is quietly falling apart, and it is worth understanding why before you commit a quarter to either side.

The people doing the work got there first. In a June 2026 thread on r/dataengineering about where micro-batching ends and streaming begins, the most upvoted answer collapsed the distinction in one line: “At the end of the day a stream is a batch of 1. You are still conceptually optimizing latency in either scenario as well but the difference is in the orders of magnitude.”

That is closer to how the infrastructure now works than most architecture diagrams are.

The engines stopped making you choose

Apache Flink 2.0 landed on March 24, 2025, the first major release in the nine years since 1.0, built from 25 FLIPs and 369 issues. Two changes matter here. State moved out to remote storage as its primary home, with local disk demoted to cache, which decouples how much state a job holds from the machine it runs on. And Materialized Tables arrived, which the project describes as letting users “declaratively manage both real-time and historical data through a single pipeline, eliminating the need for separate codebases or workflows.”

Spark went the other direction toward the same place. Databricks previewed Real-Time Mode for Structured Streaming on August 19, 2025, and it landed in open source in Spark 4.1.0 under SPARK-53736: “First official support for Structured Streaming queries running in real-time mode for continuous, sub-second latency processing. For stateless tasks, latency can even drop to single-digit milliseconds.”

What shipped there is a trigger type, not a new engine or a separate API. It is the same query running at a different latency because someone changed a setting.

The log and the table became the same object

The more consequential shift is in storage, where the queue and the warehouse table are collapsing into one thing.

Confluent’s Tableflow reached GA on March 19, 2025, turning Kafka topics directly into Iceberg tables with no ETL job in between, and expanded on October 29, 2025 to Delta Lake and Unity Catalog. Redpanda shipped Iceberg Topics GA on April 7, 2025, where a topic property makes the topic materialize itself as a table. Aiven’s equivalent claims it removes roughly 60 percent of Kafka sink connector use cases, which is a vendor number but a plausible one given what a sink connector actually does.

Then in June 2025, Alibaba donated Fluss to the Apache incubator, a columnar streaming storage layer whose stated purpose is closing the gap where teams run Kafka for streaming and Iceberg for analytics with a bridge between them. Its 0.8 release added Flink Materialized Tables with a declared freshness target.

That last detail is the whole argument in miniature. You state how fresh the table needs to be. The engine decides how to run. Batch and streaming stop being architectures you choose and become a number you configure, in the same way that Iceberg stopped being a bet worth agonizing over.

What actually made streaming expensive

KIP-1150, which proposes writing partition data straight to object storage instead of broker disks, passed its vote on March 2, 2026. Its justification is worth reading closely, partly because it comes from the Kafka project rather than a vendor, and partly because it does not talk about compute at all. It identifies cross-availability-zone replication traffic as “the most substantial infrastructure cost for Apache Kafka operators on hyperscalers today,” and prices it: AWS charges $0.02 per GiB, Google Cloud $0.01 per GiB.

Streaming was expensive largely because of network egress between zones, rather than because processing events continuously is inherently costly. That is a fixable problem, which is why roughly a dozen vendors now ship some form of direct-to-object-storage Kafka.

It also deserves the caution that Jack Vanlightly, himself a proponent, offered in October 2025: “Cross-AZ charges are typically what are on people’s minds, but it’s a mistake to think of S3 simply as a cheaper disk or a networking cheat.” Object storage carries its own latency and per-request charges, so the bill moves rather than disappearing.

The savings claims deserve an audit, and one exists

Every diskless vendor advertises a percentage. WarpStream’s pricing page leads with “Kafka Streaming at 80% Lower Cost.” AutoMQ claims over 90 percent. Confluent’s Freight clusters claim up to 90 percent. An AWS blog co-published with AutoMQ claims 94 percent.

Claimed cost savings from diskless Kafka, and who is doing the claiming

Every figure below was published by a party selling the alternative. None has been independently reproduced.

Item	Claimed saving versus self-hosted Kafka
WarpStream vendor-reported	80%
AutoMQ vendor-reported	90%+
Confluent Freight vendor-reported	up to 90%
AWS with AutoMQ vendor co-published	94%
Independently reproduced no such figure exists	none

Those numbers are all produced by the party selling the alternative, and one person actually checked the arithmetic. Stanislav Kozlovski, a former Confluent Kafka engineer, audited WarpStream’s public cost calculator in December 2024 and found it overstated self-hosted Kafka costs by 2.8 times in the base case, rising to 5.6 times once realistic optimizations were applied. He documents the mechanisms: SSD pricing used where HDD pricing belongs, an instance recommendation five to ten times larger than needed, a replication term that double-counts, and a compression ratio adjusted in a way that raises Kafka’s cost while leaving WarpStream’s own figure untouched on identical inputs. His conclusion at the time was that self-hosted Kafka came out cheaper on GCP and Azure.

We are not claiming the diskless architecture is bad. The KIP’s own reasoning about cross-zone traffic is sound, and it passed an Apache vote on the merits. The claim is narrower and worth holding onto: the headline savings percentages are marketing artifacts, the only independent audit found the baseline inflated several fold, and nobody has reproduced them since.

The part nobody can show you

Three things came up empty when we went looking, and the absences are more useful than another vendor chart.

Nobody has published a credible like-for-like cost comparison of batch, micro-batch, and true streaming on the same workload. Not a vendor, not a university. What circulates instead are contradictory blog anecdotes, including two posts by the same author that reach opposite conclusions. If someone quotes you a multiplier for what streaming costs, ask where it came from.

There is also no independent survey with disclosed methodology showing what share of pipelines are actually batch versus streaming, or what latency teams genuinely require. Confluent’s annual Data Streaming Report is real research with a real methodology, but it surveys people pre-screened as “familiar with data streaming” at companies with 500 or more employees. By construction that cannot tell you an industry base rate, and it cannot count the teams who evaluated streaming and said no.

The newest argument for streaming is the thinnest of the three. A lot of content right now asserts that AI agents require sub-second data freshness, and as far as we can tell every source making that case sells streaming or CDC tooling. We could not find one independent study measuring whether agent task success actually degrades with data that is minutes old rather than milliseconds old. The latency budgets quoted for voice and chat agents are about response time, which is a genuine constraint, but responding fast is not the same requirement as reading from a table refreshed a moment ago.

While we are clearing folklore: the line that data “loses its value within seconds” has no rigorous source. The closest real finding is Jordan Tigani’s 2023 observation from BigQuery logs that week-old data is about 20 times less likely to be queried than same-day data. That is a statement about query frequency in one company’s logs. It has been inflated into a law about value that he did not make.

What this looks like in a real system

The most convincing evidence that the distinction is becoming an infrastructure property rather than a philosophy is not an essay. It is a pull request.

PostHog’s data warehouse team spent July 2026 discovering that their own job queue did not know the difference. One PR from July 13 notes their compaction ran unconditionally after every final batch, which is fine for a scheduled sync finishing a few times a day and wrong for a CDC schema finishing every extraction tick. A second from July 17 quantifies the consequence: CDC streaming merges “are a structurally different workload from bulk full-refresh and incremental loads: in prod-US they are 52% of queue batches from only 12 teams, with a p95 batch duration 5.5x” the rest.

That is what the distinction looks like once it stops being an architecture diagram and becomes something your scheduler has to model correctly.

Aeolus view. Asking whether a platform should be batch or streaming tends to produce an argument. Asking what freshness a particular table owes the thing that reads it tends to produce an answer, and usually a cheaper one. Run that question table by table and most of the debate goes away, because for the majority of tables the honest answer is hours, and for a small number it is seconds. Build that small number properly and leave the rest alone. The teams we see struggling are the ones who picked an architecture first and worked out which tables needed it afterwards.

Where this leaves you

Ben Rogojan’s heuristic in his own piece on this question is a good one and worth keeping: if a system needs to react, think real-time; if it needs to be analyzed, think batch. What the last eighteen months add is that the cost of being wrong is falling, because the engines, the storage layer, and the table formats are all converging on letting you change the answer without rebuilding.

Two things follow. Treat freshness as a per-table requirement you write down and revisit, not a platform-wide identity. And treat every cost percentage in this space as a claim requiring a source, including the ones that support the choice you already want to make.

If you are weighing this for a specific pipeline and want a second opinion on whether the latency is genuinely required, we are glad to talk it through. Often the useful answer is that a cheaper table would do, and we would rather tell you that than sell you a rebuild.