Your 2026 data stack has one new layer

The most common assumption we hear from teams planning a 2026 data platform is that getting ready for AI means adding AI-native tools to the stack. A vector database here, an agent framework there, possibly a warehouse migration.

That is not what this year looked like. The layers of the stack have the same names they had in 2024. What actually happened is that four of them consolidated into two vendors, the storage format argument quietly ended, and one genuinely new layer appeared. That new layer is the one worth your attention, because it is the one nobody can sell you off the shelf.

The stack consolidated, and every deal was pitched as AI infrastructure

Ten months, four transactions.

Ten months that consolidated the transformation and orchestration layers

1. 3 Sep 2025

   Fivetran acquires Tobiko Data

   The company behind SQLMesh and SQLGlot.

2. 13 Oct 2025

   Fivetran and dbt Labs announce an all-stock merger

3. 25 Mar 2026

   Fivetran contributes SQLMesh to the Linux Foundation

   Announced at KubeCon Amsterdam.

4. 1 Jun 2026

   The merger closes, and dbt Core v2.0 ships the same day

   Core and the Rust Fusion engine move onto a shared foundation.

5. 13 Jul 2026

   Prefect acquires Dagster Labs

   The two most widely adopted Airflow successors, under one roof.

Fivetran acquired Tobiko Data, the company behind SQLMesh and SQLGlot, on September 3, 2025. Six weeks later it announced an all-stock merger with dbt Labs, which closed on June 1, 2026 with George Fraser as CEO and Tristan Handy as President. In between, it contributed SQLMesh to the Linux Foundation at KubeCon Amsterdam on March 25, 2026. One company now stands behind both major SQL transformation ecosystems.

On the same day the merger closed, dbt shipped Core v2.0, ending the split between the Python Core and the Rust-based Fusion engine. In dbt’s words: “From now on, dbt Core and Fusion will be built on a shared foundation.” Core remains Apache 2.0.

Then on July 13, 2026, Prefect announced it is acquiring Dagster Labs, putting the two most widely adopted Airflow successors under one roof. Read the framing carefully. Prefect describes the combined company as Dagster for defining outcomes, Prefect for executing them, and FastMCP for governing agent access. The New Stack’s headline was blunter: “it’s not a data pipeline story.”

Notice the pitch each time. Fivetran and dbt named the combined entity “the data infrastructure for trusted AI agents.” Prefect bought an orchestrator and talked about MCP. Nobody is selling ETL anymore.

Meanwhile the incumbent conceded the design argument. Airflow 3.0 went GA on April 22, 2025, described by the project as “the biggest release in Airflow’s history,” with asset-oriented scheduling, DAG versioning, and a client-server task execution interface. The asset-centric model that Dagster spent years arguing for is now in Airflow itself.

One thing worth saying plainly: none of this changes what you should build next week. If you run dbt and Airflow today, you still run dbt and Airflow tomorrow. Consolidation is a procurement and roadmap risk, not an architecture one.

The storage argument quietly ended

Iceberg v3 was ratified in mid-2025 and adds deletion vectors, row lineage, default column values, and variant and geospatial types. AWS shipped V3 deletion vector and row lineage support on November 26, 2025 across EMR, Glue, SageMaker notebooks, and S3 Tables. Snowflake made write support for externally managed Iceberg tables generally available on October 17, 2025. Databricks put Iceberg v3 into public preview on April 9, 2026. On the catalog side, Apache Polaris graduated to a top-level Apache project on February 19, 2026.

One correction while we are here, because it circulates widely: Delta Lake was not donated to Iceberg. There was no such event. What actually exists is Delta’s UniForm feature, which exposes Iceberg-readable metadata over the same Parquet files, plus two specifications that converged independently on similar capabilities. Interoperability, not a merger.

The practical read is that table format is no longer a bet you can lose badly. Pick Iceberg, put a catalog in front of it, and spend the argument budget somewhere it matters.

The layer that is actually new

Here is what did not exist in your 2024 stack: a defined, governed contract for what a model is allowed to ask, and what it gets back.

The reason you need one is unglamorous. Language models are far better at SQL than they were, and still not good enough to point at a raw production schema. Academic benchmarks moved from close to solved on clean single-table problems to substantially unsolved on realistic enterprise schemas, which is what Spider 2.0 was built to measure. Treat any specific leaderboard number with care. A CIDR 2026 paper that re-annotated the BIRD benchmark found relative performance shifts ranging from minus 3 percent to plus 31 percent, and rank changes of up to three positions, once annotation errors were corrected. The benchmarks are noisy. The direction they point is not.

The industry’s answer is the semantic layer, and over the last year it stopped being a niche product category. dbt Labs open-sourced MetricFlow under Apache 2.0 on October 14, 2025, framing governed metrics as infrastructure for trustworthy agents, and joined Snowflake and Salesforce in the Open Semantic Interchange initiative. Snowflake made semantic views first-class schema objects rather than YAML sitting in a stage. Databricks shipped Genie Ontology on June 16, 2026 as an automatic context layer over tables, queries, and dashboards.

The vendor numbers in this space deserve a health warning. Databricks reports Genie answering 84.5 percent of questions correctly on the first attempt against 52.4 percent for what it calls the strongest general-purpose coding agent, on a 28-question suite that Databricks designed and ran itself. dbt Labs’ April 2026 comparison is more candid: raw text-to-SQL accuracy nearly doubled from 32.7 percent to 64.5 percent between 2023 and 2026, while the semantic layer returned correct results every time for questions inside its coverage. That test is eleven questions. Useful color, not proof.

The structural argument is stronger than any of those figures. If a metric is defined once and the join logic is generated deterministically, the model’s job shrinks from “write correct SQL across forty tables” to “pick the right metric and the right dimensions.” That is a smaller problem, and a much smaller blast radius when it gets the answer wrong.

The access path is MCP, and it is not safe by default

The semantic layer decides what an agent may ask. Something still has to carry the question. In practice that is now the Model Context Protocol, and most data platforms ship a server for it.

The protocol also stopped belonging to one vendor. On December 9, 2025 the Linux Foundation formed the Agentic AI Foundation, with Anthropic contributing MCP as a founding project alongside platinum members including AWS, Google, Microsoft, and OpenAI. If you were holding off on MCP because it looked like a single-vendor protocol, that objection has expired.

The specification has been hardening too. The 2025-06-18 revision classified MCP servers as OAuth Resource Servers, required clients to implement RFC 8707 Resource Indicators so a malicious server cannot obtain tokens scoped beyond itself, added structured tool output, and introduced a dedicated security best practices page.

It hardened because it needed to. Simon Willison’s lethal trifecta, published June 16, 2025, names the combination precisely: access to private data, exposure to untrusted content, and the ability to communicate externally. An agent with a warehouse MCP server and a web-fetch tool has all three by lunchtime. Prompt injection sits at the top of the OWASP Top 10 for LLM Applications, with vector and embedding weaknesses at number eight.

These are not hypothetical. Invariant Labs demonstrated tool poisoning in April 2025, where instructions hidden in a tool description are invisible to the user and fully visible to the model. In September 2025 Snyk found a malicious npm package posing as an MCP server that silently blind-copied every outbound email to an attacker’s domain. And CVE-2025-49596, a critical remote code execution flaw in Anthropic’s own MCP Inspector, carried a CVSS 4.0 score of 9.4 when it was published on June 13, 2025. Treat an MCP server as a dependency with database credentials, because that is what it is.

The design consequence is small to state and real work to implement. An agent should have its own identity rather than borrowing a human’s credentials, so that a query is attributable to a specific agent and task. It should reach data through the governed layer rather than through a general-purpose SQL tool. And the same column-level lineage and PII tagging you would want for an audit is what tells you which agent touched what.

That last point is about to get more expensive to ignore. The EU AI Act’s general application date is 2 August 2026, per Article 113 of Regulation (EU) 2024/1689. The prohibited-practices and general-purpose-model obligations are already live. The high-risk system deadline has reportedly been deferred to December 2027 through the Digital Omnibus package, though we would treat the exact dates as still settling and check them against the Official Journal before making a compliance plan.

The basics did not stop mattering

Ben Rogojan makes this argument well in his own 2026 stack write-up, and it is worth repeating: teams abandon fundamentals while chasing AI, and the operational gap shows up later. Ingestion should be boring. Cost compounds silently. Undefined quality expectations hurt more than known-bad data.

The survey evidence suggests this is exactly where attention is not going. dbt Labs’ 2026 State of Analytics Engineering report, fielded between December 2025 and February 2026 across 363 respondents, found 72 percent prioritizing AI-assisted coding against 24 percent prioritizing AI-assisted pipeline management, which is where testing, observability, and quality controls live. Over the same period, the share rating “increasing trust in data” as important rose from 66 percent to 83 percent. Everyone wants the trust. Far fewer are funding the thing that produces it. That is dbt’s own survey and dbt sells into this problem, so read it accordingly, but the shape of the finding matches what we see in the field.

Where the investment is going, versus where trust actually comes from

Share of respondents naming each as a priority.

Item	Share of respondents
AI-assisted coding	72%
AI-assisted pipeline management testing, observability, quality controls	24%

Two research results are worth knowing if you are building retrieval. Chroma’s context rot work, published July 14, 2025 across 18 models, found that “models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows.” Degradation starts well before the context limit, so a larger window is not a fix. And Anthropic’s contextual retrieval results remain the clearest demonstration that the pipeline around the model does the heavy lifting: adding generated context to each chunk before indexing cut retrieval failures by 35 percent, combining it with keyword search reached 49 percent, and adding a reranking step reached 67 percent. Same corpus, same model.

A note on the numbers

This field runs on a few statistics that do not survive contact with their sources, and we would rather not repeat them.

“87 percent of data science projects never reach production” traces to a 2019 VentureBeat article reporting a remark made by an IBM executive on a panel at VentureBeat’s own conference. There is no published methodology or sample behind it. “85 percent of AI projects fail” is a rewriting of a 2018 Gartner forecast that through 2022, 85 percent of AI projects would deliver erroneous outcomes due to bias. That is a different claim, about a window that closed four years ago.

The MIT NANDA figure that roughly 95 percent of organizations see no measurable return from generative AI is real and recent, but narrower than its circulation suggests, and it is frequently pressed into service as a data-quality statistic. It is not one. The report attributes the gap to a learning problem, stating that the core barrier to scaling “is not infrastructure, regulation, or talent.” Quoting it in support of a data argument is a mistake a careful reader will catch.

RAND’s study of why AI projects fail is the more useful document, and note that its widely quoted “more than 80 percent” is an external estimate RAND cites rather than a RAND finding. RAND’s own contribution is root-cause work drawn from 65 practitioner interviews, two of whose five identified causes are directly about data and infrastructure readiness. We made the longer version of that case in Enterprise AI runs on data engineering.

Aeolus view. The consolidation news is loud and mostly does not require you to do anything. The quiet requirement is that every AI feature you ship creates a new consumer of your data, one that cannot ask a colleague what a column means and will confidently invent an answer instead. Building the layer that constrains what it can ask is unglamorous work with a much better return than another round of model selection. If your metrics are not defined in one place yet, start there, not with an agent framework.

Where this leaves you

If you are planning a platform for the next twelve months, the honest version is short. Storage and format are settled enough to decide quickly. Ingestion, transformation, and orchestration are consolidating, which is a contract and roadmap question rather than an engineering one. The work that will actually determine whether your AI features are trustworthy sits in the layer between the model and your tables: defined metrics, an access path with its own identity, lineage good enough to answer what an agent touched, and retrieval built with the care the research says it needs.

We build and advise on exactly that layer, so we are not neutral about its importance. If you want a second opinion on where your stack sits, we are happy to have the conversation, and if the answer is that your foundations need work before any agent touches them, we will tell you that straight.