The 80 percent statistic is from 1998

Almost every article about unstructured data opens the same way: 80 to 90 percent of enterprise data is unstructured. It is stated as settled fact, usually without a citation, and it is doing a lot of work in a lot of business cases.

We went looking for the source. It is a Merrill Lynch report from 16 November 1998.

Where the number actually comes from

The report is “Enterprise Information Portals” by Christopher Shilakes and Julie Tylman, and the relevant line hedges even in the original: unstructured data comprises the vast majority of data in an organisation, “some estimates run as high as 80%.” No survey. No census. No methodology. An analyst estimate, offered tentatively, in a document written before most enterprise data in existence today was created.

Twenty-eight years later it is still circulating, having picked up confidence and lost its hedge along the way. It has been re-cited by analyst firms and vendors so many times that the repetition now looks like corroboration. It is not. It is the same 1998 estimate wearing different clothes.

The most honest treatment we found comes from Deloitte, which in a 2017 Tech Trends chapter simply admitted it: “What percentage of the data in existence today is unstructured? No one knows for sure. The generally accepted figure has long been 80 percent, known as ‘the 80 percent rule,’ though recent estimates put the number closer to 90 percent.”

That is a major firm conceding in print that the industry’s most-cited data statistic is folklore. MIT Sloan repeated the 80-to-90 range in 2021 attributed to “multiple analyst estimates,” which is the tell: many estimates, no measurement.

Why this matters beyond pedantry

You could argue the number is directionally right and the fuss is misplaced. Most organisations do hold a lot of documents, images, transcripts and logs. Fine.

The problem is what the statistic gets used for. It is the load-bearing premise in business cases for platforms, and it is uncheckable, which makes it the perfect premise. Nobody can falsify it, so nobody argues with it, so the conversation skips straight past the question that actually determines whether the project works: which specific documents, and what do you need out of them?

A 90 percent figure implies a sweeping programme. The real answer is usually a few document types that carry disproportionate value, and those can be scoped, measured, and finished.

What actually breaks is the parsing

Once you get specific, the difficulty stops being volume and becomes fidelity. The failure mode is that your extraction is subtly wrong and nothing tells you.

The best independent evidence here is OHRBench, an academic benchmark built on 8,561 real document images across seven domains with 8,498 question-answer pairs. Its finding, in its own title, is that OCR hinders retrieval-augmented generation: errors introduced at the parsing stage cascade downstream, and the model has no way to know its source was corrupted.

Practitioners describe the same thing less formally. In a 2024 Hacker News thread on parsing PDFs for RAG, one commenter put the expectation-setting bluntly: “PDFs are extremely messy under-the-hood so expecting perfect output is a fool’s errand.” The advice that thread converged on was not one tool but a toolchain, with different extractors for prose, images and tables, because no single parser handles all three well.

What breaks, consistently: scanned and handwritten documents, multi-column layouts, nested and merged tables, and anything where the visual arrangement carries meaning the text stream loses.

Retrieval quality is an engineering result, not a model result

The most useful published numbers in this area are Anthropic’s, from its contextual retrieval work in September 2024. They measured the top-20-chunk retrieval failure rate and improved it in stages, without changing the model.

Reduction in failed retrievals from pipeline changes alone

Same corpus, same model throughout. Each step builds on the one above it.

Item	Reduction in retrieval failures
Contextual embeddings	35%
Plus keyword search	49%
Plus reranking	67%

Two things are worth taking from that. The first is that hybrid retrieval beats pure vector search, which remains underappreciated by teams who reach for a vector database as step one. The second is that every improvement came from the pipeline, which is the general shape of this work: the model is rarely the constraint.

The governance problem nobody has solved

Here is where we have to be honest about the limits of the current state of practice.

Structured data governance is a reasonably mature discipline. You can tag a column as personal data, propagate that tag through lineage, and enforce access at query time. The equivalent for unstructured data is much weaker. A PII scanner over a document store is probabilistic rather than exhaustive, and once you have embedded that document, the personal data now also exists as vectors in an index, derived from a source you may later be asked to delete.

We looked for authoritative guidance on that specific problem and did not find any. What exists is vendor content asserting that it matters. There is no settled practice for embedding provenance, no standard for recording which source produced which vector, and no regulator guidance we could find addressing derived embeddings directly. If you are handling regulated data this way, you are ahead of the map, and worth being deliberate about it.

Aeolus view. Skip the 90 percent framing entirely, because it will not survive contact with a finance director who asks where the number came from. Pick the two or three document types that actually carry decisions, measure how often your current extraction gets them right, and fix that. A retrieval system over a corpus you have measured beats one over a corpus you have only estimated, and it can be built in weeks rather than quarters.

Where this leaves you

Unstructured data is genuinely worth working on. Contracts, support transcripts, engineering documentation and claims files hold information that never reaches a warehouse, and language models have made that information reachable in a way it was not five years ago.

Just do not start from a statistic that predates the modern web. Start from a document type, a question you need answered from it, and a way to check whether the answer is right. If the extraction is not measured, the retrieval quality is not knowable, whatever the dashboard says.

If you are weighing up a document-processing project and want a second opinion on scope, we are glad to talk it through. Often the honest answer is that the corpus is smaller and more tractable than the pitch suggested.