You Have a Data Hygiene Problem Masquerading as a Model Problem

Adobe’s 2026 AI and Digital Trends Report, conducted with Oxford Economics across 3,000 executives, is the cleanest public data on what enterprises are actually dealing with on data foundations for AI. The headline number: 75% of organizations cite data integration and quality as their top challenge for implementing agentic AI: ahead of talent gaps, ahead of ROI clarity, ahead of every other concern the survey measured. Only 44% say their data quality and accessibility are currently adequate for AI at all. And only 32% of organizations list data quality, unification, and governance among their top AI investment priorities.

Read those three numbers together. Three out of four enterprises know data is their top barrier. Less than half think their current data is adequate. Less than one in three is making data their priority. The strategic blind spot is visible in plain numbers in the survey data: organizations admit data unification is what’s limiting their AI progress, and then they don’t fund the fix. They fund the model selection conversation instead, because the model selection conversation produces a press release and the data unification work produces a Jira backlog.

What sits beneath this (across every organization I’ve seen deploying AI on top of internal knowledge in 2025-2026) is the same shape: knowledge bases full of stale, duplicate, contradictory, or partially-permissioned documents, surfacing as AI quality problems that get misdiagnosed as model problems. The model gets blamed. The retriever gets blamed. The product team panics. The actual problem sits beneath all of it, unaddressed, because data hygiene is unglamorous work that nobody got hired to do.

A typical example of what this looks like inside an enterprise knowledge base: multiple versions of the same policy document, written across different organizational eras, none explicitly deprecated. A 2019 version coexisting with a 2021 update, a 2023 revision that “supersedes prior versions” but where the prior versions weren’t actually removed, and the current policy signed by a senior executive that the retriever returns for some queries and not others. The model is not hallucinating in this kind of environment. The model is correctly summarizing whichever of the six versions the retriever happened to surface first. The “model problem” is a corpus problem. The fix is not a smarter generator. The fix is a librarian.

The numbers are uncomfortable enough to lead with. They make the case better than any narrative I can write.

Adobe’s 2026 AI and Digital Trends Report (conducted with Oxford Economics across 3,000 executives) finds that 52% of organizations admit their current data unification and structure limits the advancement of their AI initiatives, that 75% of organizations cite data integration and quality as the top challenge for implementing agentic AI, and that only 44% say their data quality and accessibility are currently adequate for AI. Gartner has been more direct, predicting in a February 2025 statement that 60% of AI projects unsupported by AI-ready data will be abandoned by the end of 2026. Anthropic’s published Contextual Retrieval work demonstrates the upside in the opposite direction: a single chunk-prefixing change to how documents are embedded reduced retrieval failures by up to 67% in their benchmarks. Modest data and ingestion improvements produce outsized accuracy gains because retrieval errors compound across the pipeline in ways that data improvements correct upstream of everything else.

These are not subtle numbers. They are saying that the majority of enterprise AI failure is being driven by something most enterprises are not investing in, while they pour money into something (model selection) that has substantially smaller leverage on outcomes.

What follows: the case for why, the data hygiene audit I now run with every enterprise AI engagement, and the architectural point that determines whether your hygiene work compounds or has to be done twice.

The strongest version of “bigger models will compensate”

The case for ignoring data hygiene is, like every steelman in this set, partially right. The argument deserves engagement before dismantling.

The argument goes like this. Frontier models have larger context windows, better disambiguation, and more sophisticated reasoning than 2024 models did. A modern generator, given six contradictory expense policies, can recognize the contradiction and either flag it, surface the most recent one, or ask the user which version applies. The model’s improving reasoning capacity is, in effect, a substitute for cleanup work. As models keep getting smarter, the tolerance for messy data goes up, and the marginal value of cleaning the corpus goes down.

There is also the practical version of the argument. Data hygiene is genuinely hard organizational work. It requires owner identification, document deprecation processes, governance committees, change management. It is expensive and politically fraught in ways that vendor selection is not. Given finite engineering bandwidth, the rational choice is to invest in the layer that can be improved by writing code (the model layer, the retrieval layer) rather than the layer that requires inter-departmental negotiation.

Both points have weight. Frontier models are better at disambiguation than older ones. Data hygiene is organizationally hard. There is a real reason most enterprises haven’t fixed this.

The part that doesn’t survive contact with production data is the assumption that the model’s disambiguation capability scales gracefully with corpus messiness. It doesn’t. The model can disambiguate two contradictory documents reliably; it can disambiguate three or four with effort; past a certain corpus messiness threshold, the model’s selection becomes essentially arbitrary, and the user experience becomes “this AI gives different answers to the same question on different days.” That is not a smarter-model fix. The non-determinism is structural at that point, and it lives in the retrieval distribution, not in the generator’s reasoning capacity.

The Anthropic Contextual Retrieval finding I opened with is the empirical version of this. A single upstream change (how chunks are framed before they are embedded) produces up to a 67% reduction in retrieval failures. That is not a saturating curve. It is evidence that most enterprises are sitting on enormous unrealized accuracy gains that no model upgrade can produce for them.

The honest framing: the model is getting better at disambiguating messy data, but the rate of model improvement is slower than the rate at which most enterprise corpora are getting messier, and the engineering investment in data hygiene is the cheapest way to bend the curve in your favor.

What hygiene actually means

A lot of teams hear “data hygiene” and translate it to “we need a data lake” or “we need a Snowflake instance” or “we need a data engineering team.” None of those are wrong, but they are not what this essay is about. The hygiene problem in AI applications is much more specific, and much more tractable, than a generic data infrastructure problem.

The audit I now run on enterprise knowledge bases before they get connected to an AI system has seven binary checks. Score yourself. Most enterprises pass zero to two of these. The retrieval improvements I argued for in essay two are downstream of fixing the items below.

One. Single source of truth identified per topic. For every topic the AI system answers questions about, there is exactly one document that is authoritatively current, and that document is tagged as such. Earlier versions are either deleted, archived in a separate corpus that the retriever never queries, or explicitly marked as superseded. The expense policy example I opened with fails this check at the corpus root.

Two. Freshness is tracked. Every document has a last_verified date, not just last_modified, which is a noisy proxy. Stale documents either get re-verified by an owner on a schedule or get demoted from the retrieval set automatically. Documents older than some threshold that haven’t been re-verified should not be the first thing the retriever returns.

Three. Conflicts are surfaced rather than buried. When two documents in the corpus disagree, the retrieval system should know it, and the system should either prefer the newer one explicitly, ask the user which applies, or refuse to answer. Conflicts that produce random behavior depending on retrieval ordering are the worst possible failure mode because they erode user trust without any single answer being identifiable as wrong.

Four. Permissions are enforced before retrieval, not after generation. The retriever should not be able to pull documents the user lacks access to. This is a security property and also a quality property: a system that retrieves restricted documents and then redacts them produces strange artifacts, while a system that filters at retrieval produces clean answers grounded only in what the user is entitled to see.

Five. Schema and metadata are consistent enough for filtering. “Policy” vs “policy” vs “POLICY” should all match in metadata filters. Document types should be enumerated, not free-text. The schema should be designed so that the retrieval system can filter by category, owner, date, or scope without having to fuzzy-match against a thousand spelling variants.

Six. Deduplication is semantic, not just hash-based. Hash deduplication catches identical files. It does not catch the case where the same policy was uploaded as a PDF, a Word doc, and a copy-pasted Confluence page. Semantic deduplication (clustering on document similarity, then deciding which version is canonical) catches these cases. Most enterprise corpora have a duplication rate, by semantic measure, of 30-50%. That is exactly the same multiplier as the data-quality-to-accuracy finding.

Seven. The corpus delta is auditable. What changed in the corpus this week? Who added what? What was deprecated? The answer should be available, and it should be easy to query. Without a corpus delta, you have no way to investigate the moment three weeks ago when the system started giving wrong answers, because you cannot reconstruct what the corpus looked like then.

Seven items. Most enterprise knowledge bases pass zero to two. The teams that fix these items see retrieval accuracy improvements that no model upgrade in the next eighteen months will produce.

Why this is structural, not technical

Something direct about why data hygiene is so consistently neglected: the reason is not what most engineers think it is.

The technical work of data hygiene is not hard. The audit above is implementable in any standard data engineering stack in a quarter. There are no algorithmic challenges, no research problems, no waiting on a vendor roadmap. The technology to do this has existed for years. What is hard is the organizational work, and that is where most efforts fail.

Single source of truth requires someone with authority to declare which version of a document is canonical. That person needs the political standing to override departmental ownership. Freshness tracking requires document owners to commit to a re-verification cadence, and someone has to enforce it. Permission enforcement requires alignment between the access policy of the knowledge base and the access policy of the document repository. None of these are engineering problems. All of them are governance problems.

This is why I now tell organizations that data hygiene is the first thing they should staff for an enterprise AI initiative, not the last. Most do the reverse: they pilot the AI system on the existing corpus, see the quality problems, blame the model, and then six months in start the painful work of corpus cleanup with a worse organizational mandate than they would have had at the start. The teams that do hygiene first ship faster overall, because the pilot starts producing valid results instead of debugging exercises.

What this requires architecturally

The audit above is a one-time cleanup, but the conditions that produced the messy corpus in the first place do not go away on their own. Without architecture, the corpus rots again on a multi-quarter timescale, and you end up doing the hygiene work twice. The architecture is what makes the cleanup durable.

The architectural pieces, on the build path: ingestion pipelines that enforce metadata at upload time, deprecation workflows tied to document ownership, freshness alerts that demote stale content automatically, semantic dedup running on a schedule, and corpus-delta logging that gives you a queryable history. AOCore’s knowledge-base layer enforces these as architecture: permissioned retrieval, versioned ingestion, freshness metadata, audit logs of every document that flowed through every query. The point is not that AOCore magically makes your corpus clean. The point is that AOCore makes the cleanup stick, because the architecture continuously enforces the conditions that the cleanup established. Most teams discover the data hygiene problem too late because they have no visibility into which document the model actually saw when it produced the wrong answer; AOCore is that visibility built in.

For organizations that aren’t going to build this infrastructure from scratch, AODex inherits the same hygiene posture as a finished product. Users upload documents and they get versioned, permissioned per workspace, and tied to the conversation lineage that produced any answer. The no-training-on-customer-data guarantee is architectural, not contractual. Buyers don’t audit their RAG hygiene because the audit story is the architecture; they upload, query, and see the citation back to the document that produced the answer. The clean-corpus posture is the default state of the product, not a setting they have to configure or a roadmap item they have to wait for.

The thing I keep wanting to say to executives about data hygiene is the thing they are most resistant to hearing. The AI quality problem you are trying to solve with a better model is, most of the time, a librarian problem. It has been a librarian problem for the entire history of information systems, and AI has not changed the underlying truth; it has only made the consequences of bad librarianship more visible by surfacing them in plain language to end users. The fix is not the model. The fix is the work nobody wanted to do, that was overdue before AI showed up, that AI has now made urgent.

Do the hygiene work. Then talk to me about model upgrades.