Stop Upgrading the Model. Upgrade the Pipe.

Gartner’s February 2025 prediction (that 60% of enterprise AI projects unsupported by AI-ready data will be abandoned by the end of 2026) is a useful piece of math to keep on the wall. So is the corollary the firm published alongside it: that 63% of organizations either do not have, or are unsure whether they have, the right data management practices for AI. Set those two numbers next to each other and the most common enterprise AI question becomes mathematically suspect. The procurement conversation that begins “our AI initiative isn’t producing the results we promised the board, we think we need to move from provider A to provider B” is being held inside an organization that, on Gartner’s data, has roughly a two-in-three chance of not having measured whether its current system is working well enough to be evaluated.

The question to ask back, before any model migration decision, is can you measure the quality of your current system? Across the production AI conversations I’ve watched in 2025-2026, the answer is some version of “not really,” “we look at user feedback,” or “we have a dashboard for latency.” In which case: the team should not be evaluating a model switch. The team should be evaluating whether it has any idea what it would be buying with the switch, because upgrading a system you cannot measure is not an upgrade; it is a change with hope attached. The change is also expensive.

This essay is the economic argument that follows from the previous four. The model is the easy part, retrieval is where most failures live, trust scaffolding determines adoption, and evals are the infrastructure that makes any of it improvable. From those four pieces falls one conclusion that any CFO can verify against last quarter’s spend: upgrading the model is the wrong place to be spending most AI engineering budget right now. The wins are bigger, the cost is lower, and the durability is higher if that money goes into the pipe instead.

What follows: the math, the decision tree I now use with executives, and the architecture point that turns “upgrade the model” from a multi-month integration project into a routing parameter.

The strongest version of “the model is where the leverage is”

The case for prioritizing model upgrades is real. It deserves serious engagement before dismantling.

Frontier model improvements compound across the entire stack. Every layer above the model (retrieval, tools, evals, workflow) benefits from a smarter generator behind it. A modest improvement in the underlying model can translate to more improvement at the application level, because every layer that depends on the model inherits the gain. A team that focuses on the model layer is choosing the multiplier; teams that focus on infrastructure are choosing the additive components. Over a long enough horizon, the multiplier wins.

There is also an organizational version of the argument. Vendor relationships matter. The team you’re integrated with (their roadmap, their reliability, their pricing trajectory) affects every other decision you make. Picking the right provider early and standing behind that pick is a coherent strategic posture, and constantly second-guessing it produces switching costs nobody planned for.

Both claims have partial truth. Model improvements do compound. Vendor stability does matter. There is a defensible argument that says: pick a frontier provider, follow their roadmap, optimize your stack for their capabilities, and let the next generation of their model lift your entire product.

The part that doesn’t survive contact with the data is the assumption that the wins available at the model layer right now are large enough to justify the spend they consume. They aren’t, by an order of magnitude, and I can show you the math.

The math

The numbers, drawn from published research and production data, all run in the same direction:

Andrew Ng’s HumanEval result is the cleanest single experiment in this space. GPT-3.5 with a single prompt scored 48.1%. GPT-4 with a single prompt scored 67%. GPT-3.5 wrapped in an agentic workflow scored 95.1%. The model upgrade from 3.5 to 4 produced an 18.9-point improvement. The harness change on the cheaper model produced a 47-point improvement. The harness change was 2.5× the size of the model upgrade and ran on a model that costs an order of magnitude less per call.

Stanford’s DSPy team has reported comparable numbers across different tasks. A basic RAG system over StackExchange data goes from 53% to 61% accuracy through systematic prompt and pipeline optimization, without changing the underlying model. A ReAct agent goes from 24% to 51% on the same data, same model, harness changes only. Across multi-use-case studies, the DSPy literature documents 6–17 percentage point gains from optimization, with the largest gains coming from co-optimizing instructions and few-shot examples.

The retrieval side is even more striking. Pinecone Research has documented 15–30% improvement from hybrid search (BM25 + vector with Reciprocal Rank Fusion) over pure vector retrieval. Cohere’s published benchmarks on BEIR show their Rerank 3.5 model improves over hybrid-search baselines by 23.4%. Anthropic’s Contextual Retrieval technique reports up to a 67% reduction in retrieval failures from a chunk-prefixing change to the embedding pipeline alone. None of these wins involved changing the generator model.

Datadog’s 2026 State of AI Engineering report adds the infrastructure dimension: 5% of LLM call spans report an error in production, and 60% of those errors are rate limits: infrastructure problems, not model problems. The thing breaking in production for most enterprise AI deployments is not the model’s reasoning. It is the team’s failure to handle rate-limit backoff, retry logic, and provider fallback.

Now translate those numbers into capital allocation. A frontier model upgrade, properly executed, requires re-evaluation work, integration changes, prompt tuning to the new model’s behavior, regression testing, and roll-out planning. In an enterprise setting this is a multi-month, multi-engineer project. The wins available at the end of it (assuming everything goes well) are typically in the single-digit-percentage range on top of a reasonable baseline. A retrieval improvement project on the same system, with the same team, produces gains measured in tens of percentage points and tends to take less time because retrieval changes are more localized than provider migrations.

The ratio is not subtle. Engineering teams spending on model upgrades are pulling on the smallest available lever, and they are doing it because the model upgrade is the most visible decision in the AI stack; it is the one the board hears about, the one analysts cover, the one Slack debates revolve around. Visibility and leverage are not the same thing.

The decision tree

I walk CIOs through this tree before they approve any model-related infrastructure spend. The tree is meant to surface what’s actually broken, because the answer almost always points away from the model.

First branch. Can you measure current quality? If you cannot (no eval suite, no faithfulness scoring, no production sampling) then the answer is invest in evals first, full stop. Without evals, you cannot tell whether a model upgrade helps. You will spend the budget, ship the new model, and have no data to confirm or refute the success of the project. You will be exactly where you are now, six months later, with less money and the same question. Building the eval suite costs less than the model migration, and the eval suite tells you whether the migration is worth doing.

Second branch. Is retrieval recall above 80% on a representative test set? If not, fix retrieval first. The math is in the previous essay. Hybrid search and reranking against your current corpus is the single highest-ROI change available to a production RAG system. A model upgrade buys a marginal generator improvement on top of bad context; a retrieval improvement fixes the bad context. The latter is cheaper and produces more measurable gains.

Third branch. Is the dominant user complaint about wrong answers (hallucination) or about behavior (refusal, format, tone)? If it’s about behavior (the system is too cautious, too formal, refuses too much) that is sometimes a model problem, and a frontier upgrade can help. If it’s about wrong answers, the previous two branches already told you what to do, and a model swap is the wrong fix.

Fourth branch. Are you hitting rate limits, latency complaints, or provider outages? If yes, the problem is routing and infrastructure, not the model. The fix is a gateway with multi-provider routing, caching, and fallback. This fix often improves user-perceived quality dramatically while not changing the model at all, because what users were experiencing as “the AI feels bad today” was usually “the AI is timing out, retrying, and falling back to a worse path.”

Fifth branch. Have you exhausted the previous four? If you have (you can measure quality, retrieval is solid, the complaints are about generation quality specifically, and infrastructure isn’t the bottleneck) then a model upgrade is worth evaluating. Even then, the right way to do it is to ship the new model behind a routing layer, A/B test it against the incumbent on real traffic with the eval suite running, and roll it out based on data rather than benchmark scores.

I will note something about that fifth branch: in two years of doing this with real organizations, I have had it apply maybe twice. The fifth branch is real, and there are cases where the model is genuinely the bottleneck. They are much rarer than the org-chart energy around model selection would suggest.

Why this is more obvious than it sounds

The reason the decision tree above is hard to internalize is not that the math is unintuitive. It is that the visibility of the model layer is so much higher than the visibility of everything below it. When a new model launches, the board reads about it. When the retrieval pipeline gets re-tuned, the board does not read about it. The model upgrade has a press cycle attached; the retrieval improvement has a Jira ticket attached. Executives who want to be seen leading on AI tend to pull on the lever that produces a press release, which is the model lever, even when the leverage is somewhere else.

This is the failure mode I now watch for explicitly. When an organization is making the fifth-branch decision (fine, let’s evaluate a model swap) and the prerequisites in branches one through four are in place, model evaluation is straightforward. When an organization is making a model decision before the prerequisites are met, model evaluation is theater. It looks like progress. It produces a deliverable. It justifies the AI budget for one more quarter. But it does not produce better outcomes, and the gap between the company that does this and the company that fixes the pipe instead compounds over time.

The CFOs I work with tend to internalize this faster than the CTOs, because the CFOs are the ones whose mental model is “cost per unit of quality improvement,” and the math works out the same way in any spreadsheet. Model upgrades are the highest-cost, lowest-yield investment available in most AI stacks right now. That is not the same thing as saying don’t ever do it. It is saying: do it last, after you have done the things that produce more quality per dollar.

What this requires architecturally

The decision tree is only runnable if model choice is a routing parameter rather than a code change. If swapping models requires three weeks of integration work, you will not A/B test them; you will pick one based on Slack opinion and live with the choice. If swapping models is a configuration parameter at the gateway layer, every model decision becomes an experiment with real evidence behind it.

This is the architectural argument I have made in every essay in this set, and I make it once more because it is the precondition for the entire economic case in this one. AOCore is the version of this gateway we built for AOCyber: a hundred-plus models behind one API, model swaps as configuration rather than code, observability that captures every call so the eval suite can replay history against a new model before any user hits it, and routing logic that handles rate limits, fallback, and provider availability automatically. The decision tree above is something you can run against an AOCore-fronted stack in an afternoon. Without that gateway layer (running directly against each provider’s API) it is a multi-month integration project per provider, which is why most teams stop after picking one provider and never run the experiment.

For organizations that aren’t going to staff an infrastructure team to build this gateway from scratch, AODex is the same architecture delivered as a workspace product. Users get a hundred-plus models through one interface, the model substitution happens behind the product as new providers come online, and the buying organization does not need to renegotiate a contract, change their workflow, or migrate their knowledge base every time a frontier model leapfrogs the last one. The “should we switch?” debate doesn’t happen because the substitution happens at the architecture layer, not at the procurement layer. Buyers get the upgrade-the-pipe benefit without ever having to run the decision tree themselves.

The math on this essay is uncomplicated and the conclusion is unpopular. Stop upgrading the model. Upgrade the pipe. The pipe is where the wins live, the pipe is where the durability lives, and the pipe is the thing the next model upgrade (whenever it eventually does become worth doing) will run on top of. Build the pipe first. The model decision becomes easy once you have.