The Model Is the Easy Part

In early 2024, Andrew Ng published a result that should have ended an industry debate and instead got buried under the next month’s leaderboard launches. He took GPT-3.5 (the cheaper, weaker, older model) wrapped it in what he called an “agentic workflow,” and ran it against HumanEval, the standard coding benchmark. GPT-3.5 on its own scored 48.1%. GPT-4 with a single prompt scored 67%. GPT-3.5 inside the workflow scored 95.1%.

A weaker model wrapped in better engineering nearly doubled the performance of a stronger model by changing how it was used, not which one was used. That is not a small finding. It is the cleanest single statement of where production AI value actually lives in 2026, and the industry has spent the eighteen months since trying to forget it.

Let me make the case for that without pretending the model doesn’t matter. It does. Frontier capabilities compound. A team that ignores them is just choosing a different way to lose. But the question on every CIO’s desk this quarter (which model do we standardize on, when do we upgrade) has the leverage exactly backward. Three years of production deployments have made it embarrassingly clear that the model is now the easy part. The defensibility, the difficulty, the differentiating engineering have all moved somewhere else.

The strongest version of the model-first argument

I’m going to give the model-first view its best shot before I take it apart, because the steelman is real and ignoring it is how you end up writing a polemic instead of a useful piece.

The strongest case for prioritizing model selection comes from Rich Sutton’s Bitter Lesson, and it cuts deep. Sutton, summarizing seventy years of AI research in 2019, argued that every generation of researchers tries to build knowledge into their systems through clever engineering, and every generation watches that engineering get absorbed by the next round of scaling. Hand-crafted features got eaten by deep learning. Hand-crafted reasoning chains got eaten by chain-of-thought prompting, which got eaten by native “thinking” modes in the latest frontier models. Tool calling started as scaffolding glued together with regex and ended up as a first-class API the model speaks natively. The pattern is reliable enough to be a law: whatever engineering you build on top of the model today, the next model will subsume. The implication, if you take it seriously, is that engineering investment around the model is a depreciating asset. The model is the appreciating one.

If I had to argue against my own thesis, that’s where I’d start. Sutton is right about individual techniques. The harness I would build in 2023 looks ridiculous in 2026; the harness I would build today will look ridiculous in 2028. Specific scaffolding gets absorbed. That part of the argument is settled.

The mistake is in the next step: the assumption that because individual techniques get absorbed, the practice of engineering around the model will also disappear. It hasn’t. The work just keeps moving up the stack. When chain-of-thought got absorbed, we didn’t stop doing prompt engineering; we started doing context engineering, and the leverage there is bigger than CoT ever was. When tool calling got absorbed, we didn’t stop doing integration work; we started doing harness engineering, and the production wins are larger than the gains from tool calling ever produced. The scaffolding that gets absorbed is the finished scaffolding. The scaffolding being built right now (the part you’d actually invest in this quarter) is at the frontier where the model is still struggling.

So the honest version of the bitter lesson, for an operator in 2026, is: individual techniques depreciate, but the discipline of engineering around the model is on a treadmill that pays compound interest. The teams that stopped doing that engineering in 2024 because “the next model will handle it” lost two years of position. The teams that kept doing it gained those two years.

The category error

The move that has cost the industry two years and an enormous amount of misallocated investment works like this. Somebody looked at the rapid pace of model improvement, noticed that benchmarks were moving fast, and concluded that all the value in an LLM application lived in the model layer. Everything else was just plumbing: temporary, replaceable, unimportant.

That conclusion would be correct if LLM applications were single-call API wrappers. They were, briefly, in 2023. They haven’t been since. A production LLM application in 2026 is a stack. A model at the bottom, sure. Above it sits a retrieval pipeline, a context engineering layer, a tool integration layer, an evaluation harness, an observability layer, a routing and fallback layer, a guardrails layer. On top of all of it, a workflow design that determines whether users can actually get value out of any of it. The published evidence on where the gains live is unusually consistent. Andrew Ng’s HumanEval result is the cleanest single data point: same task, same benchmark, the harness change on the cheaper model produced a 47-point quality improvement against the model upgrade’s 18.9-point improvement, a ratio of roughly 2.5×. Stanford’s DSPy research has published comparable numbers across different tasks: a basic RAG system over StackExchange data goes from 53% to 61% accuracy when systematically optimized, and a ReAct agent goes from 24% to 51%, all on the same underlying model. Anthropic’s Contextual Retrieval work reports up to a 67% reduction in retrieval failures from a single architectural change to the chunking pipeline. Datadog’s State of AI Engineering 2026 report finds that 5% of production LLM calls error out, and 60% of those errors are rate limits: an infrastructure problem rather than a model problem, dominating the failure profile of agents in production. None of those wins came from a model upgrade. All of them came from engineering work upstream and around the model layer.

The category error is treating “the LLM application” as synonymous with “the LLM.” It isn’t. The LLM is one component. The application is a system. Systems are where engineering organizations earn or lose their durability.

A map of where the value actually lives

After building, advising on, and watching dozens of production LLM systems, the picture I keep coming back to looks like this. A layered diagram, top to bottom, with two annotations per layer: where the wins are and where the attention is. The inversion between the two is the entire argument of this essay.

At the top of the stack sits model selection: choosing GPT vs. Claude vs. Gemini, picking the right tier within a provider, deciding when to upgrade. Where the attention is: maximum. Where the wins are: small and shrinking. The frontier is converging, the price floor is collapsing, and provider risk (as the Anthropic-Pentagon crisis in February 2026 made painfully clear) is now larger than the quality delta between any two top-tier models.

Below that sits prompt engineering: how you phrase the instruction. Where the attention is: large. Where the wins are: marginal beyond a reasonable baseline. Frontier models in 2026 are good enough at understanding intent that the elaborate role-priming and chain-of-thought scaffolding that defined 2023 prompt work has been mostly absorbed into the model’s native capability. Past the baseline, the additional gains from manual prompt refinement are small.

Below that sits context engineering: what information you place in the model’s working memory, in what order, with what priority. Andrej Karpathy’s framing: if the LLM is the CPU, the context window is the RAM, and your job is operating-system design. Where the attention is: rising fast. Where the wins are: large.

Below that sits retrieval: how you find the right context to put in the window in the first place. The published evidence is consistent that the majority of RAG failures in production are retrieval failures rather than generation failures: ASTUTE RAG (Google + USC, ACL 2025) reports that under realistic conditions roughly 70% of retrieved passages don’t directly contain the true answer, and Stable-RAG (2026) shows that even when the gold document is in the retrieved set, simply reordering passages causes the generator to produce different answers. I’ll write more about this in the next essay, but the short version is: every “the model hallucinated” complaint is more likely to be a retrieval miss than a generation defect. Where the attention is: low. Where the wins are: enormous.

Below that sits tool design: the APIs your agent can call, the structure of their inputs and outputs, the way they handle pagination, errors, rate limits, and authentication. Datadog’s 2026 State of AI Engineering report found that 5% of all LLM call spans report an error, and 60% of those errors are rate limits. Rate limits are not a model problem. They are a plumbing problem.

Below that sits evaluation: the harness that tells you whether any of the layers above are working. Hamel Husain, who has spent the last two years consulting on production LLM deployments, has been consistent on one point: unsuccessful AI products almost always share a single root cause, which is the failure to build a robust evaluation system. The teams that ship reliably have evals. The teams that ship and pray do not.

Below that sits observability and routing: knowing what happened on every call, being able to swap models without a code change, handling fallback when a provider is unavailable. This is the layer that turns a fragile single-vendor integration into a portable production system.

And at the bottom (but only in the architectural sense, not in terms of importance) sits workflow and user experience design. How the user is guided through the interaction. Where confidence is surfaced. Where the system can say “I don’t know.” Whether actions require confirmation. Whether sources are cited. This is the layer that determines whether your beautifully-engineered system actually gets adopted, or whether it dies in pilot after the first confident wrong answer goes to a customer.

Model selection sits at the top of attention and the top of org-chart energy. The wins are everywhere below it.

What this means in practice

If you accept the map (and after eighteen months of production deployments, I don’t know how a serious operator looks at it honestly and doesn’t), three things change about how you allocate engineering investment.

First, which model to use stops being a strategic question and becomes a routing parameter. If you’ve built the infrastructure to swap models, the question becomes “what does the eval say?” and the answer is something an automated pipeline can produce. If you haven’t, the question becomes “what does the loudest engineer on Slack think?” and the answer is whatever model was on top of the benchmark leaderboard last Tuesday. The first version is a system. The second version is theater.

Second, upgrading the model stops being the default response to quality problems. If your hallucination rate is high, the question isn’t “should we move to a smarter model”; it’s “is retrieval recall above 80%, are we doing faithfulness evals, do we even know where the bad answers are coming from?” The published evidence on harness-versus-model gains is consistent across multiple research groups, and the gap is not small. The teams that keep reaching for model upgrades when the problem is retrieval are spending capex on the smallest available lever.

Third, engineering investment around the model stops being optional. The bitter lesson is real for individual techniques. It is not real for the discipline. The teams that stopped investing in eval harnesses, observability, and context engineering because they expected the next model to handle it are now two years behind on infrastructure their competitors quietly built while the leaderboard arguments raged. That gap is the actual moat. It is small, technical, unsexy, and durable in a way that “we standardized on Claude 4” never was.

What to build

If you’re building this yourself, the place to start is the gateway. Not because any gateway is magic, but because the gateway is the single point in your architecture where model selection becomes a routing decision instead of a code change. Once that’s true, every other piece of the harness (eval instrumentation, observability, guardrails, PII tokenization, audit logging, rate-limit handling, routing across providers) has a natural home. The gateway is where the harness lives. AOCore is the version of that gateway we built for AOCyber, designed from the start around the assumption that any single provider can become unavailable on any given day, and that the work an engineering team does should accumulate above the model layer, not be re-built every time the model below it changes. If you’re going to staff a team to do this engineering, build the gateway first and the rest grows out of it.

If you’re not building this yourself (and most organizations shouldn’t be) the question is different. The question is whether you can buy a product where someone else has already done this work, where the lessons of the production stack have been engineered through before the software reaches you. AODex is the version of that product we built on top of AOCore: a hundred-plus models behind one interface, persistent memory across sessions, knowledge bases with RAG and citations, a configurable persona library, team workspaces with shared knowledge bases, and underneath all of it the same gateway architecture that makes the model a substitutable parameter. The point of AODex is that the lesson of this essay (the model is the easy part) is already shipped. You get the workflow, the trust scaffolding, the audit posture, and the no-training-on-your-data guarantee as architecture, not as a roadmap item. The model swap happens behind the product. You buy the result.

Either path is defensible. Building the gateway yourself is right for engineering-led organizations that need the harness to be load-bearing infrastructure. Buying AODex is right for everyone else, which is most companies. The path that is not defensible is the one where the conversation stays on “which model” while the engineering work that actually determines outcomes never gets staffed at all. That is the path most of the industry is still on. The next ten essays in this set are about the parts of the production stack where the wins actually live, and how to spot when you’re working on the wrong layer.

The model is the easy part. The work is everywhere else.