Evals Are the Product Spec

Hamel Husain, after spending two years consulting on dozens of production LLM deployments, has been consistent on one point that I think is the single most important thing anyone has written about AI product development in the last three years. Unsuccessful AI products, he says, almost always share a single root cause: the failure to build a robust evaluation system. Everything else (prompts, retrieval, model selection, workflow design) is downstream of whether you can measure quality. If you can measure quality, every other problem becomes tractable. If you cannot, every other problem is unsolvable, because you have no way to know whether your fixes are fixes or whether they are regressions in disguise.

I believe this in a way I believe almost nothing else in this space. Not in the sense that “evals are important” (everyone agrees evals are important and then ships without them) but in the sense that the presence or absence of evals is the strongest single predictor I have ever seen of which AI products will still exist in two years. Stronger than model choice. Stronger than team size. Stronger than funding. The teams that have evals improve in public; the teams that don’t, regress in private. Over a 24-month horizon, the gap between those two trajectories is enormous, and it compounds.

The case for why, the maturity ladder I now use to evaluate AI products, and the boring architectural point that makes the whole thing runnable: those are the three sections that follow.

The strongest version of “evals come later”

Most teams that ship AI products without evals are not stupid. They are responding to a real argument. The argument deserves a fair shot before I dismantle it.

The argument goes like this. Evals are infrastructure investment. Infrastructure investment competes with feature investment. In an early-stage AI product, the team’s job is to find product-market fit, which means iterating fast on features and learning from users. Building an eval harness before you have a product is premature optimization. You should ship, watch users, fix what they complain about, and add evals once you know what to measure.

This is not a stupid view. It is the view that built most successful early-stage software companies in the 2010s. Test-driven development was supposed to be how everyone shipped, and then a generation of startups ignored it, shipped without tests, found product-market fit, and added tests later. The history of that decade rewards “ship without tests, fix forward.” Why would AI products be different?

Concede something real: the “ship now, eval later” view is correct for the first month of an AI product. In the earliest exploratory phase, when you don’t know what the system is supposed to do well, designing an eval is genuinely premature. You’d be measuring against the wrong target, and the act of measuring would lock you into the wrong target. So the literal first build, the demo for the executive team, the initial pilot with a friendly customer: yes, fine, ship those without evals. I have done it. I am not arguing that day-one evals are mandatory.

The argument breaks down the moment the product moves from exploration to deployment. The reason is specific to AI products and was not true of the 2010s SaaS world: an AI product that ships without evals does not just lack a regression net. It actively regresses without anyone noticing. Models change behavior on provider-side updates. Retrieval quality drifts as the corpus grows. Prompts that were correct for last quarter’s data are subtly wrong for this quarter’s. User behavior changes the distribution of inputs, which changes the distribution of outputs, which changes where the failure modes cluster. None of these would have been issues in a traditional SaaS product. All of them are silent quality regressions in an AI product. Without evals, you don’t even know they’re happening until a customer complains, and by then the regression has been operating against you for weeks or months.

The honest version: the “evals come later” argument is right for week one and catastrophically wrong by month six, and most teams that adopt it never schedule the upgrade.

The eval maturity ladder

When I’m asked to assess an AI product team, I place them on a five-rung ladder. The leaps between rungs are non-linear; rung one to rung two is the biggest single quality improvement any team can make, and most teams never make it.

Rung zero: no evals. Vibes. The team ships changes based on whether the engineers think the output looks better. Quality is measured by tickets filed, which is to say, by which failures customers cared enough to complain about. This rung is where most AI products live today. Including, I am sorry to report, most enterprise AI products with eight-figure budgets behind them. The dirty secret of the industry is that organizations with hundred-person AI teams routinely ship at rung zero.

Rung one: manual spot checks. The team has a habit, more or less consistent, of eyeballing twenty or thirty outputs after each meaningful change. Usually one engineer does it. Usually they catch the worst regressions before deploy. Sometimes they miss things, especially the slow drifts. This rung is better than rung zero because at least someone is looking, but it does not survive headcount turnover and it does not scale past the smallest products. It also produces no historical record: last quarter’s eyeball check is gone, so you can’t tell whether you’ve gotten better or worse over time.

Rung two: binary LLM-as-judge against a fixed test set. The team has a set of input-output pairs they care about (maybe a hundred, maybe a thousand) paired with a judge prompt that scores each output as pass or fail. The judge is itself an LLM. The whole thing runs in CI on every change. Regressions block deploy. This is the leap that separates serious AI products from hobbyist ones, and it is, by my count, the single highest-ROI engineering investment most AI teams can make. The leap from rung one to rung two changes how the team works: every change becomes measured, every regression becomes visible, the rate of improvement starts to compound. Hamel’s research suggests teams operating at this rung end up spending 60-80% of their development time on evaluation and error analysis. That sounds like a lot until you realize that the teams not doing it are spending most of their time responding to user complaints, which is the same work with a six-week delay and worse data.

Rung three: calibrated judges. The judges from rung two are themselves evaluated against human-labeled data: calibrated against true positive rate and true negative rate, debiased, validated to agree with domain experts at a measurable rate. This step is the one that turns evals from “useful signal” into “trustworthy ground truth.” Without calibration, the judge can be wrong in systematic ways and you’d never know: the judge is happy with answers that experts would call bad, or the judge rejects answers that are actually fine. With calibration, the eval suite is something you can ship a release on. Most teams skip this rung because it requires labeled data from actual domain experts, which costs money, time, and a specific kind of organizational discipline. The teams that do it run for years on the same eval harness because the harness has earned the right to be trusted.

Rung four: eval-driven development. Every prompt change, every retrieval tuning, every model swap, every workflow tweak runs the full eval suite as part of the change set. Quality is a property of the diff, the same way correctness is a property of a diff in a tested codebase. Regressions block release. Improvements get shipped with quantitative claims. The team’s velocity on quality improvements goes up dramatically because they can iterate without fear: try a change, see whether the evals improved, ship it if they did, revert if they didn’t. The decision cycle on a quality experiment shrinks from weeks to hours.

Rung five: continuous production evals. Live traffic is sampled, scored against the eval rubric, and fed back into the gold standard set. The eval harness improves itself over time, catching new failure modes as users find them and adding them to the regression suite. Drift is detected automatically. Model swaps can be A/B tested against historical traffic before any user sees them. This is the maturity level the leading AI product organizations operate at, and very few enterprises ever reach it.

Most teams that take eval work seriously stall somewhere between rung two and rung three. That’s already a different universe than rung zero. The leap from “no evals” to “binary judge on a fixed test set with CI integration” is the leap that determines whether the product exists in two years.

What binary judges, and a benevolent dictator, actually mean

Two specific points from Hamel’s writing that I think don’t get enough airtime, because they save teams from the most common eval mistakes.

The first is that binary judges beat Likert scales, almost universally. A judge that scores outputs on a 1-5 scale sounds more sophisticated than a pass-fail judge, but it produces worse data. The reasons compound: the difference between adjacent scale points is subjective and inconsistent across annotators, statistical significance requires larger samples to detect, and annotators default to middle values to avoid making hard decisions. A binary judge (pass or fail, with the criteria written down) forces clearer thinking and produces cleaner data. The temptation to make evals “more nuanced” by adding scale gradations is one of the most reliable ways to make evals less useful.

The second is the benevolent dictator pattern. One person, ideally a domain expert who understands the users, owns the quality decision. Not a committee. Not the team’s collective intuition. One person, with the authority to say “this output is correct” or “this output is wrong” and the responsibility to write down the criteria they used. The pattern survives turnover because the criteria are written down; it survives scale because the criteria can be encoded into the judge. But it requires that someone, on day one, be willing to be the person whose taste defines what the product is for. Teams that won’t appoint that person produce eval suites that drift, contradict themselves, and slowly lose alignment with the users they’re supposed to represent. There is no committee replacement for a single owner of quality criteria.

The benevolent dictator does not have to stay involved forever. The first version of the eval is built around their judgment; the team takes over the operations; the dictator periodically audits drift. But the origin point of the quality criteria has to be a person whose taste the team trusts. Without that, the eval is a popularity contest.

What this requires architecturally

Evals are not a feature you bolt on at month six. They are an infrastructure layer that determines what every other layer of the AI product can do. To build evals well, you need to capture every model call, every retrieved chunk, every assembled prompt, every completion, and every user reaction. You need to be able to replay historical traffic against new versions of the pipeline. You need to be able to slice failures by user segment, by query type, by document corpus, by model. You need to be able to ship a model swap, run the eval suite against the new model on yesterday’s traffic, and decide whether to roll it out before any user hits the new model.

None of that is possible without a layer in your architecture where every call is captured. Build that layer first, and evals are something you turn on. Skip it, and you’ll be retrofitting eval infrastructure into a production system three years from now, by which point the silent regressions have already done their damage.

This is the case for the gateway one more time. AOCore is the version of this we built: the gateway captures every call, the eval suite is a query against that data, model swaps run against the historical traffic before they hit a single user, and the benevolent dictator’s gold standard is built from production traces tagged by an actual domain expert. The teams building AI products on top of AOCore get to rung two on day one because the data the eval suite needs is already there. Most teams reach rung two by spending six months building the same infrastructure as a side project; the gateway shortens that to a configuration step.

For the organizations that aren’t going to staff a quality-engineering team to learn this discipline from scratch, AODex is what rung-two-and-up looks like as a shipped product. The platform’s behavior has been engineered against quality criteria upstream of the buyer; they don’t need to build their own eval harness to validate that source citations are accurate, that abstention works when it should, that PII is tokenized before reaching providers, or that the trust scaffolding from the previous essay actually delivers what it claims. The eval lift this essay argues for is real and substantial, and AODex is the version where someone else has already done the lift before the customer logs in.

If you take only one thing from this essay, let it be this: an AI product without evals is not a product that lacks a regression suite. It is a product that is silently regressing right now, against you, and will keep regressing until you either build the eval harness or stop deploying it. There is no third option, and the longer you wait, the more debt you accumulate in a layer that compounds in cost roughly as fast as the model layer compounds in capability.

Evals are the product spec. Without them, you have a product that does whatever the model felt like doing on the day you last looked.