Pilots Lie. Production Tells the Truth.

Gartner’s working prediction is that 60% of AI projects initiated in 2024 and 2025 will be abandoned by the end of this year. LangChain’s most recent State of Agents report puts a more specific number on the cliff: 57% of organizations have AI agents in production, and 32% of those cite quality as their top barrier, with the dominant root cause traced to context management, not model capability. Datadog’s 2026 production data shows that the systems that survive the cliff and the systems that don’t are running comparable models. The model is the same. The production experience is not.

Think about this gap as the second pilot. Every enterprise AI project has two pilots, whether the org admits it or not. The first one is the official pilot: the one in the deck, the one the executive sponsor signs off on, the one with curated users and controlled inputs and a friendly outcome. The second pilot is the one that happens to your production deployment in the first ninety days, against real users and real data and real edge cases. The official pilot succeeds, and then the second pilot determines whether the project ships or fails.

Most enterprise AI projects pass the first pilot and fail the second one. The post-mortems blame the model, or the team, or the change management. The actual cause is almost always the same: the pilot was structurally unable to surface the failure modes that production then surfaced immediately, and the organization had no plan for the gap.

This essay is the closing of this set, and it is the one that ties everything else together. The model is the easy part because the model is the unchanged part between pilot and production. Everything in the previous ten essays (retrieval, trust UX, evals, the human baselines you’re measuring against, calibration, data hygiene, plumbing, staffing) is what makes the pilot-to-production transition survivable. The teams that did that engineering survive. The teams that put their faith in the model fail in production on issues the model could not have fixed.

What follows: what actually changes between pilot and production, what to plan for before the cliff hits, and the architectural posture that turns the second pilot from “first real failure” into “actual deployment.”

The strongest version of “pilots predict production”

The case for treating pilots as predictive deserves a fair shot.

The argument goes like this. Pilots exist to de-risk. We pick a representative slice of users, give them a representative set of tasks, and measure how the system performs. If the pilot is designed well, it predicts production with reasonable fidelity. The point of the pilot is precisely to avoid surprises in production. A pilot that fails to predict production is a badly-designed pilot, not a fundamental property of pilots.

There is also a structural version. Engineering teams do not have infinite time. The pilot is a checkpoint. It is, by definition, a smaller-scale version of the production system. If we cannot trust the pilot to give us signal, we have no way to make the deploy/no-deploy decision short of putting everything in production and hoping. That is not a defensible strategy.

Both have weight. Pilots should be predictive. A well-designed pilot is meaningfully more useful than no checkpoint at all. The “pilot is a deception” framing is too cynical; pilots have real value, and the teams that skip them entirely tend to do worse.

The part that doesn’t survive contact with production AI specifically is the assumption that the structure of an AI pilot is similar enough to the structure of production that pilot results transfer. They aren’t. The transfer rate from pilot to production in AI is dramatically worse than the transfer rate in traditional software, and the reasons are specific to how AI systems interact with real-world variability. Traditional software pilots fail to predict load patterns, performance issues, occasionally novel bugs. AI pilots fail to predict what the system does, because the system’s behavior is a function of inputs the pilot did not see.

Bottom line: the argument is right in principle and wrong in practice for the current generation of AI systems. Pilots could predict production if they were structured to surface the failure modes production will surface. Almost none are. The fix is not to skip the pilot. The fix is to know what changes between pilot and production, and to design the pilot (or the post-pilot transition) to cover the gap.

What actually changes

The gap, item by item, drawn from production deployments I have seen succeed and fail. Each item is a thing that is different between your pilot and your production. Each item is a thing that your pilot was structurally unable to test for. Each item kills deployments when the organization didn’t plan for it.

User diversity. The pilot users were chosen because they were friendly, available, technically capable, and motivated to make the project succeed. They are the easiest 5% of your eventual user population. Production users include the remaining 95%, who are skeptical, time-pressured, less technically fluent, and motivated by their own job rather than by the success of your AI project. The interactions look different. The questions are worded differently. The patience for friction is lower.

Query distribution. The pilot users asked the questions the team had imagined they would ask. Production users ask the questions the team did not imagine. The long tail is where most of the interesting failures live. Pilot test sets are bounded; production query distributions are not. The system that handled the pilot’s 50 question categories often has no answer for the 51st, and the 51st is what a real user asks.

Data freshness and variability. The pilot ran against a snapshot. Production runs against a moving target. Documents are added and deprecated daily. Some upstream system the retriever depends on changes its schema. A regulatory update lands and three documents in the corpus are now outdated. None of this happens during the pilot week, because the pilot week is short. All of it happens in production, on a schedule the team did not anticipate.

Integration breadth. The pilot called three tools. Production calls thirty. The integrations that worked individually start interacting in ways the team did not test for: tool A’s output is tool B’s input, but tool A occasionally returns null and tool B silently produces wrong answers when it gets null input. The pilot didn’t catch it because the pilot used tool A and tool B separately.

Permission complexity. The pilot ran as a single user with full access. Production has roles, tenants, departments, and ACLs. The retrieval system that returned the right document in pilot now has to filter by permission, and the filter logic interacts with the ranking logic in non-obvious ways. Sometimes a user has permission to see a document but not to act on its contents. The pilot has no concept of this distinction; production lives or dies by it.

Volume. The pilot ran at 100 calls per day. Production runs at 100,000. Rate limits fire. Retry storms cascade. Concurrency bugs surface that single-user testing could not have produced. Costs scale non-linearly because the inefficient query the pilot did once is now happening a hundred times an hour.

Failure visibility. During the pilot, every failure was watched. The team reviewed every wrong answer. The eval data was hand-curated. In production, failures arrive faster than humans can review them, and the team’s relationship with the system shifts from “we know what it does” to “we hope we’d notice if something went wrong.” Without observability, the team is operating blind. With observability, the team is at least operating with data.

Prompt injection surface. The pilot users did not try to break the system. Production users include adversarial actors, curious tinkerers, and people who copy-pasted something from a forum without realizing it was a prompt-injection attack. The system that worked when nobody was attacking it has a different failure profile when people are.

Seven changes. Most pilots are structurally unable to test for any of them. The teams that survive the cliff are the teams that know the changes are coming and design the post-pilot phase to surface them quickly, not the teams that hope the pilot results transfer cleanly.

The pilot-to-production diff checklist

Before teams put an AI system into broad deployment, I now ask them to enumerate eight items. Score each: do we have a plan for this, beyond hoping?

One. User diversity plan. How will the rollout sequence users from most-friendly to least-friendly, with explicit checkpoints where the system’s behavior against each new user segment is measured before expanding? Most organizations skip this and go straight from pilot users to full deployment. The fix is staged rollout with eval gates at each stage.

Two. Query distribution monitoring. What is the plan for noticing when production users start asking question types the pilot didn’t cover? At a minimum, this requires categorizing incoming queries, flagging the ones that don’t fit existing categories, and routing them to human review.

Three. Data freshness alerts. What process will detect when a document in the corpus has gone stale? Who owns updating it? What does the system do when a query asks about a document that has been flagged as stale?

Four. Integration regression tests. Which tool combinations are exercised? When a downstream API changes, who catches it? The answer “we’ll notice when something breaks” is not a plan; it’s the absence of one.

Five. Permission and tenancy testing. What is the test plan for permission logic under realistic role configurations? This is the failure mode most likely to surface as a security incident rather than a quality issue, and it usually gets caught by the wrong team (the security team, after the fact).

Six. Volume and cost ceiling. What are the per-user, per-team, and per-tenant cost ceilings? What does the system do when it hits them? Cost runaway in production is a board-level event; the time to plan for it is before deployment.

Seven. Observability investment. How will the team know what the system did, in what order, with what result, after the fact? Most teams answer this with “we have logs.” The follow-up question is whether the logs can be queried to answer the diagnostic questions in the previous essays. Usually they can’t.

Eight. Adversarial input handling. What is the prompt injection test suite? Who runs it on a schedule? What is the incident playbook when an injection succeeds? The systems that survive this question in production have answered it before launch.

Eight items. Most production AI deployments answer “we’ll figure it out” to most of them. The teams that have written plans for most of these items are the teams that successfully transition pilot to production. The teams that haven’t are running an unstructured experiment on their users.

What this requires architecturally

The pilot-to-production transition is the moment when the engineering investment from every previous essay either pays off or doesn’t. The retrieval pipeline either holds up under variability or doesn’t. The eval harness either catches the regressions or doesn’t. The trust scaffolding either contains the confident wrong answers or doesn’t. The plumbing either handles the rate limits and OAuth failures or doesn’t. The calibration either lets users navigate real outputs or doesn’t. The data hygiene either survives corpus growth or doesn’t.

Everything I have argued for across the previous ten essays in this set is, in the end, infrastructure for surviving the second pilot. The model is not what changes between pilot and production. What changes is the load on every other layer of the stack (retrieval, trust UX, evals, the human baselines you’re measuring against, calibration, data hygiene, plumbing, staffing) and the layers that were not engineered for production load are the layers that fail.

The architectural lesson is the same one I have been making in every essay, in different forms. The gateway is where the engineering investment accumulates, because the gateway is where every call passes through and therefore where observability, eval instrumentation, routing, calibration, audit, and rate-limit governance can live in one place. AOCore is the version of that gateway we built for AOCyber, because we kept watching organizations integrate AI directly into their applications and then discover, at the second pilot, that they had no place to put any of the engineering the next phase required. The gateway is not a magic bullet. It is the architectural position from which the rest of the engineering work becomes possible.

For organizations that aren’t going to staff a team to engineer this themselves, the second pilot is somebody else’s problem. AODex is the product version where the second pilot has already happened on someone else’s nickel. The diversity of users, the variability of queries, the integration breadth, the rate-limit handling, the prompt injection exposure, the audit posture: every item on the pilot-to-production diff checklist has been engineered through before the buying organization ever logs in. The buyer’s “pilot” becomes onboarding, not validation. They are not running an experiment to see whether the system survives production; they are deploying a product that has already cleared that bar.

The deepest pattern from three years of watching enterprise AI deployments succeed and fail: the projects that fail are the ones that treat the pilot as predictive of production. The projects that succeed are the ones that treat the pilot as a separate problem (necessary, useful, but structurally different) and that plan explicitly for the cliff between them. The work that determines whether the second pilot succeeds is exactly the work this entire set of essays has argued for: retrieval, evals, trust, human baselines, calibration, data hygiene, plumbing, staffing.

That work is not the model. The model is the easy part. The work everywhere else is what determines whether the AI product you ship in 2026 is still running in 2028 or whether it shows up on Gartner’s abandonment statistic.

Build the work. Then ship the pilot. Then survive the production deployment.

The model will be fine. The rest is up to you.