The Hardest Part of Agents Is the Plumbing

Sometime in mid-2025, “agents” became the word every AI vendor used to describe what they’d previously called “chatbots that could call APIs.” The marketing was breathless. The demos were impressive. The keynotes promised a world in which AI would not just talk about doing things but actually do them: book travel, file tickets, manage workflows, write and merge pull requests, run businesses. Pick your favorite hyperbolic deck from that year. They all said roughly the same thing.

Two years later, the production data on agents is in, and it tells a less glamorous story. The agents that work are not the ones with the smartest reasoning. They are the ones with the most carefully engineered tool layer. Datadog’s 2026 State of AI Engineering report puts a number on it that I cannot stop thinking about: of all LLM call spans that report an error in production, 60% of those errors are rate limits. Not bad reasoning. Not hallucination. Not tool selection mistakes. Rate limits. The dominant failure mode of agentic systems in 2026 is the agent making an API call that the downstream service refused to serve, and the agent not knowing what to do about it.

The Datadog number is the surface of a deeper pattern. Agent failures in production look like reasoning failures (“the agent got confused, it called the wrong tool, it gave up halfway through”) and are almost always integration failures. The OAuth token expired. The pagination response was malformed. The downstream API changed its schema. The tool description was vague and the model picked the wrong one. The retry didn’t back off correctly. The rate limit fired and the agent did not know how to wait. None of those are model problems. All of them feel like model problems to the user watching the agent fail.

This essay is about why the plumbing is where the difficulty actually lives, what tool design looks like when done well, and the architectural choice that determines whether your agent is going to be a successful infrastructure project or a debugging nightmare.

The strongest version of “the model will handle it”

The case for de-emphasizing plumbing comes from a real and improving trend. It deserves serious engagement.

Frontier models in 2026 are dramatically better at function calling than they were two years ago. They handle tool selection more reliably. They format JSON correctly without coaxing. They reason about which tool to call from a description with surprising fidelity. The progress here is genuine and ongoing. There is a defensible reading of the trend that says: as the model gets better at the tool layer, the engineering investment in tool design becomes unnecessary. The model figures out which tool to call. The model formats the inputs. The model handles errors. The plumbing dissolves into the model’s native capability.

There is also the protocol-standardization argument. The Model Context Protocol (MCP) is real progress. Tool discovery, schema standardization, and unified API gateways are genuinely converging on industry standards in 2026 in a way they weren’t in 2024. The hand-rolled integration code that dominated early agent work is being replaced by registries and protocols. The “plumbing” of 2024 looks legitimately obsolete now.

Both points have weight. The model is better at function calling. The protocols are converging. Agent development is meaningfully less painful than it was eighteen months ago, and a team that invested heavily in custom orchestration scaffolding in 2024 should be ripping a lot of it out today.

The part that doesn’t survive contact with production is the assumption that all the plumbing dissolves. The protocol problems are dissolving; MCP and unified APIs really are eating that layer. But the integration problems are not, because integration problems are not protocol problems. They live in your authentication flow, your downstream API’s quirks, your tenancy model, your rate limit governance, your error recovery semantics. Those problems are yours. They are not standardizable, because they are specific to the systems you are integrating with and the way your business uses them. No protocol upgrade absolves you of them.

A concrete example. In your demo, you store one API key in a .env file. In production, you have 5,000 users who each need to connect their own Salesforce account, their own GitHub, their own Gmail. You now have to build an OAuth client that handles redirects, securely stores refresh tokens per user, refreshes them five minutes before expiry, and serves them to the agent at request time. That is not a protocol problem. MCP does not solve it. The model does not solve it. It is a piece of software you have to write, and most of the production agent failures I have debugged in the last year traced back to some flavor of “we did not think hard enough about token lifecycle.”

More precisely: the “model will handle it” argument is right about the protocol layer, wrong about the integration layer, and the integration layer is where most of the work is.

Where agent failures actually live

These are the categories of failure I see in production agent deployments, in rough order of frequency. The next time someone tells you an agent is “confused,” here’s what to look for first.

Rate limit failures. The single most common production agent failure. The agent calls a downstream API. The API responds with a 429. The agent does not know what to do. It either gives up, retries immediately and gets rate-limited again, or hallucinates an answer based on whatever incomplete data it already has. The fix is exponential backoff with jitter, implemented at the tool layer, never in the agent’s reasoning loop.

OAuth token expiry. The agent has been working fine for an hour. The user’s token expires. The next tool call fails with a 401. The agent interprets the 401 as a tool error and gives up. The fix is automatic token refresh at the integration layer, invisible to the agent.

Pagination failures. The user asks for “all the tickets in this project.” The downstream API returns page 1 of 47. The agent’s tool description does not mention pagination. The agent reasons over the partial data, produces an answer, and is wrong. The fix is to handle pagination at the tool layer: either fetching all pages before returning to the agent, or returning a stable cursor and clear instructions for “call me again with this cursor for more.”

Tool description ambiguity. The agent has 30 tools. Three of them sound similar. The agent picks the wrong one, repeatedly. This is a documentation failure dressed as an agent failure. The fix is writing tool descriptions like you are explaining them to a new hire, with one concrete example of when to call each tool.

Schema drift. The downstream API quietly changed its response format. The agent’s tool wrapper still works, but the data coming back has a new field structure. The agent reasons over malformed data. The fix is contract testing on tool wrappers, run on a schedule, with alerts when downstream schemas change.

Idempotency bugs. The agent tries to create a ticket. The first attempt fails with a network error. The agent retries. The retry succeeds. The original request also succeeded, but the response was lost. The user now has two tickets. The fix is idempotency keys at the tool layer: every action gets a stable client-generated ID that the downstream service can use to detect duplicates.

Cost runaway loops. The agent enters a tool-calling loop. Each iteration costs a dollar. The user has not set a per-request budget. The agent racks up a four-figure bill in ten minutes before someone notices. The fix is per-request cost ceilings and maximum iteration counts, enforced at the orchestration layer.

Notice something about that list. Not one of those failures is improved by upgrading the model. All of them are improved by careful engineering of the layer between the agent and the world. The model is the part of the system that decides which tool to call and what arguments to pass. Everything else (the auth, the retries, the pagination, the schema validation, the budget enforcement) is software that you write, and the quality of that software determines whether the agent succeeds in production.

A tool design rubric

The rubric I now run against any tool an agent will be allowed to call has six binary questions. Score yes/no on each. Anything below four out of six and your agent’s failures will be your code, not your model.

One. Does the tool name unambiguously communicate intent? “create_ticket” is good. “do_action” is bad. The agent reads the name as part of selection. Ambiguous names produce wrong selections.

Two. Does the description include one concrete example of when to call it? The model uses the description to decide between tools. A description that says “Creates a ticket” is worse than one that says “Creates a support ticket. Call this when a customer reports an issue, NOT for internal task tracking; for that, use create_task.”

Three. Do errors return structured information that tells the agent what to do next? “Rate limited, retry in 30 seconds” is useful. “Error 429” is not. The agent’s next action is determined by what the error says; vague errors produce vague behaviors.

Four. Is pagination handled by the tool, not by the agent? If the underlying API paginates, the wrapper should either fetch all pages transparently (for small result sets) or expose a paginated interface with clear cursors and instructions. The agent should not be reasoning about page tokens.

Five. Is auth refresh invisible to the agent? Token lifecycle (refresh, retry on 401, reauth flow on expired refresh tokens) happens at the integration layer. The agent never sees a 401. If it does, you’ve designed your auth wrong.

Six. Is the tool either idempotent, or does it return a stable handle? An idempotent tool can be retried safely. A non-idempotent tool needs to return a handle (a ticket ID, a transaction ID) that the agent can use to check whether a prior attempt succeeded before retrying.

Six out of six is production-ready. Below four out of six and the agent’s failures are going to be debug-hours that read like reasoning failures and are actually plumbing failures.

On native SDKs and the death of LangChain

A pattern related to this argument deserves to be said out loud. The agent framework landscape in 2026 is going through a quiet correction. LangChain, which solved real problems in 2023 around abstracting model providers and chaining calls, has become abstraction overhead that most production teams are now stripping out. Native SDKs from each provider have caught up on the surface area that LangChain originally added value to. Frontier models handle function calling, memory management, and multi-step reasoning natively, often better than the framework’s abstractions did.

The 2026 production consensus, from teams I respect, is roughly this. Go native. Use the provider SDK directly. Reserve LangChain (specifically LangGraph) for one specific use case: complex cyclical workflows with state management requirements that the native SDK doesn’t handle. For everything else (standard agent patterns, tool loops, conversational interfaces) the native SDK is faster to develop against, simpler to debug, and produces code you’ll actually understand six months later.

The pattern is the same one this essay keeps surfacing: scaffolding that solved 2023 problems is being absorbed by the model layer or by the native SDK, and the work that isn’t getting absorbed is the integration plumbing that lives between your agent and your downstream systems. That layer doesn’t have a framework that fixes it for you. It is software you have to write.

Where this lives architecturally

Tool plumbing (auth lifecycle, retry policies, rate limit governance, schema validation, idempotency, cost enforcement) is middleware. It should not live in your agent’s reasoning loop, and it should not live duplicated in every tool wrapper. It should live in one architectural layer, with consistent policies, observable behavior, and the ability to be updated without touching the agent code.

The natural home for that middleware is the gateway. Not because gateways are the only place to put it (you can build it yourself in a service mesh, an API gateway, a hand-rolled middleware layer) but because the gateway is the architectural point where the agent meets the outside world, and the gateway is the layer where these policies belong. AOCore is the version of this we built for AOCyber, with tool-layer middleware handling auth refresh, exponential backoff on rate limits, per-request cost ceilings, pagination wrappers, and structured error returns. The agent sees clean tool calls. The plumbing lives where plumbing belongs, in infrastructure.

For organizations that are not going to staff an integration engineering team to build that middleware from scratch, AODex is the plumbing-solved version delivered as a product. Web search and fetch, knowledge-base retrieval with citations, document upload, persistent memory across sessions, and a configurable persona library: each of those is a tool integration that someone would otherwise have to wire up, monitor, and maintain. Rate-limit handling, auth lifecycle, retry logic, and pagination all live beneath the workspace. Users consume the finished product; they never see the integration debugging that would otherwise consume an engineering quarter.

The promise of agents in 2024 was that the model would do the hard part and you would write a thin wrapper. The reality in 2026 is that the model does the easy part (selecting tools, formatting calls, reasoning over results) and the wrapper is the hard part. The teams that have internalized this are shipping agents that work. The teams that are still upgrading the model in search of better agent reliability are debugging rate limits at midnight, blaming the model, and not getting better.

The hardest part of agents is the plumbing. Build the plumbing first, and the model becomes the easy part again.