The Design→Test Loop Finishes Closable Problems. It Can't Touch Open Ones.

In July 2024, DeepMind set its AlphaProof and AlphaGeometry 2 systems on that year’s six International Mathematical Olympiad problems, and they scored 28 points out of 42, one point below the gold-medal threshold, a mark that by score alone would have placed them among the strongest competitors in the world. They did not sit the exam the way a human does: the problems were first translated by hand into formal mathematical language, and the systems took anywhere from minutes to three days where a human gets two sessions of four and a half hours. Hold those caveats, though, because the mechanism is the point. AlphaProof’s proofs were written in Lean, a formal proof assistant, and machine-checked; the geometry problem went to AlphaGeometry 2, which produces its own verifiable symbolic proofs rather than Lean. Neither system had to persuade a panel of judges. Each generated candidates and threw them against a verifier that returned, for every one, a total and trustworthy verdict: valid, or not.

That verifier is the entire story. AlphaProof works not because mathematics is easy (it is nearly the hardest thing humans do on purpose) but because a formal proof is checkable. The success predicate exists, it is cheap to evaluate, and it cannot be argued with. Point a design→test loop at a problem with a predicate like that and it will grind toward an answer, because every candidate it generates gets a real grade. The question that decides whether an AI loop can own a problem end to end is not how hard the problem is. It is whether the problem closes.

What follows: what makes a problem closable in the precise sense that lets a loop finish it, why a specific class of problems stays open no matter how good the model gets, and how to tell which of yours is which before you staff it.

The strongest version of “a good enough loop closes anything”

The optimistic view deserves a real hearing, because its track record is genuinely intimidating. The set of problems we thought were open keeps turning out to be closable once someone builds the predicate. Protein structure looked like a decades-open biology problem until AlphaFold reframed it around a structural-accuracy metric it could optimize against, and in 2020 it more or less solved the CASP benchmark that had resisted the field for years. Competitive programming, natural-language translation, image classification: each looked open, each acquired a benchmark and a scoring function, and each fell to a loop that could finally tell a good answer from a bad one. So the strong claim is: “open” is usually just a euphemism for “nobody has written the predicate yet,” and the frontier of closable problems is expanding fast enough that betting on any particular problem staying open is a loser’s game.

Concede all of it, because it is largely correct. Most problems people wave off as too fuzzy for AI are not open in any deep sense; they are closable problems missing a scorecard, and the moment someone builds the scorecard, the loop eats them. Underestimating that has burned a lot of skeptics.

The part that doesn’t survive is the word anything. There is a class of problems that stays open not because the predicate is hard to build but because it cannot exist, where the input space is extended by an adversary after you deploy, or where “correct” is a moving social judgment with no formal ground truth even in principle. The closable frontier is expanding, and it will never reach these, because what keeps them open is not a missing scorecard; it is a second player who gets a move after yours.

The category error

The mistake underneath most bad AI staffing decisions is treating “hard” and “open” as the same word. They are orthogonal. Sorting a billion records under a latency budget is hard and completely closable. Deciding whether a single message counts as harassment is trivial to state and hopelessly open. Difficulty is about how much work a solution takes; closability is about whether a solution can be graded. Allocate your loops by difficulty and you will confidently automate the open problems (the ones where the loop produces fluent, plausible, unverifiable output) while treating the closable ones as too mundane to bother formalizing. That is exactly backwards.

Here is the test, stated as three questions. Can you enumerate or fairly sample the space of inputs the system will face? Can you compute a success verdict for a given output cheaply, and do you trust that verdict? And (the one everyone forgets) does any adversary get to extend the input space after you ship? A problem is closable when the first two are yes and the third is no. When all three line up, the design→test loop terminates in the strong sense: it can run unattended, because every candidate it generates receives a real grade, and “better” is a direction the machine can actually climb.

Run the test and the sort separates cleanly. Theorem proving: closable, and AlphaProof proves it. A payments ledger that must never let balances disagree: closable, because the invariant is a computable predicate over states. Compiler correctness: closable, via differential testing and formal semantics. Crash-hardening a parser: closable enough, because a crash is an oracle: the program either fell over on that input or it did not.

Now the other pile. Prompt injection (the attack Simon Willison named in September 2022, and which years of frontier progress later still has no general solution) is open by construction: the defender publishes a system, and the attacker’s entire job is to invent an input the defender’s predicate never anticipated. Content moderation is open because “acceptable” is a social judgment that shifts by context, culture, and year, with no formal ground truth to check against. Fraud, spam, and market positioning are open for the same structural reason: there is an intelligent opponent, or a shifting society, supplying new inputs after you commit. No amount of model capability closes a problem whose input space is being actively extended by someone trying to beat you.

What this means in practice

The payoff is not “don’t use AI on open problems.” It is understanding what AI is on an open problem, which is completely different from what it is on a closable one. On a closable problem, the loop is a solver: you define the predicate, hand it over, and walk away, because it terminates. On an open problem, the loop never terminates, and that is not a failure of the model, it is a property of the problem. There, AI is an amplifier inside an ongoing game. You are not going to finish prompt-injection defense the way AlphaProof finishes a theorem. You are going to run the loop forever, as a permanent red team, because the other player is also running theirs.

This distinction quietly determines your org chart. Closable problems should be pushed at the loop aggressively and staffed thin; the human writes the predicate once and supervises the ratchet. Open problems should be staffed as games: someone owns the ongoing contest, expects the goalposts to move, and measures success as “we are still ahead this quarter,” never as “done.” Teams that confuse the two either under-automate the closable work (paying humans to do what a verified loop would do for tokens) or, far more dangerously, declare victory on an open problem and ship a defense that an adversary strolls through the following week.

What this requires architecturally

You can see the closable/open line drawn straight through security products, because security is the open problem par excellence. The reason a gateway like AOSentry (the governance and guardrail layer inside AOCore) exists as a standing, always-on inspection layer rather than a fix you install once is that its core problems do not close. Jailbreak and prompt-injection screening is not a feature you ship and retire; it is a game you staff, which is why it lives at the gateway and runs on every request in and every response out, indefinitely. The company’s own public framing of the attack surface says it plainly (prompt injection is the SQL injection of AI) and the operational consequence is the tell: a model’s safety training has to defend against every possible attack, while an attacker needs only one that works. That asymmetry is the signature of an open problem, and the only sane architecture for an open problem is a permanent amplifier, not a one-time solver.

The mirror image holds on the closable side. Wherever a business process genuinely closes (a reconciliation with a checkable invariant, a document transformation with a verifiable output) the right move is to hand it to a loop under governance and let it run, spending human attention only on the predicate and the audit. Most organizations have both kinds of work tangled together and staffed identically. Pulling them apart is most of the value.

What to do first

Before you point an AI loop at anything, run the three questions out loud. Can you enumerate the inputs? Can you compute a verdict you trust? Does an adversary get a move after you ship? Three answers of the right shape (yes, yes, no) and you should be handing that problem to a loop today and reallocating the humans who used to do it. A “yes” on the third question means you are not looking at a problem you can solve; you are looking at a game you have to play, and you should staff it like one: permanently, adversarially, with no expectation of a finish line.

The reason this matters more every quarter is that the closable pile is growing and the open pile is not shrinking. As models improve, more of your work becomes loop-finishable, which means the human share concentrates harder onto the problems that never close. So learn to tell them apart now, because that sorting is about to become the main thing you do. The loop finishes what closes. Everything that stays open stays yours.