Ask for Attacks, Not Tests

In late 2024, a team at Google published a result that should have unsettled every engineer who has ever written a test suite. An AI agent they called Big Sleep had found a real, exploitable security vulnerability (a stack buffer underflow) in SQLite, one of the most widely deployed and heavily tested pieces of software on earth. SQLite is fuzzed continuously and its test suite is legendary in the field. And the agent found a flaw that all of that testing had missed (including a purpose-built fuzzing run of more than 150 CPU-hours that failed to rediscover it) not by generating more test cases, but by reasoning toward a specific kind of weakness the way an attacker would. By Google’s own careful phrasing, it was “the first public example of an AI agent finding a previously unknown exploitable memory-safety issue in widely used real-world software.”

The result matters less for what it says about SQLite than for what it exposes about two very different capabilities we lazily file under the same word. When people say “AI can write tests,” they are describing one thing. What Big Sleep did was a completely different thing that happens to also produce a test at the end. One samples the space of plausible, typical checks. The other searches, adversarially, for the input that breaks a specific claim. Test generation is a completion task that samples the happy path; vulnerability finding is adversarial search with a concrete target and a real success signal, and the prompt is the whole difference between coverage theater and a weapon.

What follows: why “write me some tests” reliably produces the least valuable tests, why the same model becomes dangerous the instant you change what you ask it for, and how to convert that capability into a discipline instead of a party trick.

The strongest version of “generated tests still add coverage”

The defense of AI-generated tests is straightforward and partly correct. A model can paper a codebase in unit tests in minutes: it reads a function, infers its intended behavior, and emits a battery of cases that exercise it. Line coverage climbs. Branch coverage climbs. Some of those tests will, in fact, catch a real regression later when someone changes the function and forgets a case. Compared to the very common baseline of no tests at all, a wall of generated tests is a genuine improvement, and it is free. If your bar is “more coverage than we had yesterday,” generated tests clear it easily, and I won’t pretend that’s worth nothing.

Here is where it expires. Coverage measures which lines ran, not which invariants hold, and generated tests overwhelmingly assert that the code does what it currently does, including, faithfully, whatever it currently does wrong. A completion model asked for tests samples the distribution of typical tests for code like this, which means it encodes the present behavior as the definition of correct and then guards it. A happy-path test suite doesn’t challenge the code; it photographs it, and a photograph of a bug is not a test that catches the bug, it’s a test that protects it.

The category error

The confusion is between a test and an attack, and they point in opposite directions. A test, as the word is usually used and as a model usually generates it, asserts expected behavior: given this input, expect this output. An attack starts from a stated invariant (a property that must hold no matter what) and searches for any input that violates it. The first confirms; the second refutes. Ask an LLM to “write tests” and you get confirmation, because completion is a confirming operation: it produces the tests that typically accompany code like yours, and typical tests walk the path the typical user walks. The dangerous defects do not live on that path. They live where an adversary pushes (the malformed input, the boundary, the interleaving, the value nobody expected) which is exactly the region a happy-path sampler never visits.

Change the request and the same model changes character. Do not ask “write tests for this function.” State the invariant (“this parser must never read out of bounds,” “this endpoint must never return one tenant’s data to another,” “balances must never go negative”) and ask the model to find an input that breaks it. Now you are not sampling typical behavior; you are running adversarial search with a concrete target and a real success signal, which is the mode in which frontier models are startlingly good, because “did the invariant break?” is a crisp, checkable predicate the search can climb. Big Sleep is the vivid version of this, but the effect shows up in ordinary work: give a model an invariant and a mandate to violate it and it will find edge cases your happy-path suite structurally could not, in the same codebase, in the same afternoon.

What this means in practice

The move is to stop asking for tests and start asking for attacks, then convert every attack that lands into a regression test. That inversion (invariant first, attack second, regression test third) flips the model from a coverage-theater generator into a red-teamer whose successes automatically harden the system. The tests you keep are no longer photographs of current behavior; they are tombstones for specific ways the code was shown to be breakable, which is the only kind of test that was ever worth much.

There is a trap inside this that the next post is entirely about, so here is only the warning: when a model finds an attack and you ask it to write the regression test, it will cheerfully write a test for the exact input it found: the specific payload, the specific byte, the specific tenant ID. That test is nearly worthless, because it guards against one instance of a whole class of failures. The value is in generalizing the landed attack to its class: unsanitized input reaching the query builder, not '; DROP TABLE; any cross-tenant read, not tenant 4171 reading tenant 9. The model will not do that generalization unless a human makes it, and forcing that generalization is exactly the adversarial, constraint-owning work this series keeps landing on.

What this requires architecturally

This is also, precisely, what a security product is: adversarial search against stated invariants, productized and run continuously. A guardrail like AOSentry’s jailbreak and prompt-injection screening (part of the governance layer inside AOCore) is not a pile of happy-path tests hoping the model behaves. It is a standing attack posture against a stated invariant: the model must not be manipulated into doing the thing it must never do, and every request is probed against that invariant before it reaches the model, every response before it reaches the user. The company’s own public line (that prompt injection is the SQL injection of AI) is a claim about invariants, and the architecture is the operationalized version of “ask for attacks”: build the machinery that keeps trying to break the property, forever, and turns each successful break into a control.

It’s worth being honest about the same trap on the build side. A harness like DevFlow can auto-generate tests “matching your existing patterns and coverage,” and that convenience is exactly the happy-path risk named above: patterns and coverage are a description of what the code already does. The right way to use it is to pair generation with adversarial prompts against explicit invariants, so the suite you accumulate is a record of attacks survived rather than a mirror of behavior shipped.

What to do

Change the words. Ban “write tests for this” as the default request and replace it with “here is the invariant; find an input that breaks it.” Do it for the properties that would actually hurt you if they failed (data isolation, monetary correctness, memory safety, authorization) and treat every attack that lands as a finding, not a nuisance. Convert each finding into a regression test, and then make a human generalize that test from the instance to the class before it goes in the suite. Point the same energy at your dependencies and your own past assumptions, not just new code.

The comforting version of AI testing (point it at the repo, watch coverage turn green) is the version that protects your bugs and tells you you’re safe. The useful version is adversarial, uncomfortable, and never quite finished. Ask what the code does and it will tell you it works. Ask what breaks it and it will show you where you bleed. Ask for attacks.