How to verify AI agent work: the test that proves your tests can fail

There is a specific failure that happens when you delegate work to an AI agent, and it is not the one people warn you about.

The warning you have heard is: the agent will confidently tell you something is done when it isn’t. That is real. But the fix for it is well understood — don’t accept the agent’s summary, demand the output. Read the state, not the claim.

This post is about what that fix does not cover, and about the day it cost me twenty-five minutes of total authentication downtime with a green test suite sitting on top of it.

The suite that passed

On 12 August 2026 a build went out behind a six-test pre-promote gate. Here it is, verbatim:

Test	Result
Login — unknown employee ID	401 ✓
`forgot-employee-id` — missing email body	400 ✓
`forgot-employee-id` — unknown email	200 neutral ✓
`redeem-code` — fabricated code	400 neutral ✓
`GET /api/users/1` — no session	401 ✓
`GET /api/auth/debug-proxy` — no session	401 ✓

Six for six. The build shipped seven security fixes. One of them changed a trust proxy setting, which broke session-cookie establishment on every route in the application.

Before you read on, look at the table again and answer one question:

What in this list would fail if the application returned an error to every request ever made?

Nothing. Every row asserts a refusal — a 401, a 400, a neutral 200, a rejection. A build that refuses every request in existence passes all six tests.

And one did.

Why this is not a bad-test problem

Every one of those six tests is correct. Each passed for the right reason inside its own frame. If you reviewed them the way tests are actually reviewed — one at a time, asking “does this assert the right thing” — you would find nothing wrong, because nothing is wrong with any individual test.

It is a failure of the shape of the suite, not of any test in it. That is what makes it survive review.

And here is the part that should concern anyone who has been getting more rigorous about verification, not less:

The discipline generates the defect.

The whole practice of “distrust the claim, read the state” — the practice that is correct, that I teach, that I built a product around — pushes you toward assertions that prove the absence of an effect. Unknown ID rejected. Unauthenticated request refused. Fabricated token denied.

Those catch a system that did the wrong thing. They cannot catch a system that does nothing at all. The harder you lean on the discipline, the more of your suite becomes refusals, and the larger the blind spot grows.

What it actually cost

The outage ran 00:10 to 00:35 UTC. Twenty-five minutes.

Roughly twenty of those twenty-five minutes were spent misdiagnosing it as a cookie scope problem. The rollback that finally fixed it discarded six unrelated, correct security fixes, because the rollback granularity was the commit rather than the change.

Now the honest part. The store was closed and nobody tried to log in. The observed cost was near zero.

That was timing, not process. The same build at 18:00 on a Friday takes a full shift offline in the middle of training. If you only count the outages that happened to hurt, you will never fix the process that produced them, because most of them get lucky.

The fix: a positive control

Every acceptance battery opens with a positive control. One test in which a real user completes the ordinary action, and the server confirms it happened.

The details matter more than the idea:

A real account performing the real action. Not a mock, not a fixture that bypasses the path under test.
Confirmed by a server response, not a UI state. In this incident the UI state was correct — the dashboard rendered. Asserting on the UI would have passed.
Reaching past the first call. The login itself returned 200. The assertion has to be a subsequent authenticated request succeeding, because that is what broke.
It runs first, and it gates. If the positive control fails, nothing else in the battery means anything, and you do not record the other results. A green row underneath a broken positive control is worse than no row, because it reads as coverage.

Three rules follow from it, and all three are free:

Count the ratio before running the suite. If zero tests would fail against a system that answers “no” to everything, the suite is unfinished regardless of its length.
Every changed path gets at least one success assertion. If a change touches login, one test must be a login that works.
A failed positive control invalidates the run. The other results were measured against an instrument that was not working.

The same defect one level down

There is a second-order version of this, and it is harder to see because the suite around it looks full.

A guard harness running under set -uo pipefail had assertions shaped like this:

check_output "$FIXTURE" | grep 'the finding' | grep -q 'the detail'

The upstream check exits 1 whenever it has findings — by design, correctly. Under pipefail, that exit status propagates no matter what grep matched. Written in the natural shape for “this must NOT fire” — && bad … || ok … — the assertion lands on ok every time, for any input, including a wholly broken check.

Four tests were in that state. Two had been since the day they were written.

The immune form is command substitution, which discards the upstream exit status:

printf '%s' "$(check_output "$FIXTURE")" | grep -q 'the detail'

But nothing made the difference visible until a positive assertion went through the same pipe and failed.

Which gives the two rules that generalize:

A test must be shown to fail before it is trusted to pass. Break the thing on purpose once and watch the assertion go red. An assertion never observed failing is a claim about a claim.
Pair every must-not with a must. “X must not appear” is satisfied by silence, and silence has many causes: the check errored, the fixture was empty, the pipeline swallowed the status.

Why this matters more when an agent is doing the work

Everything above is true of a suite you wrote yourself. It is worse when you delegate, for a reason that is structural rather than a comment on any agent’s competence.

When an agent reports “all tests passed,” it is reporting the output of an instrument. Nobody checked whether the instrument works. The agent is not lying — the tests genuinely passed. The summary is a faithful report of a meaningless result.

This compounds with a second habit. When an agent reports the result of a command instead of surfacing the command’s output, it is grading its own homework. “I ran it, it passed” is a claim; the verbatim output is evidence. The risk is not deception — it is that the agent’s summary is a lossy compression of its own expectations. It reports what it believed the output meant, and a belief-shaped summary cannot catch the case where the output contradicts the belief. That contradiction is exactly what a gate exists to catch, and exactly what a summary discards.

I have watched this pay off concretely. In one session, a routine two-line UI change had the agent write a temp test script that minted a token through the real auth middleware. One line read:

const SECRET = process.env.JWT_SECRET || 'your-secret-key';

Flagged as test hygiene, then one follow-up question: did you invent that string, or copy it? The grep answer came back with two matches — in server/index.ts and server/routes/brand-assessment.ts. The fallback was not a test placeholder. It was production auth code in two files: if JWT_SECRET were ever unset, the server would accept any token signed with a guessable, repo-grepable string.

Nobody was auditing those files. The task never touched them. A security review scoped to the diff would have passed two clean lines. It surfaced because a human read the agent’s throwaway artifact closely enough to ask about one string — and because the grep’s actual output was demanded rather than its narration. The result had initially been buried under a summary sentence. It was one unexpanded fold away from staying invisible.

That is the non-obvious argument for gated agent loops. Defence-in-depth is the obvious pitch. The real return is that a gate reviewing everything the agent produces becomes continuous, incidental review of the code the agent merely imitates.

What to do this week

Open your acceptance suite and count the ratio. How many tests would fail against a dead system? Ninety seconds.
Break one test on purpose and watch it go red.
For anything you hand to an agent: specify the evidence that must come back with the work, not just the work. “Run the tests” gets you a sentence. “Run the tests and surface the full output, then stop” gets you a gate.

The gate format I use — how to specify a task so the proof comes back attached to it, plus three worked domain variants — is written up as The Agent Handoff Spec, $39: ziplinecreative.gumroad.com/l/agent-handoff-spec

The ratio check is free, and it is the one that would have caught this. Go count.