Your AI Agent Says the Task Is Done and It Isn't. Here's the Harder Problem.

Everyone who delegates real work to an AI agent learns the first lesson the same way. The agent reports success. You go and look. It isn’t done — or it’s done somewhere else, or it’s done to a file that nothing serves, or the thing it edited didn’t save and nobody checked.

So you adapt. You stop accepting summaries. You demand output instead of narration. You put checks in front of the work, and the checks run on every change, and after a while you relax a little, because now there’s something between you and a confident wrong answer.

This is about what happens next, which almost nobody writes about.

A passing check is also a claim. Every instinct you built about not trusting the agent applies, unchanged, to the instrument you’re using to not trust the agent. And a broken instrument that’s wrong in the comforting direction is worse than no instrument, because you’ve replaced a healthy doubt with a green tick.

Here’s one of mine, with the measurements.

---

The contradiction

On 29 September 2026 a guard that walks my documentation corpus and checks whether pointers between files still resolve printed this:

✗ front-door pointers do not resolve
   CLAUDE.md→replit.md

That file genuinely doesn’t live in this corpus — it lives in a different repository. Which is known, deliberate and documented, so there’s an exemption list, and it’s on it. Line 24, [external] replit.md. Present. The bytes are identical to the token the check extracted.

And in the same run, a few lines earlier, the check that asks “does every exemption still apply?” — passed.

So in one execution the tool reported that the exemption list is fully valid, and that the thing on it isn’t exempt. Both cannot be true. When a tool contradicts itself inside a single run, you have stopped looking at a data problem.

It failed because it matched

The lookup was three lines:

printf '%s\n' "$(grep -E '^\[(external|known-gap)\] ' … | sed -E 's/^\[[a-z-]+\] //')" \
  | grep -qxF -e "$(basename "$n")"

Produce the list, pipe it to grep -q, ask whether the name is in it.

grep -q exits the instant it matches. That is the entire purpose of the flag — it’s an existence test and it shouldn’t read the rest. But two processes are involved. grep reads. printf writes. If grep finds its match and exits while printf still has bytes to push, printf writes into a pipe with no reader, and the kernel kills it: exit status 141.

set -o pipefail is on, because it should be — it’s the flag that stops a broken pipeline reporting success. It hands the if the first non-zero status in the chain, which is 141.

So the pipeline fails because the match succeeded. A hit reads as a miss. And the sentence that should bother you is the inverse: the healthier the match, the sooner the reader exits, and the likelier the whole thing inverts.

Ten runs

Reasoning about a race is how you produce a confident wrong answer, so I measured it. Ten runs, same expression, nothing changed between them, on the machine that reported it:

141 141 0 141 141 141 0 141 141 0

Seven of ten. Not a bug that fires and not a bug that doesn’t — one that fires most of the time, which is the worst of the three, because three consecutive clean runs look exactly like a fix.

It only ever fired for that one file out of twenty entries, because replit.md is line one of the list. First line means the reader can match on its first read and exit at the earliest possible moment, which is when it wins the race most often. The other nineteen sit far enough down that printf finishes first. For over a month the check was fine — nineteen-twentieths fine.

The same expression in a Linux sandbox, bash 5 and GNU grep: zero failures in twenty runs.

Had I triaged this on the cloud box instead of the laptop that reported it, I’d have closed it as non-reproducible, and my evidence for closing it would have been twenty consecutive green runs. Reproduce on the machine that reported it. That rule is cheap and I nearly skipped it.

The turn: I had already built the tool for this

This failure had happened to me before. 19 August 2026, different check: it asked whether my daily backup job was registered — launchctl list | grep -q. Same shape, same inversion. Across 200 trials it returned non-zero 5 times on a pattern that was present every time. The gate told me nothing was backing up my corpus; by hand, seconds later, in the same shell, the job was registered with last exit zero. Cost: one session of chasing the wrong thing.

So I did the responsible thing and wrote a lint. One job: walk every file with pipefail on and find anything piping into an early-exit reader. It had been passing every run since August.

Line 119 of that lint:

*'$('*|*printf*|*echo*) continue ;;

continue means skip. Any statement containing printf, echo, or a command substitution is treated as a safe producer and never examined. The reasoning is written in the file: a builtin writing a small, finite string completes its write into the 64 KB pipe buffer before the reader can exit.

That premise is false, and now measured false. Buffer size is not the mechanism. If the reader exits before the writer’s write() lands, the writer dies regardless of how few bytes it was carrying. My list is about 600 bytes — one hundredth of the buffer the premise rests on — and it failed 70% of the time.

Which means that between August and 29 September I carried a tool asserting “this corpus has no invertible checks” while one sat in a guard, in the block whose own comment describes the bug.

And then line 144 — the lint’s printed remediation, the advice it hands you when it does find something:

printf %s "$(cmd …)" | grep -q …                  ← drains before piping

The lint recommends the construct that just failed. Anything repaired by following its advice inherits the defect, and the lint then certifies the result clean.

Twelve lines above the broken lookup, in the same file, written by the same hand, the identical shape had been fixed correctly for a different call, with the comment “Capturing first removes the race entirely.” I knew the answer. I applied it twelve lines away and not here.

The obvious fix was wrong, and measuring is the only reason I know

Obvious fix: take printf off the safe list. I measured it before applying it, which turned out to be the only interesting decision in the whole investigation.

files with pipefail: 26 · checks examined: 156
84 newly flagged — 75 of them in _guard/test-guards.sh

Eighty-four new findings, seventy-five of them in the portability lint’s own rules — rules that are measurably working. That suite runs 272 of 272 green, repeatedly, which it could not do if those statements inverted at anything close to the rate mine did.

So the obvious fix trades one false negative for seventy-five false positives, in the tool whose failure mode is being ignored for accusing too much. Wrong lever.

The differential

Four configurations, one variable at a time, ten runs each, all on the machine that reported it:

Producer	Match position	Result
inline $(…)	line 1 of 20	inverted 7 of 10
precomputed variable	line 1 of 20	clean, 10 of 10
inline $(…)	line 20 of 20	clean, 10 of 10
precomputed variable, -qE	line 1 of 20	clean, 10 of 10

It needs both conditions at once. An inline command substitution in the producer’s argument, and a match early enough that the reader exits before the writer’s write lands. Either alone is safe. The substitution forks a subshell immediately before the pipeline, and that fork is the head start that lets the reader reach its match first. Move the match to the last line and the reader must read everything, so the writer always wins. Remove the fork and the writer always wins.

Now look at the safe list again, and at the first thing on it. $( — command substitution. The data says that is the danger marker. The lint exempts the one construct measured to invert. Not an incomplete safe list. A backwards one.

And here is what I don’t know, stated deliberately: the same shape with no substitution in the argument, against payloads of 1 KB, 55 KB, 224 KB and 449 KB — three times the pipe buffer, worst-case immediate match — reported correctly four times out of four. So payload size predicts nothing in either direction, and I have not isolated why those two cases differ. Saying more than that would be the confident wrong answer this whole system exists to prevent.

I was wrong twice inside one investigation

For the record. First I concluded the fix was narrowing the safe list — measured, wrong, 75 false positives. Then I treated the $( entry as defensible reasoning that merely needed a test — wrong again; it’s inverted.

Two reversals of my own conclusion in one sitting, both caught by measuring instead of arguing. Every single thing I got wrong, I got wrong in the reasoning step. Nothing survived contact with a measurement. That is not a confession; it is the method doing its job in public.

Three rules

1 · Every entry on a safe list, allow list or exemption list needs a test that fails when you remove it. If pulling the entry doesn’t turn something red, it’s either unnecessary or unverified. There is no third case. A safe list is the least-tested code in any tool: the assertions get fixtures, the exemptions get reasoning. Reasoning is what produced printf — plausible, written down, and wrong.

2 · If your tool prints advice, the advice is part of the tool and needs a fixture. Line 144 was executable guidance with nothing behind it.

3 · Prefer construction over detection. Match position is the real predictor here and it is not statically knowable — a lint can see the fork; it cannot see where the match will fall. So a detection rule either misses real cases or accuses sixty working ones. The remedy is to make the failure unconstructable: where the producer is a pipeline, drop -q and capture first. Without an early exit there is nothing to invert, at any match position, fork or no fork.

That third one is the general move, and it is the same move whether you’re guarding against a race condition or against an agent reporting work it didn’t do. Don’t build a better detector for the confident wrong answer. Build a handoff in which the confident wrong answer cannot be constructed.

For the status, because it would be easy to imply otherwise: the lint is still unfixed, on purpose, as of 1 October 2026. The single instance is repaired. I know the obvious fix is wrong and I don’t yet know the right one. I’d rather carry an honest open item than a green tick I haven’t earned.

---

If you delegate work to an agent

Three questions worth being able to answer before the next handoff:

1. What must the agent show you, rather than tell you, before a step counts as done?
2. What is the first read in the handoff — because a missing thing is loud on a read and silent on a write?
3. What checks the checker?

I wrote those three into a template, with three worked variants and the case study of a 25-cycle driver bug an agent closed against a gate while I ran zero commands.

The Agent Handoff Spec — $39

It won’t make your agent smarter. It makes the handoff the kind of thing that fails loudly instead of confidently — which is the only kind of failure worth trusting.