← All Solo, Not Alone posts

ai-coding process

The Review Pipeline That Shipped Nine Errors

Four implementers and four reviewers passed nine false glossary entries in The Lien, each doing its job correctly. Then the fix repeated the mistake.

The Lien is a card game, and on 2026-09-21 I gave it a glossary. Twenty-four entries, each a short paragraph telling the player what one of the game's own words means. Rushing, Played Out, Sway, Ear, and so on.

The work ran through the same pipeline I use for everything on this project. A spec, then an implementation plan, then subagents that each take one task from the plan, then a reviewer per task, then a whole-branch review at the end. It is a careful process and it has caught a lot.

This time the whole-branch review found that nine of the twenty-four entries were false. Rushing had been given Played Out's trigger. Played Out was described as something that happens to the enemy, when it caps the player's own hand at two cards. Ear was said to do nothing until something heals you. Hoodoo said a purchase changes the run you are in, which is backwards, since purchases carry into later runs. Sway was said to add to every point of Block, which by the lesson I wrote afterward predicts 16 Block where the game gives 9. Resonance, Linger and Silver Dimes were wrong too, and one entry's historical half made a claim about the twelve-bar form that bars 4 and 8 contradict.

Where the errors came from

None of the nine came from an implementer misreading code. The entry prose was written into the implementation plan itself, from assumption, and nobody checked it against the handlers it described. Four implementers then wrote it into the game verbatim, because the plan told them to. Four task reviewers passed each entry, because each one correctly checked that the entry matched its brief.

Every step did what it was asked. Nobody was asked whether the brief matched the game.

The line I recorded in the repo's lessons file says it better than I can paraphrase it: "The thing built to catch errors is what distributed them, and it did so faster and more consistently than a human copying by hand ever would." A person copying nine bad paragraphs by hand might have stopped at one and said that is not how Rushing works. The pipeline had no step where that could happen.

Then the fix did it again

The correction pass was made by opening each handler and writing from it. That is the right method. Sway's fix still came out wrong.

The first draft of the correction said Sway was added "once to every Block card." The code adds it once per Block effect, and three shipped cards carry two Block effects. One of them, Steady, is in the starter deck. With Sway at its starting value, a first-fight Steady with its rider firing pays 9 Block, and the corrected entry predicted 8. The implementer had read the right file and written the wrong unit. A reviewer caught it by sweeping every card for the shape the sentence implied, not by rereading the file.

There is a coda. A separate audit four days later, one reader going through all 25 entries against the source (a 25th had been added since ship), found that Ear's corrected text was also wrong. The fix said Ear pays from the first fight. In the code the bonus is Ear minus one, and Ear starts at one, so the starting point adds nothing until a level-up. That audit flagged 13 of the 25 entries for new wording, three of them as outright wrong.

What I changed

The rule I wrote down is narrow. Prose that describes behavior is authored by reading the implementation, never by a plan restating it. A plan may name which file to read. It must not supply the sentence. When a brief does supply one, the reviewer's question has to be "does this match the code," asked explicitly, because nobody asks it otherwise.

The broader point for anyone working alone with agents is about where the blind spot sat. My review steps were checking conformance, each to the document one level up. That catches drift between layers. It cannot catch an error that was in the top layer from the start, and more layers only copy it more faithfully. When I am the only human in the loop, part of my job is to look at a pipeline that passed and ask what it was measuring against.

More on the game at /games/the-lien, and the rest of this series at /notes.

← All Solo, Not Alone posts