Green Checks That Proved Nothing
A suite reported Passed with 193 tests when it should have run 434, and a layout test reported 52px of clearance while the button sat 43px off screen. Three ways a green result stopped meaning what it said.
The Lien's test suite is large and mostly written by agents, so I read a green result as a claim and ask what it is a claim about. Three cases in about a month, 2026-08-11 to 2026-09-08, each showed a different way that claim can be empty.
A pass that ran less than half the suite
On 2026-08-11, two runs in a row came back with a Passed! line and a total of 193 tests. The suite has a floor, a minimum count it expects to see, and the floor was 434. Every test that needs the Godot runtime was missing, and the runner script refused both results because of that floor. Without it I would have merged on a green line that had run less than half the tests.
The cause took an A/B/A/B isolation to find. Godot was finishing its entire shutdown and then never terminating. The test framework waited, hit its compile timeout, killed the process and reported that the rebuild had failed. The variable was one setting, runnable=true on the Android export preset. With it, the process hung. With it off, the process exited in about ten seconds, and an empty scratch project on the same binary did too, which ruled out the engine and the machine. The mechanism is inferred, not traced, and the notes say so.
The write-up also refuses to close an earlier case. On 2026-08-10 the suite had been truncated to 180 tests on a day that was also green several times with the preset present, which that setting could not have produced. Two causes, one signature, one still unexplained. Calling it solved would have repeated the mistake the whole investigation was about.
Tests that passed with the feature deleted
On 2026-09-04 a plan for showing the player's companion shipped tests that measured the wrong thing in five places. Six tint and riding-alone tests passed with the feature they named fully deleted. A portrait test could not detect the image shrinking to zero height. A test written to pin a safe-area bug did not pin it, since deleting the line it covered left every assertion green.
The fifth was the one that taught me something new. A safe-area test set synthetic margins on a live node, and that node's own resize handler immediately overwrote them with the real inset of the machine the test ran on, about 38 pixels. The test was asserting against a number nobody chose. At a zero inset, which is likely on CI, it would have passed against the exact unfixed code it existed to catch.
A reference that moved with the bug
The worst case came on 2026-09-08. DeckBuilderButtonClearanceTest measured a button against the rendered rect of the safe-area container, on the belief that this was the true edge of the screen. A Godot container is clamped to at least the combined minimum size of its children. Once the column inside it overflowed, the container grew to match, and the test compared the column against itself. It reported 52 pixels of clearance. The real button sat 43 pixels off screen.
The earlier cases were tests that measured nothing. This one measured something, passed, and passed while the defect it was written to catch was live. The fix reads the safe area from the display server directly, because nothing a child does can move that number.
What I changed
Each case got its own narrow rule. The runner enforces a floor on the test count. A test for a fix has to be shown failing with the fix deleted. A reference a test compares against must be something the code under test cannot move.
The shared point is plainer. A green line says that the assertions that ran were true. It says nothing about whether they were the right assertions or whether all of them ran. When agents write most of the tests and I am the only human reading the result, the one habit worth keeping is asking what the pass would have looked like if the code were broken.
More on the game at /games/the-lien, and the rest of this series at /notes.