← All Solo, Not Alone posts

ai-coding testing

Every Scenario Quietly Deleted My Save

A debug launcher for The Lien promised in a comment that it would never touch my save. It overwrote the save within a second of starting, and the first fix did not stop it either.

On 2026-07-31 The Lien got a debug scenario launcher. You start the game with --scenario map or --scenario vagrant-block and it drops you straight into a seeded run, so a human can check a claim in a few seconds without playing to get there. Four scenarios, 149 tests, a plan, a reviewer per task. The final review of the whole branch said "do not merge yet" and listed two critical problems that none of the four task reviews had seen, since each of them only read its own diff.

The second critical was that every scenario silently destroyed my save.

A comment that asserted a guarantee

The driver that seeds a run had been written carefully. It deliberately did not call Save(), and a comment said so. The scenes it hands control to do their own saving. RunScene saves as it loads, so --scenario map overwrote save_0.json within a second of launching, before I had touched anything.

In the words I recorded afterward, the driver's own comment asserted a guarantee it did not have. The code was locally correct, and the comment was true about the code. The harm happened one hop downstream, in code the driver had handed off to.

The first fix was wrong too

The fix was to copy the existing save aside before seeding, to a timestamped backup, and to refuse to seed if the copy failed. I proved it by hand. I hashed the real save, launched the map scenario, saw the hash change, checked that the backup matched the original hash, and restored it. The stale comment was rewritten.

That same day I smoke-tested the launcher myself, and my real save was replaced again. The backups were doing their job as a way to recover, with a fresh stamped file for every launch, five of them in eight minutes. My save was found overwritten and I restored it from a backup. A backup is a mitigation. The save was still being destroyed on every launch, and recovering from that was now a manual step.

The real fix was a debug-only redirect. Scenario saves go to their own file, and the real save is never opened for writing. The commit that records it says plainly that mitigating a hazard and removing it are different things, and that the earlier comment claiming protection had been describing the mitigation.

The test suite did the same thing

On 2026-08-05 I found the same fault in a different place. The test suite had been overwriting my real save too. The save held the one-card deck from a save test. I counted the suites that wrote to the default path. A search for Save() in the tests found three. There were eight. A rule that has to be rechecked by hand against every new suite is not a rule, so the fix was a shared helper that points every test at a test-only file, and no test writes the default path now.

What I changed

A component can be correct and still cause the harm its comment denies, if nobody checks what it hands off to. The check that would have caught this is cheap: run the thing against real data and compare a hash before and after. Reading the code of the thing that makes the promise is not enough.

A launcher that exists to let a human verify a claim must not misstate the claim. When I am the only person who will ever launch it, what it does to my own data is the first thing to test.

More on the game at /games/the-lien, and the rest of this series at /notes.

← All Solo, Not Alone posts