testing: stop cheating — the honesty program (inputs in, observations out)

[mtklein]

(Claude here:)

Owner directive, 2026-08-03, triggered by #74: "We need to stop cheating in our testing... If you are not playing actual game states that have been reached through actual game choices, you are just wasting time pretending." This issue tracks the program until the whole test system plays for real.

The rule

A test or mint script may inject controller input and read emulated memory to assert things. It may never write emulated game state. Determinism comes from fixed, honestly-minted fixtures plus frame-exact input (TAS-style), never from rigging. Fixture honesty is transitive: a savestate or anchor minted by a script that poked state is contaminated, and so is everything derived from it.

What the census found (2026-08-03, three parallel audits)

• 796 state-write call sites across the 82 suite tests; 233 of 327 .lua files under tools/tests write state.
• 2 of ~110 fixtures are honest (battle_doorstep, first_battle — cold power-on, pure input). Everything else is contaminated, chiefly by the kill-bit idiom ($3EEC |= $80) that lives inside seven lib/ot6_field.lua navigation helpers and fires on every random encounter, plus per-leg pins/clamps/pokes. Every tracked battery anchor was cut through a ZMENUSTATE poke the README itself forbids, and carries harness-authored witness bytes.
• Every boss from the Whelk to Number 128 has been "won" by kill-bit in the mint chain. No automated play has ever beaten a boss honestly past the two existence proofs below.
• The cost of this showed up as v0.6 Phantom Train #74: gen_sabin_train.lua pins HP/MP, freezes everyone but Sabin, and clamps the Phantom Train to 1 HP — so a break loop that real players cannot complete sailed through every gate since v0.6.
• Two in-tree proofs that honest play works: battle_vargas.lua drives Blitzes by real pad edges, and gen_sabin_magitek.lua wins its battles honestly because kill-bitting them softlocks. The honest rewrite is engineering, not research.

The plan (landing order)

1. Static checker ratchet (tools/check_state_writes.py, in the make test checker family): every write idiom in tools/tests/**/*.lua fails the build unless it's in a checked-in burn-down waiver list that may only shrink. New cheats become impossible to land immediately. (In progress.)
2. Cheap principled conversions: the four save-menu ZMENUSTATE pokes → real pad drives (the repo already has the template and its own rule requiring this). (In progress.)
3. Honest library mode: replace the kill-bit in the nav helpers with real battle clearing (tap-A / flee via the engine's own run mechanic), replace H.loadState's codex SRAM wipe with pre-boot battery injection through the existing anchor path.
4. Runtime gate: compose.py prepends a prologue that removes emu.write*/setState/cheats/reset from every composed script before user code runs; H.loadState keeps the one confined savestate-load handle. Static + runtime = defense in depth.
5. Provenance: mint stamps additionally bind the artifact hash and the ancestor's stamp hash, and carry a gate-contract version, so make test can verify the whole chain was minted input-only, transitively. Landing the version string deliberately invalidates every existing stamp.
6. Root-first honest re-mint of the entire frontier (the ninja graph already orders it and auto-invalidates descendants). Suite tests convert wave by wave to honest fixtures + real drives; the burn-down list shrinks to zero.

Known hard legs (each gets its own resolution, none get a rig)

• Banon river/hideout legs: any death is Game Over — needs real survival play.
• Shadow's 1/16 post-battle leave roll on the Sabin legs — TAS-style timing, not the $1DD2 pin.
• The Gau Dried-Meat feed: a probe measured it undrivable even on vanilla — either the probe missed the human-possible input or there is a real game-side question. Investigate before touching game code.
• Number 128 with solo Locke at 151 HP: the upstream party menu choice needs revisiting — honest play may simply lose, which would itself be a finding.
• The Zozo bridge "hanging" encounter is un-settleable and currently un-mintable without suppression — likely a product bug to fix first.
• The banquet's 24 fights inside a 14400-frame timer: feasibility open even with cheats; honest play may force retuning.
• The Phantom Train itself (v0.6 Phantom Train #74): the honest mint must beat it with a real strategy; at current tuning that is a razor-thin line, so the fix for v0.6 Phantom Train #74 and the honest re-mint of that leg land together.

Policy note on mechanism tests

A small quarantined class injects inputs the game can only produce rarely or never on cue (deliberate VRAM corruption for the font-restore path, the 1/65536 zero-checksum save, legacy-format codex migration). These are unit tests of mechanisms, not claims about gameplay; they stay, individually waived and loudly labeled, and may never produce fixtures. Everything that claims to test actual play converts, no exceptions.

[mtklein]

(Claude here:)

Route call from the owner for the honest re-mint: the Number 128 fight should NOT be attempted solo — the upstream party selection (the Zozo-exit / regroup menu the current chain waves through, leaving Locke alone at 151 HP by the minecart) should bring Edgar and Sabin along. The honest gen for that stretch drives the party menu for real to pick them up before the factory descent, and the 'solo Locke may simply lose' concern is withdrawn.

[mtklein]

(Claude here:)

Pilot wave results — the root of the fixture chain now plays for real:

• First honest boss kill in the chain's history: the Whelk, beaten with a real strategy (and the attempt exposed two errors in the whelk balance script's own 'tutorial' policy — Terra's Heal Force cell was mismapped to Ice Beam on her column-major grid, and attacks queued late in a shell-up window retarget into the shell and eat counters). Honest fight ≈ 18.4k frames vs ~1-2k for the old kill-bit.
• whelk_doorstep, arvis_wake, narshe_streets, moogle_doorstep all minted with zero state writes (honest cost at the doorstep: +23% frames). Additive lib support: H.fightBattle / H.fleeBattle / opts.honest on the navigators — kill-bit paths untouched until each leg converts.
• The static checker is live on main (23367ba): 1617 write sites in 235 files under 374 shrink-only waiver pairs; any NEW write fails make test immediately.
• The four save-menu ZMENUSTATE pokes are converted to real pad drives (integration pending its gate).

New hard-leg finding (Marshal / Moogle-defense set-piece): played honestly, every guard march converges single-file into party 1 at the (14,14) choke; party 1 wins two waves and wipes on the third; squads 2/3 have no path past the choke. Worse: a party wipe there is not a Game Over but a corpse-jam softlock (guards jam behind the dead party indefinitely) — if that reproduces for a human player, it's a shippable bug on its own, since players will wipe on this set-piece. A dedicated investigation with a real passability dump is running; if the set-piece is not honestly winnable as tuned, that's a #74-class balance finding.

[mtklein]

(Claude here:)

Marshal/Moogle-defense follow-up — the set-piece is honestly winnable, and the honest gen now plays it end to end. The earlier 'impossible' reading was bad data from a stale-broken seeded fixture (frozen NPC script pointers made the marches motionless and Y-switching misbehave). Re-measured fresh:

• The three-squad design works exactly as the game's own dialog teaches: deploy all squads to interception stations during the pre-wave quiet (east loop, west loop, choke), collisions auto-engage, each squad absorbs two waves, and the Marshal falls to boosted attacks (R then A-A-A) with a defeat-retry ladder as the human-realistic fallback. Tuning is fine given the three-squad design — it only looked impossible through a single-goalie lens.
• moogle_defense and moogle_cleared are minted honestly (zero writes), and the downstream worldmap_narshe mint booted them green.

Two player-facing findings from the wipe path (players WILL wipe here):

1. A lost wave battle parks on the 'Annihilated' screen waiting indefinitely for a keypress — fine for a human, but worth knowing.
2. Real bug: the defense's loss path (_ccaaba) is missing the fade_in the win path has — after pressing through the defeat, the field stays black while guards keep marching invisibly until the genuine game-over triggers. Screenshot evidence in the probe logs. Fix looks one-line-shaped (add fade_in/wait_fade to the loss branch); queued after the current integration lands.

[mtklein]

Fyi I have played v0.7 through the shiva/ifrit fight in magitek factory. The game is playable at least through there; if you think it is stuck at any point before that, you are misinterpreting the game state.