Skip to content

test: eleven more checks that can pass while being wrong #71

Description

@mtklein

The v0.9 test audit proved three suite tests could pass while being wrong, each by breaking the thing it covers and watching the test stay green (c65324b). All three are fixed. Eleven more suspects were identified but not fixed, each with the concrete ROM change it would fail to notice. This issue tracks working through them.

This is the repo's canonical failure mode — "a check that can pass while being wrong" — and the fixed three show it is not hypothetical: battle_entry reported green in 2 seconds on a ROM with jsl Ot6SeedShields deleted, i.e. with every enemy in the game unbreakable, while its own header calls it "the quick iteration loop for battle/break-system changes."

Ranked, highest value first. Each line names the change that would slip through:

  1. battle_hudclobber.lua:159-161 — asserts junkFrames == 0 with a control that proves only the hazard, never that the HUD renders. Make Ot6BgHudFlush_ext's veil permanent, or delete the under-enemy HUD entirely: still green. The "repaints once the tiles are whole" half — the thing the file is named for — is untested. battle_whelkwipe.lua:242 already contains the control this needs, so it is a copy, not a design problem.
  2. battle_divines.lua:241-247 — the "honest reduced fallback" has no positive witness, and guard HP is re-pinned every drive tick (:122, pinGuardHp set :220), so damage is structurally unobservable. Make Ot6Oblivion's unbroken arm power-0 or bail entirely and all four assertions pass — a non-Broken Oblivion silently becomes a wasted turn plus 99 MP.
  3. battle_gaufight.lua:587-596 — the shield ledger is asserted against a natural world-map encounter (walkIntoEncounter, :225-255), unpinned, and nothing asserts any hit body is bludgeon-weak. On a non-bludgeon roll the assertion degenerates to false == false. Delete the OT6_BLUDG arm from Ot6ClassChip and it is a coin flip whether the suite notices. This is also a flake waiting to be blamed on something else. One-line fix, or pin the formation.
  4. battle_runic.lua:779-788driveUntil checks its predicate before the body, so if turns(celes) were already 0 the walk returns immediately and droppedEarly stays false. The entering value is logged (:776-777) but never asserted. One line.
  5. battle_slotsboot.lua:315-326 — assert-in-one-branch; the gated arm has only an H.log. On any run where reel 1 lands the 7 in a $2f49.2-forbidden battle, the entire tier-3 promise silently vanishes. Nondeterministic, which is worse than stably wrong.
  6. battle_reveal_poweron.lua:102-105 — accepts $BF or $FF or $00 for a '?' cell while running under OT6_RAM_POWERON=AllOnes, so an unpopulated shadow line reads $FF and passes. No fieldHudPresent() or glyphCanary() control anywhere in the file. Gut Ot6BgHudLine's weakness-cell write: green.
  7. visual_f2.lua:31-40if alive then … with no assertion that alive was ever true; both lobos dying inside the blind 600-frame window skips the headline check.
  8. battle_mpcost.lua:188-192 — self-detecting mode inverts every expectation. Turn OT6_MP_COSTS off in the shipped build and the default suite run detects "off", asserts SwdTech is free, and passes. Only Makefile:140's byte-compare, outside the file, stands in the way.
  9. battle_class.lua:243-245 and battle_codex.lua:36-37 (slots 2 and 3 only) — true write-then-read tautologies. H.loadState writes 0x4f/0x38 into all four codex page headers on every load (lib/ot6.lua:306-312) and codexBase() always returns one of those four, so they cannot fail for any ROM change. battle_codex's slot-1 case is real — it deliberately plants 0x37 at :22.
  10. battle_reveal.lua:95-97 (residual after the fix) — part 1's real anchor clsW == 0x02 is gated on sp == 0 with no assertion that a species-0 monster is present. If the battle_doorstep formation changes, part 1 collapses to three zero-checks that InitBattle's clear satisfies alone.
  11. Inert but harmless, listed so nobody re-finds them: battle_breakvector.lua:167,225,300,325 compute purely from Lua literals; battle_breaktbl.lua:133,135 and battle_breakfloor.lua:61 are guaranteed by their own locator sequences; battle_bushido.lua:224-229 is an empty-bodied for loop that reads like a ceiling sweep.

The bar for closing any of these

Demonstrate the failure. Make the ROM change named above, watch the test stay green, fix it, watch it go red on the same ROM, restore, watch it pass. An argument that a test is sound is worth very little here — all three of the confirmed cases looked fine on inspection.

And do not weaken anything to get green. If an assertion turns out to be wrong rather than weak, say so explicitly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions