oracle-gate-a-change: add the red-first rule — an oracle that passes before the change measures nothing

[avrabe]

The pattern

oracle-gate-a-change says to write the missing oracle first, then "land code that flips the oracle from red to green". In practice the step that actually catches the bug is the one between those two: run the oracle before implementing and confirm it is RED. Skipping that confirmation produced three separate false-green oracles in a single scry session — each of which would have shipped as evidence.

Three real instances, all from one session

1. A predicate satisfied by the initial state. Test asserted "some program point has local 0 = [0,0]" to prove a new guard refinement pinned a loop counter. (local i32) is zero-initialised, so it matched the entry point. Passed before the feature existed. Fixed by asserting position-specifically (the last point), after which it went genuinely red.

2. A fixture selected around the failing case. Test asserted an ID survives inserting instructions earlier in the same function. The chosen insertion happened to contain no operator of the same kind — the one case where the property holds. An adversarial reviewer found the general property is false. Worse: the insertion also changed the analysis result, so the fixture dodged the property in two independent ways.

3. A gate predicate with the wrong polarity. A claim-check claim bound a doc's version string to a count-max predicate over Cargo.toml. count-max only fails when n > max, so when the version drifted the pattern matched 0 times and the claim stayed green — an "oracle that measures nothing" (the skill's own phrase) in the gate meant to prevent exactly that. (Filed separately as claim-verification: claim-check.py count-max greens a 0-match — presence claims need a count-min dual #131.)

Note the range: a unit test, a property test, and a CI gate. It is not a testing-hygiene issue, it is an oracle-design issue — which is this skill's subject.

Suggested addition

A short "Confirm the oracle is red" step, plus two anti-patterns:

• An oracle that passes before the change is not an oracle. Run it first; if it is green, it is measuring something other than the property. The two commonest causes: the predicate is satisfied by the initial/default state (zero-initialised values, empty collections, absent files), or it is an upper bound where a presence check was needed.
• A fixture chosen after the property is written is suspect. If you picked the input because the test passed, you may have selected around the failing case. Ask what the property claims in general, and whether your fixture is the easy instance.

And a note that this applies to gates as much as tests — a CI check whose predicate can silently match nothing is the same failure with a longer blast radius, because a green check is read as evidence by everyone downstream.

Filed from Claude Code after the pattern recurred three times in one session on pulseengine/scry.

[avrabe]

Landed in PR #162 as step 2c, "Confirm the oracle is RED — before implementing" (oracle-gate-a-change 0.4.0), framed against the existing 2b: 2b proves a gate runs; 2c proves it can fail. Both anti-patterns are in, plus the gates-as-much-as-tests note.

One deliberate change: it cites one instance, not three. I tried to reproduce all three before writing:

• Instance 3 (polarity) — reproduced. count-max fails only when n > max, so a 0-match stays green. The implementation's own comment now reads "greens a 0-match, since 0 > max is false", and count-min exists to catch drift-to-absent. This is the worked example in the skill.
• Instance 1 (initial state) — could not reproduce. In the scry probe the points start at pc=2 with [0,9]; there is no [0,0] point, and the pre-fix assertion was never committed, so there is nothing to run.
• Instance 2 (fixture) — likewise no committed pre-fix state to re-run.

The rule is sound and generalizes regardless; the evidence is one verified case, so the skill claims one. Citing unreproducible instances as field evidence would be the same error the skill warns about — an assertion that reads as verified because it appears next to verified things.

The tamper/negative-control material went to gate-potency 0.2.0 instead, since it audits a shipped check rather than demonstrating red-first on a change being authored. It also picked up a rule this exercise produced: pair the negative control with a positive one — a bogus trust root makes varve verify exit 1 with no realm and is ignored with one, which isolates the mechanism rather than merely showing that something fired.