fix(op1): lock canonical data by manifest_hash, not an absolute-path blocklist

[atosonik]

The bug: op1 rejects every honest bundle

After the canonical bump (#86/#655), op1's check_canonical_data_source is self-contradictory:

1. proof/runner.py:444,449 injects an absolute realpath for --manifest and --data-base-dir (intentional, to pin shard resolution).
2. recipe train.py main() writes those into cfg; final_state["config"] records them via asdict(cfg).
3. check_canonical_data_source rejects any absolute path (_NONCANONICAL_PATH_RE = ^\s*(?:~|\.\.|/)).

So every honest run trips it (validator.py → fatal op1 reject) — including the validator's own /app/data re-run. Reproduced: running the runner's exact train_cmd on c813831 records config.manifest_path=/.../data/data_manifest.json → check_canonical_data_source → False.

It also never caught the original threat: "/home/root/diony/recipe/data/data_manifest.json" ends in the canonical suffix yet points at miner data — a path string can't tell canonical from contaminated.

The fix: lock by content, not path

Enforce the data-lock by content — final_state.manifest_hash (bound into bundle_hash, tamper-evident) must equal the validator's canonical training-manifest hash. This is exactly what data/manifest.py::manifest_hash() already promises: "the value extended into the proof-test attestation user_data so validators can verify which manifest the miner trained against." The validator just wasn't checking it.

• Honest absolute runner paths now pass (content matches).
• Redirects/contamination are caught regardless of path string.
• Falls back to the original strict path guard when the canonical manifest can't be recomputed.

Tests

tests/test_integrity_trained.py: all existing path-guard tests still pass; adds 3 content-lock cases (honest absolute path accepted, redirect-by-hash rejected, fallback guard). 26 passed.

Single file changed (validator/integrity.py) + tests. No change to measured paths (validator/ is not in container_measurement).

[karpabot]

Thanks for catching the #85/#86 contradiction — the diagnosis is exactly right (the #655 runner pin injects an absolute path into hash-bound final_state.config, and the allowlist then rejects it).

Three issues with the manifest_hash content-lock, verified against the live netuid-40 validator:

1. Inert on the live validator. The box has no RECIPE_DIR/data/data_manifest.json (the manifest is host-generated by data.prepare and is not materialized on the validator), so _canonical_train_manifest_hash() returns None and the code falls straight back to the same absolute-path guard it is replacing. Honest bundles stay rejected — it is a no-op in production until the canonical training data is materialized on the validator.

2. No single canonical manifest_hash exists. The 6 scored bundles carry 5 different manifest_hashes (each miner runs its own data.prepare over FineWeb; the hash is content-addressed by relpath+sha256, so any difference in shard set/order changes it). If the manifest were present, this would reject honest miners whose legitimate prep hashes differently — including the current clean king 7fd43cef (55370528...), which differs from the two most recent pending bundles (fe983ecb...).

3. It does not stop the threat that is actually crowning. manifest_hash is miner-produced in final_state. The live exploit is off-protocol training with a fabricated final_state (attest canonical code, train the checkpoint elsewhere); that attacker simply writes the canonical hash and passes. A content check over a miner-declared field cannot catch a miner who fabricates the field.

Since resolution is already runner-pinned (proof/runner.py chooses --data-base-dir, overriding miner config), gate-4 really just needs to stop false-rejecting the runner own absolute path. The minimal fix is to relax it (accept the runner-pinned absolute path, keep rejecting ~ and ..), not add a content-lock that is inert + risky.

The durable defense is pre-crown re-derivation (re-run a slice of training on canonical data, check the loss trajectory reproduces), which we are building now. Heads-up: this also overlaps validator/integrity.py with the op1-hardening in #89 (timing gate + require-final_state + fraud-checkpoint blocklist), so they will need reconciling.