fix(data): repair inaccurate code locations

[Sam-Hui-dot]

Closes #4

Summary

This PR adds a conservative, reproducible workflow for auditing and repairing inaccurate code locations in VulnGym.

It:

• validates every entry_point, critical_operation, and trace[*] node against the exact repository commit recorded by the entry;
• searches in the required order: recorded location → ±5 lines → original file → complete repository;
• repairs only unique, sufficiently informative source matches;
• preserves ambiguous, missing, empty, or unsafe cases in a manual-review queue;
• produces a complete repaired JSONL dataset and source-backed audit artifacts;
• includes an independent verifier and focused regression tests.

The original data/entries.jsonl is not modified. The repaired dataset is written to data/entries.fixed.jsonl.

Repair policy

Matching tolerates:

• CRLF/LF differences;
• tabs and repeated horizontal whitespace;
• leading and trailing whitespace;
• contiguous multiline snippets.

Location semantics are intentionally conservative:

• an integer line remains a start anchor, including for multiline code;
• a "start-end" value must match both range boundaries;
• repaired ranges use the natural matched span instead of preserving an incorrect old width;
• ellipses are never interpreted as wildcards;
• empty and low-information snippets cannot establish a shifted location;
• verified entries require a unique verbatim match before an automatic repair is allowed.

Exact matching always runs first. A secondary annotation projection removes only clearly annotator-authored CJK // or # suffixes from the snippet side. Source text and ordinary source comments are never broadly stripped.

When a location changes, only explicit file or line anchors in desc are synchronized. Unrelated numbers such as 第 1 步 and descriptions without location references remain unchanged.

Full-dataset results

The complete dataset was scanned:

• 408 entries;
• 2,889 code-location nodes;
• 23 repositories;
• 166 repository/commit pairs;
• 166/166 source commits available;
• 2,841 nodes accepted at their recorded location;
• 19 unique verbatim repairs;
• 29 nodes retained for manual review.

Repair strategies:

• 17 nearby_unique;
• 2 same_file_unique;
• 0 speculative or fuzzy repairs.

The 29 manual-review rows consist of:

• 22 snippets not found in the complete pinned repository;
• 5 snippets containing ellipses that must not be treated as wildcards;
• 1 empty snippet with no source-location evidence;
• 1 projected candidate in a verified entry that lacks a complete verbatim match.

Each manual row includes a stable review_id, an expected_node_sha256, the review reason, candidate evidence, and a pinned source URL.

Priority samples

• entry-00103 trace[1]: 605-614 → 606-614
• entry-00103 trace[2]: 146-155 → 146-154
• entry-00185 trace[1]: 75-82 → 76-82
• entry-00320: all six nodes already matched
• entry-00511: all seven nodes already matched

Explicit line references in the affected entry-00103 descriptions were synchronized without changing step-number semantics.

Audit artifacts

This PR includes:

• data/entries.fixed.jsonl — complete repaired dataset;
• reports/fix_diff.csv — full before/after values, strategy, source URL, and source SHA-256 for every repair;
• reports/needs_human.csv — stable manual-review queue;
• reports/code_location_sources.csv — per-commit source coverage;
• reports/code_location_report.json — machine-readable counts and integrity hashes;
• reports/code_location_report.md — human-readable run summary;
• reports/code_location_acceptance_matrix.md — requirement-by-requirement evidence.

All generated artifacts use deterministic ordering, UTF-8, LF line endings, and final newlines.

Independent verification

scripts/verify_code_location_outputs.py verifies the generated artifacts independently from the repair resolver.

It confirms that:

• both datasets satisfy schema and report-ID mapping validation;
• every dataset mutation has exactly one diff row;
• applying only the diff rows recreates the fixed dataset exactly;
• no code field was changed;
• every verified automatic repair is verbatim;
• repaired snippets exist at their recorded commit/file/span;
• source hashes and pinned URLs match;
• manual-review rows still bind to their original nodes;
• source coverage and report totals are internally consistent.

Validation

Commands executed:

python3 -m unittest discover -s tests -p "test_repair_code_locations.py" -v
Ran 36 tests
OK

python3 scripts/verify_code_location_outputs.py
artifact verification passed: 408 entries, 2889 nodes, 19 fixes, 29 manual, 166 source commits

Additional checks completed:

• full online run: 71.876 seconds, below the 10-minute requirement;
• cached offline reproduction completed successfully;
• repaired dataset rescan: 0 additional fixes;
• input and output SHA-256 were identical during the idempotence scan;
• two full cached runs produced byte-identical generated artifacts;
• UTF-8/LF and trailing-whitespace checks passed.