feat(mine): expand CVE->fix-commit sources (M1 OSV, M2 residual finder, M3 NVD) - #27
Open
songwen6968 wants to merge 37 commits into
Open
feat(mine): expand CVE->fix-commit sources (M1 OSV, M2 residual finder, M3 NVD)#27songwen6968 wants to merge 37 commits into
songwen6968 wants to merge 37 commits into
Conversation
S1: add mine/dedup.py (KnownSet — cve_ids + full-length shas). process_datasets now collapses records on the commit sha only (has_sha), fixing the B8 mirror duplicates the old (project, base_commit) instance_id key missed. No cve_id dedup: the final set may legitimately carry duplicate cve_ids on distinct commits. Refactor process.py into thin orchestration: - lift CVERecord + code_test_split to mine/cve_record.py - move MorefixesHandler / ReposVulHandler to mine/sources/ behind a Source protocol (get_dataset -> records(known)) with a SOURCES registry + SOURCE_BY_NAME - per-source "N records collected" logging moves up to the orchestrator Behaviour-preserving (imports + a code_test_split micro-test verified). Also: get_repo_dir names clones owner__repo (S4) so different owners sharing a repo name never collide on disk at mass-clone scale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sources/osv.py: OSVSource reads OSV PyPI all.zip, extracts each CVE's fix commit (GIT-range fixed shas + reference /commit/ URLs), fetches the .patch from GitHub like crawl (no clone), and funnels through code_test_split. The full 40-char sha is read back from the patch's From header so KnownSet dedups exactly even when OSV gave a short /commit/ sha. Runs last in SOURCES, contributing the sha-new residual over Morefixes + ReposVul (no cve_id dedup, per the S1 decision). Validated: extraction reproduces the offline model (5687 CVEs, 3340 with a direct commit); From-header full-sha resolution + has_sha pre-fetch skip unit-tested on real OSV data; one live fetch (CVE-2026-28681) runs the full fetch -> resolve -> split -> code_test_split path to a valid instance. Raw data at datasets/cve_records/OSV/pypi.zip (untracked). A full net-new measurement run needs GITHUB_TOKEN for the ~432 patch fetches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each source now logs +N net-new (running total) after its funnel, so a multi-source run reads out each source's marginal contribution (M1's +N) directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…record keys Rename mine/process.py -> mine/core.py (repo's core.py orchestration convention; mine was the lone violator) and process_datasets -> build_fix_dataset. Fold the CVEFixRecord type + code_test_split back into core.py -- the extracted module's boundary was unclear and only core uses them. Add --skip_verify to stop at the text-level funnel (no clone / apply-verify / test-mask), for measuring source yield without touching repos. Crawl: delete crawl.py; fetch_github_commit_patch moves to sources/utils.py, shared by OSV and Morefixes. MorefixesHandler gains an optional fetch mode (--morefixes_fetch) that rebuilds its patch dataset from the URL dataset before reading; default reads the cache. Rename clone_repos_and_verify_patches (was download_...). Logging: the per-run log dir is threaded explicitly (no shared module-global "current log dir", which clobbers across stages in one process). init_loggers + setup_logger per module; a multi-module stage's init_loggers calls each submodule's init_logger(log_dir). The run log dir is named <stage>_log_dir (core_log_dir, adaptive_gen_log_dir, check_cov_log_dir), matching the get_log_dir leaf; a per-instance dir stays log_dir. End-of-run "X saved to <path>" pointers are printed at the init_loggers call site (data outputs first, then "Logs saved to <dir>"). Deletes adaptive_gen/utils.py. Renames: CVERecord -> CVEFixRecord; processed_dataset -> fix_dataset; datasets/cve_records -> raw_cve_records (get_dataset_path keys + all refs + READMEs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
code_test_split(require_test=None) keeps both with- and without-test records in one fix_dataset; a record carries test_patch iff the fix ships a test, so downstream tells them apart by presence instead of the pipeline running twice. --require_test defaults to None (omit for both; true/false still split). Fix the CVEFixRecord schema (was B1): cwe_id:str -> cwe_ids:list[str]; test_patch/test_files are NotRequired. clone-verify keys the test_patch check on presence, not require_test. Unify the three sources behind one fetch-and-cache interface: each has a cache_path it reads and a force flag that rebuilds it. ReposVul caches ReposVul_<lang>_active.jsonl (recent + still-reachable, patch assembled) and drops the per-URL remote_status_cache. Morefixes' build_cache is its former fetch mode. OSV caches osv_fixes.jsonl -- extract fix commits, drop the shas Morefixes/ReposVul already have (the sha-new residual), fetch the rest's .patch, save -- so a run reads the cache and never re-fetches. --morefixes_fetch becomes a general --force '["OSVSource", ...]'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… cleanup
Every source's records() skips shas known already covers before its per-record work -- cheap here (a split) but it saves the finder for the M2/M3 discovery sources (the point of threading known). ReposVul drops its year filter; Morefixes keeps the 2014 cutoff as a local; delete the shared RECENT_YR_CUTOFF and the orphaned remote_status_cache. Constants unified to <SOURCE>_{RAW,URL,ZIP}_PATH + <SOURCE>_CACHE_PATH (no redundant RAW_); OSV regex/func constants lose the wrong _ (reserved for mutable module state); dataset_crawled -> dataset_fetched; clone_repos_and_verify_patches -> ..._fixes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s bug susvibes calls a commit a commit, not a sha: KnownSet.has_commit/normalize_commit/.commits, OSV known_source_commits/FROM_COMMIT, fetch helper param commit. Fix load_file (and the two manual jsonl reads): split jsonl on '\n' only, not str.splitlines() -- which also breaks on \r/\v/U+2028/U+0085 that json.dumps leaves literal in a patch string, corrupting a record (the OSV cache load crash). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…trailing whitespace Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A transient fetch during a cache build can yield a patch whose From header doesn't parse, leaving a short OSV sha as base_commit (breaks clone/verify). Keep only records whose full 40-char commit resolves; records() also guards a stale cache entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mmit General guard in code_test_split (source-agnostic): a short commit (e.g. Morefixes upstream shipping a 7-char commit_sha) can't be reliably cloned/reset, so drop it. Removes the misplaced OSV-only records() guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ttaches advisory cwe_ids Drop _build_cache (which conflated the fetch with caching) and the invented load_or_build helper. Mirror the repo's force-cache convention (check_cov_single): each source has a _fetch() doing the main network work and returning cacheable records; records() is 'if not force and cache exists: load; else: _fetch + save', then filters by known and yields. OSV now fills cwe_ids from the advisory's database_specific.cwe_ids (was []); cache schema gains cwe_ids (needs --force OSVSource to rebuild). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ay=True Per-source log now says 'raw cves' (vs 'records') for the pre-funnel count; net-new stays 'instances'. setup_logger/setup_instance_logger open their FileHandler with delay=True, so a logger that never writes (e.g. core_details.log under --skip_verify) leaves no empty file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cords -> raw_cve core_details.log was oddly named (only the clone-verify stage wrote it) and empty under --skip_verify: remove detail_logger, keep one logger per stage (its per-instance clone/verify failures go to the normal logger). Revert the delay=True FileHandler change. Rename the raw-source dir key raw_cve_records -> raw_cve (it's just CVEs, no plural). Docs: record the run_id-first logs + agent-trajectory structure and the SWE-agent migration TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
run_agent runs the Claude Agent SDK query() to completion on Bedrock, streams the trajectory to a jsonl log live (one message per line), and returns the schema-validated structured_output (options.output_format does the validation/retry). Thin, not a Port class. Verified end-to-end: us.anthropic.claude-sonnet-5 via Bedrock, output_format json_schema -> structured_output, 5-line trajectory streamed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the M2 discovery source: OSV PyPI CVEs with a repo but no fix commit in any source (residual pool = 1983), pinned by the S2 fact-based finder. - clone.py (S4): bare_blobless (git clone --bare --filter=blob:none, owner__repo) + finder_clone (reuse full clone else blobless-bare, RepoLocks-serialized); is_bare_repo predicate (bare clones fail is_git_repo's work-tree check). - find_commit.py (S2): finder_single/finder_threadpool over Sonnet 5 on Bedrock (run_agent, READONLY_TOOLS, native structured output); fact 5-way sweep prompt, repo-scoped, multi-commit rule; per-CVE trajectories under logs/curate/<run_id>/mine/find_commit/. Misses cached as commit="". - sources/osv.py: read_osv_residual (mirrors scratch_m2_dedup) + known_source_cves (MF+RV cve-id skip) + OSVResidualSource (_fetch runs+caches finder, records does commit-only dedup like OSVSource). Registered last in SOURCES; core wires run_id. Validated: pool=1983 confirmed by the code; 5-oracle finder run 4/5 exact tier-1/2 pins + 1 correct multi_commit drop (caught a MoreFixes mapping error), 0 hallucination; blobless clone verified (mssql-django 848K, full history, no blobs/work-tree). Full production run over the 1983 pool (~$1-2k) is cost-gated, not yet run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden the M2 finder for the full 1983-CVE run (patterns from
susvibes-hack-detection's agent_verify.py):
- claude.py: run_agent now returns (output, meta) with meta={cost_usd,
num_turns} (Bedrock populates cost), and RAISES on an aborted run instead
of swallowing to None. Shared is_retryable_error() splits transient
(rate-limit/overload/5xx/subprocess-crash → retry) from terminal
(max turns → don't). Shared AGENT_ENV (ANTHROPIC_MAX_RETRIES=10) +
MAX_BUFFER_SIZE (10MB; the 1MB default crashes the CLI on big git-show
diffs).
- find_commit.py: finder_single retries a retryable failure FINDER_RETRIES=2x
(30/60s backoff), else records `error`; a clean "no fix found" is
commit="" + error=None (final). finder_threadpool shows live cost in its
pbar (`N pinned, M err, $X`). Every result carries `error` and `_meta`.
- sources/osv.py: OSVResidualSource gains `resume` — _fetch(prior) re-runs
only the cached `error` entries, carrying concluded pins/misses through
untouched; records() routes force/resume/load three ways.
- core.py: --resume flag wired onto discovery sources alongside run_id.
Validated: resume done/todo split (no-cost mock); single-CVE finder smoke
after the refactor pins sqlalchemy CVE-2019-7548 with _meta cost=$0.16.
Measured full-pool cost ≈ $365 on Sonnet (avg $0.184/CVE), far below the
earlier $1-2k guess; Haiku A/B only ~20% cheaper but 3x chattier — finder
stays Sonnet.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make --resume mirror --force's input (JSON list of source names) instead of a bare flag, validate that each named source is resume-capable (only discovery sources are), and document the force/resume semantics: - force (all sources): rebuild the cache from scratch — re-run everything. - resume (OSVResidualSource only): re-run just the errored finder outcomes, keeping concluded pins/misses. - both on one source: force wins (rebuild-all supersedes rerun-errors). --resume on a non-capable source now errors with the supported list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
read_osv_residual mistook non-source GitHub URLs for the project repo: - the advisory guard checked group(2) (repo) instead of group(1) (owner), so github.com/advisories/GHSA-... slipped through as repo advisories/GHSA-x (402 of 1983 pool entries → guaranteed clone failures); - pypa/advisory-database and friends (the CVE data repo, not the project) were taken as the source (118 entries). Reject GitHub non-owner path segments (advisories/security/...) and advisory-database repos, and keep scanning references for the real repo. Pool 1983 -> 1828: 77 advisory-only CVEs correctly dropped, ~325 recovered a real source repo that was hidden behind an advisory link. 0 junk owners/repos remain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
30s * 2**attempt (30s, 60s, 120s...) instead of linear 30s*(attempt+1). Equivalent at FINDER_RETRIES=2, correct if retries grow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sources/nvd.py: the M3 haystack + recall-first prefilter (inspect/route/finder land next). - nvd_universe(): NVD (nvd.jsonl, CPE pull) → github-linked ∩ uncovered by OSV/MF/RV = 48,298. github_repos() reuses the advisory/data-URL filters. - prefilter(): recall-first — keep on any Python signal (keyword, PoC repo, unknown/Python-ish repo language incl. Jupyter Notebook); drop only when every referenced repo has a known non-Python language. repo language via GitHub API (get_repo_language, mirrors get_repo_size), threaded + cached per repo. - Validated: recall test on 4197 OSV-Python CVEs → 9.3% drop, but ~60% are correct compiled-language drops (opencv/protobuf C++); true Python miss ~1-2% (the README's accepted irreducible residual: Python fix in a non-Python-primary repo). ~14.5k unique repos need a language lookup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.ipynb-primary repos recover ~0.1% and the pipeline counts only .py as Python
(LANG_EXTENSIONS), so a Jupyter-Notebook keep is inconsistent with the funnel.
PYTHON_LANGS={'python'} was redundant with constants.TARGET_LANG; use that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M3 Phase 2:
- mine/inspect.py (new): the Haiku web-only inspect agent — per NVD CVE, read the
advisory and extract {repo, is_poc_only_ref, vulnerable_files, languages}. Pure
fact extraction, no decision. Parallels find_commit.py. Trajectories land at
logs/curate/<run_id>/mine/inspect/<cve>/.
- sources/nvd.py: route() / fix_languages() (language-configurable: no repo → drop,
target or undeterminable fix → stage2, else drop:non_target). EXT_LANG derives
from LANG_EXTENSIONS.
- Validated on 9 known cases: 9/9 route correct, 0 hallucination. Haiku vs Sonnet
A/B both 9/9; Haiku $0.096/CVE vs Sonnet $0.211 — inspect stays Haiku (~$290 vs
~$630 for the ~3k pool, equal accuracy).
Shared plumbing + tidy-up:
- claude.py: extract run_agent_retrying (retryable-failure retries + exp backoff),
shared by finder and inspect; finder_single now calls it.
- Drop the underscore prefix on module-level functions per code-style (never `_`):
claude _block_to_dict/_message_to_dict/_run, find_commit _miss/_finder_hints,
inspect _inspect_miss.
- Centralize the GitHub URL/identity parsing (COMMIT_URL/REPO_URL/REF_REPO/
FROM_COMMIT/GITHUB_NON_OWNER/GITHUB_NON_REPO) in sources/utils.py; osv.py and
nvd.py import from there (no more source→source dependency).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The GitHub URL-parsing constants (COMMIT_URL/REPO_URL/REF_REPO/FROM_COMMIT/ GITHUB_NON_OWNER/GITHUB_NON_REPO) are used only by the sources (osv, nvd), so they belong at that scope — a new sources/constants.py, mirroring the per-subpackage constants pattern (check_cov/engine/constants.py). GITHUB_HEADERS stays in mine/constants.py: mine.utils uses it too, so its scope is broader than the sources. sources/utils.py keeps only fetch_github_commit_patch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-pulled NVD keeping every usable field (was dropping CWE, ref tags, CVSS):
{id, published, last_modified, vuln_status, cve_tags, desc, cwe_ids, cvss_score,
cvss_severity, cpe_products, refs:[{url, tags}]}. 80% of NVD (98% of the M3
universe) now carries cwe_ids, closing the M3 cwe_ids=[] gap. refs keep their
tags (Patch/Exploit/Vendor Advisory — a fix-finding signal).
nvd.py: github_repos tolerates the new refs shape ({url, tags}, or bare str);
nvd_universe carries cwe_ids/cpe_products into each record. Universe rebuilds at
48,342 (98% with cwe_ids).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e → finder) NVDSource ties the M3 stages into a Source: prefilter (deterministic) → inspect (Haiku, cached) → route → finder (Sonnet, cached). Two agent caches (nvd_inspect.jsonl, nvd_fixes.jsonl) so re-runs never re-pay; --resume re-runs only errored inspects and finds. records() does commit-only dedup + patch split, mirroring OSVResidualSource. Registered last in SOURCES (most speculative). finder_record carries NVD cwe_ids + advisory-named vulnerable_files; finder_hints now feeds those files to the finder as leads. Note: the prefilter policy (should_drop) is still recall-first (60.7% survival → ~$2.8k inspect) — see the UNRECONCILED note in docs; the production policy must be chosen before the full run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The finder model sometimes emits the 2-char string '""' (literal quotes) instead of an empty commit; it was truthy so it counted as a pin and 404'd on patch fetch. Coerce any non-40-hex commit to "" so it is a clean miss. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pt split The core inspect fix: an NVD reference is often a PoC/writeup/advisory-tracking repo, not the vulnerable project's source. inspect now RESOLVES the real source repo (WebSearch added; prompt reframed to find-the-real-repo), working from the raw NVD desc + references — not the prefilter's filtered repo guess (the prefilter is only a conservative screen). Dropped is_poc_only_ref (it was a give-up/drop signal, the wrong design — route drops on repo=="" only). route reverted to repo-based. Prompts split into system + user (hack-detection's division): system = durable role/ task/strategies/constraints; user = the per-CVE data; the output contract stays in the json_schema (output_format). This is what fixed Haiku's flailing — with no system prompt the empty framing let it burn 227-377 msgs and hit max_turns. Tuned on a 10-case hand-verified ground-truth set: Haiku (split) = 10/10 route ($0.17/case) vs Sonnet 10/10 ($0.81, 5x); inspect stays Haiku. max_turns 30->50 for both inspect and finder. finder_hints now feeds vulnerable_files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eal repo-miss cause) Root-caused inspect missing zuoxingdong/lagom (a real Python repo): the SDK subprocess runs in default permission_mode, which blocks Bash `curl` on an approval prompt the headless agent can't answer. So the agent couldn't reach the GitHub search API and fell back to WebFetch(github.com/search) — a JS page that returns a bogus "0 results". It never used WebSearch (deferred in this env, unselected). With permission_mode= "bypassPermissions" the agent curls api.github.com/search/repositories (clean JSON) and resolves the real repo. Fixed on the 10-case hard-Python set: Haiku now 10/10 — finds lagom/zenml/GibsonEnv/ SQLBot/poco-claw/DeepFaceLab/websocket-server/apport, and correctly returns "" for the two with no GitHub source (nRF = Nordic binary, PCRS = Bitbucket-only). $0.10/case. INSPECT_TOOLS is now Bash+WebFetch (WebSearch is unavailable to the agent here); finder gets the same permission_mode so its GitHub-API PR lookups don't block. setting_sources=[] so neither loads the project's interactive settings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cination Migrate inspect/find_commit off bypassPermissions to fail-closed permission_mode="dontAsk": the read-only `tools` set is also pre-approved in `allowed_tools`, no PreToolUse hook — reading 1852 real finder Bash calls showed nothing dangerous to gate and containment is structural (bare clone; clone.py owns writes). Verified dontAsk auto-runs pre-approved tools headless and denies the rest. - inspect: add WebSearch to `tools` for forward-compat (Bedrock silently drops it, so it curls the GitHub API); rename _refs_block -> refs_block (module-level fns take no leading _); inspect_miss meta default matches finder_miss; tighten option comments. - find_commit: reject-only-obvious anti-hallucination clause (no security reasoning; keep the candidate when in doubt); add `additional_commits` to the schema for genuinely multi-commit fixes + clarify one-PR-multi-commit collapse. - claude.py: docstring triage -> inspect + option-list update. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sistency pass
Structure:
- sources/: inspect.py + find_commit.py moved beside the handlers that use them
- post/: secprop.py + check_cov/ grouped as the optional post-stages; each annotates
fix_dataset in place with its detail (`secprop` / `func_coverage`) plus a boolean
gate `record["post"][<stage>]`, so downstream keeps a record via
`all(record["post"].values())` (stage-agnostic)
secprop (new optional stage):
- per-instance security-property + fix-verify agent; an errored run is recorded
(secprop_miss carries `error`) so downstream tells errored from never-ran, and a
plain re-run resumes only the errored ones
- SecpropVerdict StrEnum; PASS_VERDICTS gate
check_cov:
- annotates `func_coverage` in place; PASS_LABELS gate written to post
- gates its own input by `all(post)` (a dropped secprop instance has no dev_tools
version) and skips a versionless instance instead of KeyErroring on the lookup
- rename CoverageLabel.UNKNOWN -> INDETERMINATE (a per-file "analysis inconclusive"
label, kept as a normal instance label)
Downstream (dev_tools / check_cov / adaptive_gen / build_repo):
- filter via `all(post.values())`, dropping the duplicated CoverageLabel thresholds
Other:
- READONLY_TOOLS += WebSearch (dropped on Bedrock, kept for the Anthropic provider)
- unify the agent evidence field (finder fact_evidence, inspect reason -> evidence)
- membership constants use plain {} sets (was frozenset); docstring/comment/logging
consistency fixes (stale collect/process names, current-state comments)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/instances I/O - rename core/agents/ports.py -> sweagent.py; SWEAgentPort now takes a required output_dir and sweagent_dir/conda_env keys; drop get_output_dir/remove_results and the dedicated logs/agent_runs dir (AGENT_RUN_LOG_DIR) - every agent stage writes instances.yaml + reads preds under logs/curate/<run_id>/<stage>/; adaptive_gen runs each iteration in its own iter<N>/ dir, removing the trajectory-deletion hack - remove the test.gen_prologue hint-strategy mechanism, keeping the default patch_secfix prompt only - refresh runs.sh (sonnet-4-5, cost 0.00, test-gen command, sv/sv-env-setup labels, unified relative paths) and the curate / check_cov / test READMEs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the test-synthesis agent's prompt with the secprop-driven v1 template: render the security property (invariant / vulnerable_if / secure_if / irrelevant-differences) instead of the fix diff, so the agent asserts the security consequence and covers every vulnerable_if rather than reverse-engineering fix-presence checks. Deliverable is a stdout JSON pass-map (no secresults.json file). - prompts.py: SEC_TEST_GEN_PATCH_SECFIX_PROMPT_TEMPLATE -> secprop template; drop SECURITY_PATCH. - gen_prologue.py: render record["secprop"] fields; drop SECURITY_PATCH. - utils/agents/configs/test_gen.yaml: instance_template -> v1 (test the attack not the fix, ban incidental coupling, stdout-JSON deliverable). - test/README.md: secprop is now a prerequisite; stdout-JSON deliverable. Blind 3-way judge (15 instances): v1 wins vuln-detection breadth and ties human on secure-impl generality, both over v0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rework the test-synthesis prompt (instance_template + prompts.py) to the gen-v2 design and switch the deliverable to a two-phase, repo-native stdout-JSON contract: - Design: cover every vulnerable_if (symmetric with "any fully secure impl must pass"); assert only the security consequence (no output re-normalization or extra-property requirements); a can't-run test counts as false. - Deliverable: .sv.run_gen_test.sh runs the repo's own test framework to a temp file, then a stdlib parser prints a single-line JSON pass-map -- replaces the old sectests.sh + secresults.json and fixes the stderr-interleave parse failure. GEN_SEC_TEST_CMD updated. - Wire the exit_forfeit tool (forfeit bundle). - Rename repo-visible .susvibes* -> .sv* (SUSVIBES_DIR, security_patch.diff, entry). Blind 4-way judge (human/v0/v1/v2): v2 best of all four on both vuln-detection breadth and secure-impl generality; monotonic v0->v1->v2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run the synthesis agent in-process (SWEAgentPort.run_batch) inside test/gen.py after building the rollback images and assembling the batch, so `python -m susvibes.curate.test.gen` is one command (mirrors adaptive_gen). Add settings/test_gen.yaml (Bedrock model, config, workers). Unify logs under logs/curate/<run_id>/test/gen/<instance_id>/ -- build log (LOG_BUILD) and agent trajectories together -- replacing the split test/<id>/gen_prologue.log vs test/gen_prologue/<id>/ layout. Update validate.no_test's preds path, the READMEs, and drop the now-folded manual test-synthesis command from runs.sh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Refine the Design section of the test-synthesis instance_template, from the 4-way judge analysis (v2's generality was its weaker axis; breadth had a few gaps): - vulnerable_if: "an implementation vulnerable in any variant, to any extent, must fail one of your tests" -- catch genuine vulnerability incl. partial-fix variants, without over-requiring redundant OR-related defenses. - incidental-details: reorganize the flat list into three open-ended groups (mechanism / site-or-level / code-presence, e.g. ... etc.) and ADD coupling to the site or level where the fix's effect appears (nova required LOG.debug specifically; gradio required validation inside build_proxy_request); drop the over-specific parenthetical example. - functional controls: reframe from a hard ban to "functional or non-discriminating tests are not taken into account; if a control slips in, hold it to the same bar as the security tests" -- consistent with validate extracting the discriminating tests. Prompt-only (instance_template); not yet run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found while measuring the text-level -> apply-verify rate for the first time (96.8% across all sources; every net-new estimate had multiplied by a guessed 0.55 hardcoded in a scratch script). Ran end to end as `rv_mf_m1_m2`: 2543 instances, M2 contributing 566. Bugs, each measured on real data: - `clone_github_repo` cloned every repo THREE times — its retry loop had no `return` inside the `try`, so a successful clone fell through, rmtree'd itself and re-cloned. The clone phase drops from 1h53m to ~8min for 969 repos (now also threaded, `clone_repo_threadpool`). - 62 of 1783 records were lost to `reset_to_commit` on a commit no ref reaches — a squash-merged PR head or deleted branch, which `git clone` never fetches. All 62 return 200 from the API; `fetch_commit` recovers them. `reset_to_commit` now fails once in 2645. - 4 more were lost to a git-lfs smudge failing a whole `git reset --hard` (jumpserver's 74MB GeoIP DB); the funnel only reads source text, so the mine entry point sets GIT_LFS_SKIP_SMUDGE. - `expand_tests` never ran in combined mode (`--require_test` omitted): the gate was `if require_test:` and None is falsy, so every with-test record kept a raw `test_patch` instead of a test mask. The function now passes no-test records through untouched and the gate is gone. - Three of four GitHub call sites read a throttled response as "gone": `remotely_active` cached the drop permanently, `get_repo_size` silently disabled the size gate, `get_repo_language` would burn the hourly quota. All now go through `github_get`, which waits out `Retry-After` / `X-RateLimit-Reset` and treats a 403/429 with neither header as a secondary rate limit. Dedup gains a CVE key (`KnownSet.has`): commit alone missed a fix backported across maintenance branches, where every sha genuinely differs. 4.8% of instances were that, almost all ReposVul (Morefixes drops multi-commit CVEs upstream). The three agent stages now share one per-item result cache (`load_agent_result` / `save_agent_result`, path chosen by the caller like `run_agent`'s `log_path`) and one ladder: a plain run reuses whatever is cached including an errored outcome, `--resume` re-runs only the errored, `--force` everything. The finder had no per-item cache at all, so a crash mid-batch lost the whole pool's work. secprop gains `--resume`. `clone.py` now owns every clone the pipeline makes (`full_clone` moved in from `curate/utils/common.py`), over one `clone_into` primitive, with the blobless tier inside the configured root as `.sv.blobless/` rather than reaching out to a sibling nobody configured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/mnt/data2 is 100% full with 122G left, which the M2/M3 clone volume would exhaust; the 182G of clones now live on the roomier device. Machine-local, so drop this commit rather than merge it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Expands
mine's CVE→fix-commit sources beyond the current Morefixes + ReposVul, adding three additive sources (M1→M2→M3), plus the shared agent + clone infrastructure they need. Design, evidence, and measured yields live indocs/mine-filters/.Sources
OSVSource(deterministic, no agent): OSV/GHSA handler over the commit-new residual vs Morefixes/ReposVul. ~+126 instances, measured.OSVResidualSource+ fact-based finder (Sonnet, read-only): for OSV CVEs with a repo but no commit, an agent pins the single upstream fix commit by fact (grep CVE/GHSA/PR ids, PR merge-sha, component+fixed-version) on a blobless-bare clone. ~70–150 net-new (funnel-limited), cost-gated ~$365–700.NVDSource:prefilter → inspect (Haiku, resolves the real source repo behind PoC/writeup refs) → route → finder. Re-estimated ~161 (policy C) / ~283 (policy A ceiling), cost-gated ~$800 / ~$3.1k.Shared infrastructure
core/agents/claude.py— thin Claude Agent SDK glue (run_agent+ retries + trajectory logging), Bedrock.inspect/find_commitrunpermission_mode="dontAsk"with the read-onlytoolsset also pre-approved inallowed_toolsand no PreToolUse hook (containment is structural — bare clone;clone.pyowns writes). See.claude/rules/agent-dispatch.md.additional_commitsfor genuine multi-commit fixes.Validation
inspect/finderwere re-tested against independent ground truth (established per-CVE by read-only sub-agents): finder pinned 5/5 real-fix hard cases exactly, returned correct misses, and the anti-hallucination clause caught the reverted/functional false pin. The GT round also corrected 4 bad entries in the tuning oracle.Not in scope / gated
Production discovery runs (M2 ~$365–700, M3 ~$800) are cost-gated and not run. The downstream funnel (
code_test_splitA1 split-reject; A2 multi-commit) still bounds final yield — seedocs/mine-filtersIssues A/B/C.🤖 Generated with Claude Code