Fix false-positive SecPass: positive-evidence rule, evidence model, real-log regression suite - #17
Open
compaluca wants to merge 6 commits into
Open
Fix false-positive SecPass: positive-evidence rule, evidence model, real-log regression suite#17compaluca wants to merge 6 commits into
compaluca wants to merge 6 commits into
Conversation
Structured expect blocks for parser counts and SecPass decisions on real log fixtures. No production SecPass changes yet; tests fail until susvibes.runners and evaluate_run_from_logs land in Phase 2.
Port the language-agnostic evaluation engine from the reference harness: - susvibes/runners/: TestRunnerAdapter base, pytest (Fix 9 _parse_counts, per-test extraction), django, and detect_runner/FallbackAdapter/Zope. - tasks.py: evaluate_run_from_logs as the single SecPass/FuncPass entry point; _decide_pass + _resolve_sec (variant failures, likely_passed, positive_sec_evidence); Task.evaluate wired to it. Sec-variant failures gate before the abort check (R01); no-summary vs killed disambiguation (R03-R07). - env.py: create_container(environment=...) so PYTEST_ADDOPTS=-v reaches the container for per-test evidence. - constants.py: PREMATURE_ABORT_PATTERNS (maxfail banner).
…ce_stats Make the regression suite hermetic and broaden Fix 11 coverage: - Namespace fixtures under tests/fixtures/v1/ (tied to the endor v1 dataset). - Vendor a minimal in-repo slice (instance_id + test_patch + expected_failures) for the catalog instances into fixtures/v1/dataset_records.jsonl (~96 KB), and resolve records from it first so upstream dataset drift can't break the suite. The external v1 path / stale datasets/default are gap-fillers only. - Add count_only regression fixtures R14 (plone.namedfile, PASS), R15 (zope.testrunner, FAIL), R16 (lshell unittest, FAIL) — the previously untested evidence tier, across non-pytest runners. - get_summary: aggregate evidence tiers into evidence_stats (reporting only, no decision change) for SecPass auditability.
Port test_evaluation_logic.py selectively, adapted to this branch's API (_decide_pass returns a 5-tuple with positive_sec_evidence; reasons include no_positive_sec_evidence / no_test_summary). 31 tests covering: - _decide_pass 3-tier evidence matrix: full / partial / count_only (incl. FallbackAdapter), func -> empty evidence, abort/no-summary, and the sec-failure-before-abort gate order (R01-style). - SessionResult helpers; extract_added_tests diff parser; PytestAdapter and DjangoTestAdapter per-test extraction + match_test; get_verbose_env. - get_summary evidence_stats aggregation (reporting only, ratios unchanged).
5 tasks
… current layout Brings the branch up to date with main (25 commits: eval/curate refactor, core package split, dataset_id, cve_fix_date, etc.) and ports the positive-evidence SecPass fix onto the current trunk: - Remove susvibes/tasks.py; graft _resolve_sec / _decide_pass / evaluate_run_from_logs into susvibes/eval/task.py. Route only the count sec run (default 186) through the positive-evidence path with PYTEST_ADDOPTS=-v verbose injection; func and gen_sec keep the current PassFailure path. - core/env.py: logs_parser / logs_checker read-only properties over logs_handler["count"]; container environment= (auto-merged) for verbose. - get_summary keeps the current schema (func_pass / sec_pass / num_candidates) and adds evidence_stats. - tests: fix imports; vendor the 16 R-case env specs into tests/fixtures/v1/components.json (components.json was dropped in the current layout); adapt make_env_for_instance + the two get_summary tests. pytest tests/ -> 61 passed / 3 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix false-positive SecPass: positive-evidence rule, evidence model, real-log regression suite
Supersedes #12. Same goal — make SecPass a positive claim rather than a
negative one — but rebuilt cleanly from
mainagainst a TDD catalog of realcaptured test logs, with a single production entry point, hermetic tests, and
an explicit evidence model. Full detail in
CHANGES.md.Why a new PR instead of continuing #12
commits,
.venv/message-file churn).main, test-driven againstreal
sec.txtlogs, with one decision entry point and no external pathdependency.
mlflow, saltstack, tensorflow, jinja-stays-pass) — by replaying saved logs,
so no Docker is needed to verify.
What's included
susvibes/runners/—TestRunnerAdapter+SessionResult; Pytest /Django / Fallback adapters;
detect_runner()from the Dockerfile CMD.tasks.evaluate_run_from_logs— one decision path shared byTask.evaluateand the tests._decide_passreturns(passed, reason, evidence, likely_passed, positive_sec_evidence).full/partial/count_only) recorded inreport.json— descriptive only, never gates the decision — plusget_summaryevidence_statsaggregation for auditability.endor-v1 dataset slice (
test_patch+expected_failures), so the suite isimmune to upstream dataset drift and needs no external path.
_decide_passevidence matrix,SessionResult,the diff parser, adapter per-test extraction +
match_test,evidence_stats.Deliberate deviations from #12 (documented in
CHANGES.md)no_test_summaryunless a hard-kill/timeout marker ispresent →
session_aborted:CRASH(R03–R05 vs R06–R07).positive_sec_evidence/no_positive_sec_evidenceflag passes that lack averified PASS (R09).
Not included (deferred, with justification in
CHANGES.md)detect_runner_from_outputwiring — quantified at 0/343 impact on thereal
count_onlyset, so deliberately not wired.94/343
count_onlyruns and surfaces 70 hidden sec failures; recommendednext step.
get_verbose_command/ verbose-flag injection.Test plan
evaluate_run_from_logstests/test_evaluation_logic.py)python -m pytest tests/→ 61 passed, 3 skipped (R14–R16 opt out ofthe pytest-only count check; decisions still asserted via the catalog)