Skip to content

test(presets): probe CLI/docs parity for all four presets#49

Open
0xjgv wants to merge 2 commits into
mainfrom
unit8-preset-docs-parity
Open

test(presets): probe CLI/docs parity for all four presets#49
0xjgv wants to merge 2 commits into
mainfrom
unit8-preset-docs-parity

Conversation

@0xjgv

@0xjgv 0xjgv commented May 11, 2026

Copy link
Copy Markdown
Owner

Summary

  • Probes found real drift: progressive preset was missing from test assertions and README docs
  • Tightened rejection-message check from baseline|strict|legacy to include progressive
  • Added 3 new tests: descriptions × 4, defaults key-values × 21, progressive blocking-gates integration
  • Added BDD scenario for cli-presets-parity in interlock_cli.feature
  • Fixed README: added progressive to preset comment and bullet list

Files changed

  • tests/test_cli.py — 3 new parametrized/integration tests, 2 tightened assertions, Preset import
  • tests/features/interlock_cli.feature — new cli-presets-parity scenario
  • README.md — progressive preset in comment and description list

Test plan

  • uv run pytest -q tests/test_cli.py tests/step_defs/test_interlock_cli.py — 69 passed
  • Pre-commit (ruff fix, format, basedpyright) — all green

Add parity tests asserting that baseline, strict, legacy, and progressive
all appear in `interlocks presets` output with correct descriptions and gate
values. Tighten the rejection-message check to include `progressive`. Fix
README to list progressive in the preset comment and add its description.

- tests/test_cli.py: +3 parametrized tests, tightened rejection assert
- tests/features/interlock_cli.feature: new presets-parity scenario
- README.md: add progressive to preset comment and bullet list

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new 'progressive' preset, which acts as an autopilot ratchet by using blocking gates while automatically advancing thresholds via a baseline file. The changes include documentation updates, new Gherkin scenarios for CLI parity, and expanded unit tests for preset configurations. Feedback focuses on increasing test coverage for the 'strict' and 'progressive' presets by verifying additional configuration keys such as 'enforce_behavior_attribution' and 'run_acceptance_in_check'.

Comment thread tests/test_cli.py
Comment on lines +321 to +328
# strict: all blocking gates on, mutation incremental
("strict", "enforce_crap", True),
("strict", "enforce_mutation", True),
("strict", "run_mutation_in_ci", True),
("strict", "mutation_ci_mode", "incremental"),
("strict", "run_acceptance_in_check", True),
("strict", "require_acceptance", True),
("strict", "coverage_min", 90),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The strict preset explicitly enables enforce_behavior_attribution. It should be included in this parity test to ensure the preset is correctly configured.

Suggested change
# strict: all blocking gates on, mutation incremental
("strict", "enforce_crap", True),
("strict", "enforce_mutation", True),
("strict", "run_mutation_in_ci", True),
("strict", "mutation_ci_mode", "incremental"),
("strict", "run_acceptance_in_check", True),
("strict", "require_acceptance", True),
("strict", "coverage_min", 90),
# strict: all blocking gates on, mutation incremental
("strict", "enforce_crap", True),
("strict", "enforce_behavior_attribution", True),
("strict", "enforce_mutation", True),
("strict", "run_mutation_in_ci", True),
("strict", "mutation_ci_mode", "incremental"),
("strict", "run_acceptance_in_check", True),
("strict", "require_acceptance", True),
("strict", "coverage_min", 90),

Comment thread tests/test_cli.py
Comment on lines +334 to +341
# progressive: blocking gates on, permissive floors (ratcheted at runtime)
("progressive", "enforce_crap", True),
("progressive", "enforce_mutation", True),
("progressive", "run_mutation_in_ci", True),
("progressive", "mutation_ci_mode", "incremental"),
("progressive", "run_acceptance_in_check", True),
("progressive", "require_acceptance", True),
("progressive", "coverage_min", 0), # floor; ratcheted by baseline.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The progressive preset also explicitly enables enforce_behavior_attribution. Including it here ensures full coverage of the preset's intended blocking behavior.

        # progressive: blocking gates on, permissive floors (ratcheted at runtime)
        ("progressive", "enforce_crap", True),
        ("progressive", "enforce_behavior_attribution", True),
        ("progressive", "enforce_mutation", True),
        ("progressive", "run_mutation_in_ci", True),
        ("progressive", "mutation_ci_mode", "incremental"),
        ("progressive", "run_acceptance_in_check", True),
        ("progressive", "require_acceptance", True),
        ("progressive", "coverage_min", 0),  # floor; ratcheted by baseline.json

Comment thread tests/test_cli.py
Comment on lines +364 to +372
assert re.search(r"^\s*preset\s+progressive\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*enforce_crap\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*enforce_mutation\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*run_mutation_in_ci\s+True \(preset-derived\)\s*$", out, re.MULTILINE), (
out
)
assert re.search(r"^\s*require_acceptance\s+True \(preset-derived\)\s*$", out, re.MULTILINE), (
out
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This integration test verifies that the progressive preset correctly wires blocking gates. It should also verify enforce_behavior_attribution and run_acceptance_in_check, which are key parts of the preset's configuration.

    assert re.search(r"^\s*preset\s+progressive\s*$", out, re.MULTILINE), out
    assert re.search(r"^\s*enforce_crap\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
    assert re.search(r"^\s*enforce_behavior_attribution\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
    assert re.search(r"^\s*enforce_mutation\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
    assert re.search(r"^\s*run_mutation_in_ci\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
    assert re.search(r"^\s*run_acceptance_in_check\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
    assert re.search(r"^\s*require_acceptance\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out

…nd progressive presets

Address gemini-code-assist review comments on PR #49:
- Add `("strict", "enforce_behavior_attribution", True)` to test_preset_defaults_key_values parametrize
- Add `("progressive", "enforce_behavior_attribution", True)` to test_preset_defaults_key_values parametrize
- Add assertions for enforce_behavior_attribution and run_acceptance_in_check in
  test_progressive_preset_enables_blocking_gates_when_configured integration test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant