test(presets): probe CLI/docs parity for all four presets#49
Conversation
Add parity tests asserting that baseline, strict, legacy, and progressive all appear in `interlocks presets` output with correct descriptions and gate values. Tighten the rejection-message check to include `progressive`. Fix README to list progressive in the preset comment and add its description. - tests/test_cli.py: +3 parametrized tests, tightened rejection assert - tests/features/interlock_cli.feature: new presets-parity scenario - README.md: add progressive to preset comment and bullet list
There was a problem hiding this comment.
Code Review
This pull request introduces a new 'progressive' preset, which acts as an autopilot ratchet by using blocking gates while automatically advancing thresholds via a baseline file. The changes include documentation updates, new Gherkin scenarios for CLI parity, and expanded unit tests for preset configurations. Feedback focuses on increasing test coverage for the 'strict' and 'progressive' presets by verifying additional configuration keys such as 'enforce_behavior_attribution' and 'run_acceptance_in_check'.
| # strict: all blocking gates on, mutation incremental | ||
| ("strict", "enforce_crap", True), | ||
| ("strict", "enforce_mutation", True), | ||
| ("strict", "run_mutation_in_ci", True), | ||
| ("strict", "mutation_ci_mode", "incremental"), | ||
| ("strict", "run_acceptance_in_check", True), | ||
| ("strict", "require_acceptance", True), | ||
| ("strict", "coverage_min", 90), |
There was a problem hiding this comment.
The strict preset explicitly enables enforce_behavior_attribution. It should be included in this parity test to ensure the preset is correctly configured.
| # strict: all blocking gates on, mutation incremental | |
| ("strict", "enforce_crap", True), | |
| ("strict", "enforce_mutation", True), | |
| ("strict", "run_mutation_in_ci", True), | |
| ("strict", "mutation_ci_mode", "incremental"), | |
| ("strict", "run_acceptance_in_check", True), | |
| ("strict", "require_acceptance", True), | |
| ("strict", "coverage_min", 90), | |
| # strict: all blocking gates on, mutation incremental | |
| ("strict", "enforce_crap", True), | |
| ("strict", "enforce_behavior_attribution", True), | |
| ("strict", "enforce_mutation", True), | |
| ("strict", "run_mutation_in_ci", True), | |
| ("strict", "mutation_ci_mode", "incremental"), | |
| ("strict", "run_acceptance_in_check", True), | |
| ("strict", "require_acceptance", True), | |
| ("strict", "coverage_min", 90), |
| # progressive: blocking gates on, permissive floors (ratcheted at runtime) | ||
| ("progressive", "enforce_crap", True), | ||
| ("progressive", "enforce_mutation", True), | ||
| ("progressive", "run_mutation_in_ci", True), | ||
| ("progressive", "mutation_ci_mode", "incremental"), | ||
| ("progressive", "run_acceptance_in_check", True), | ||
| ("progressive", "require_acceptance", True), | ||
| ("progressive", "coverage_min", 0), # floor; ratcheted by baseline.json |
There was a problem hiding this comment.
The progressive preset also explicitly enables enforce_behavior_attribution. Including it here ensures full coverage of the preset's intended blocking behavior.
# progressive: blocking gates on, permissive floors (ratcheted at runtime)
("progressive", "enforce_crap", True),
("progressive", "enforce_behavior_attribution", True),
("progressive", "enforce_mutation", True),
("progressive", "run_mutation_in_ci", True),
("progressive", "mutation_ci_mode", "incremental"),
("progressive", "run_acceptance_in_check", True),
("progressive", "require_acceptance", True),
("progressive", "coverage_min", 0), # floor; ratcheted by baseline.json| assert re.search(r"^\s*preset\s+progressive\s*$", out, re.MULTILINE), out | ||
| assert re.search(r"^\s*enforce_crap\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out | ||
| assert re.search(r"^\s*enforce_mutation\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out | ||
| assert re.search(r"^\s*run_mutation_in_ci\s+True \(preset-derived\)\s*$", out, re.MULTILINE), ( | ||
| out | ||
| ) | ||
| assert re.search(r"^\s*require_acceptance\s+True \(preset-derived\)\s*$", out, re.MULTILINE), ( | ||
| out | ||
| ) |
There was a problem hiding this comment.
This integration test verifies that the progressive preset correctly wires blocking gates. It should also verify enforce_behavior_attribution and run_acceptance_in_check, which are key parts of the preset's configuration.
assert re.search(r"^\s*preset\s+progressive\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*enforce_crap\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*enforce_behavior_attribution\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*enforce_mutation\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*run_mutation_in_ci\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*run_acceptance_in_check\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out
assert re.search(r"^\s*require_acceptance\s+True \(preset-derived\)\s*$", out, re.MULTILINE), out…nd progressive presets Address gemini-code-assist review comments on PR #49: - Add `("strict", "enforce_behavior_attribution", True)` to test_preset_defaults_key_values parametrize - Add `("progressive", "enforce_behavior_attribution", True)` to test_preset_defaults_key_values parametrize - Add assertions for enforce_behavior_attribution and run_acceptance_in_check in test_progressive_preset_enables_blocking_gates_when_configured integration test
Summary
progressivepreset was missing from test assertions and README docsbaseline|strict|legacyto includeprogressivecli-presets-parityininterlock_cli.featureprogressiveto preset comment and bullet listFiles changed
tests/test_cli.py— 3 new parametrized/integration tests, 2 tightened assertions,Presetimporttests/features/interlock_cli.feature— newcli-presets-parityscenarioREADME.md— progressive preset in comment and description listTest plan
uv run pytest -q tests/test_cli.py tests/step_defs/test_interlock_cli.py— 69 passed