Skip to content

Add tests for #1493#1495

Draft
prompt-driven-github[bot] wants to merge 1 commit into
mainfrom
test/issue-1493
Draft

Add tests for #1493#1495
prompt-driven-github[bot] wants to merge 1 commit into
mainfrom
test/issue-1493

Conversation

@prompt-driven-github

Copy link
Copy Markdown
Contributor

Summary

Adds tests based on the requirements in #1493.

Test Files

  • tests/test_checkup_negative_ac.py

Test Coverage

  • Total Tests: 14
  • Framework: pytest + unittest.mock
  • Status: passing (14/14)
  • Test Plan Coverage: 14/14 planned cases implemented (100%)

What These Tests Verify

These tests verify that pdd checkup correctly detects missing negative/adversarial acceptance-criteria coverage:

  1. Checkup fails when negative AC not tested (3 tests) — issue_aligned=falsesuccess=False, no push when adversarial coverage is missing
  2. Checkup passes when negative AC is covered (2 tests) — issue_aligned=truesuccess=True when adversarial near-match cases (gpt-4, openai/gpt-4o, azure/gpt-4.1-mini) are covered
  3. Happy-path tests alone do not satisfy alignment (2 tests) — Step 5 passing + Step 7 issue_aligned=false → orchestrator still fails; passing tests alone are not sufficient
  4. Adversarial probe language in prompts (3 tests) — Static content assertions that agentic_checkup_step7_verify_LLM.prompt references issue_aligned and checkup_review_loop.py contains adversarial probe family instructions
  5. Near-match pricing regression fixture (4 tests) — End-to-end regression guard for the original Add output-inclusive token and cost estimation primitives #1357/chore: PDD sync for #1357 #1361 failure: gpt-4 must not borrow pricing from near-match models

Contract Test Summary

N/A - no OpenAPI spec found

Accessibility Audit Summary

N/A - not a web test

Manual Testing Summary

N/A - all tests are automated and deterministic with no live LLM credentials required

Test Execution

python -m pytest tests/test_checkup_negative_ac.py -q

Next Steps

  1. Review the generated tests
  2. Run tests locally to verify
  3. Adjust tests if needed
  4. Mark PR as ready for review

Closes #1493


Generated by PDD agentic test workflow (18-step)

…bing

Adds tests/test_checkup_negative_ac.py with 14 test cases across 5 classes
covering all scenarios from issue #1493:
- Checkup fails when negative AC is not tested
- Checkup passes when negative AC is covered
- Happy-path tests alone do not satisfy issue alignment
- Adversarial probe language verified in prompts
- Near-match pricing regression fixture (gpt-4 vs openai/gpt-4o)

All tests are deterministic and require no live LLM credentials.

Closes #1493

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@prompt-driven-github prompt-driven-github Bot mentioned this pull request Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test: #1490

1 participant