Skip to content

[Good first issue]: Add a retrieval-poisoning Failure Zoo specimen #22

Description

@thangldw

User outcome

A contributor can add one inspectable retrieval-poisoning regression to the RAG Failure Zoo and see RAGOps block it for a named, deterministic reason.

Scope

  • Add one synthetic case or response override under scenarios/japanese_troubleshooting/.
  • Preserve a passing baseline and add one intentionally poisoned candidate.
  • Add the specimen to docs/evaluation/benchmark.md.
  • Add expected and negative tests.

Acceptance criteria

  • The baseline passes.
  • The poisoned candidate exits 2 with a named finding or gate.
  • The fixture contains no real secrets, customer data, or live URLs.
  • ruff check . and PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest -q pass.

Non-goals

Do not claim that one synthetic specimen proves production security or broad prompt-injection resistance. Do not change a public schema or default gate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions