Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2571,8 +2571,8 @@ All remaining open issues are enhancements (no bugs as of 2026-07-18). Prioritiz
### #353 — Eval-gated prompt profile canary and rollback — CLOSED (slice 1)
**COMPLETE (PR #374, 2026-07-18)**: `mapify prompt-profile list` command shipped. Manifest format: `.map/prompt-profiles/<id>/manifest.json` (required: `id`, `title`, `version`; optional: `description`, `owner`, `targets`, `eval_requirements`, `rollback_notes`). Active pointer: `.map/prompt-profiles/active.json` (`{"active": "<id>"|null}`). `list` command shows table with ID/title/version/status/description and active marker; `--json` for machine-readable output; stale-pointer warning when `active.json` names a non-existent profile. 20 tests in `tests/test_prompt_profile_commands.py`. Future slices: `diff`, `activate`, `rollback`, `report` commands + integration with #291 preset composition as rendering substrate.

### #339 — GRACE semantic code-contract anchor eval
Eval harness comparing inline code contracts vs prompt-injected vs no-anchor variants for bug-fix workflows. Variants: baseline, inline (LEX/MIN), prompt-injected (INJ), stale (LIE). First slice: fixture structure + report schema (JSON). No external model calls required for slice 1.
### #339 — GRACE semantic code-contract anchor eval — CLOSED (slice 1)
**COMPLETE (PR #375, 2026-07-18)**: `src/mapify_cli/grace_eval/` package shipped. Pure data + deterministic logic layer; no external model calls. Contracts: `GraceFixture` (fixture.json schema), `VariantRunRecord` (per-run JSONL row), `VariantAggregate` (rolled-up stats with vs-baseline deltas), `SweepFinding` (stale/contradictory contract detection result), `GraceReport` (root report object with schema_version). Six variants: `baseline`, `inline`, `lex`, `min`, `inj`, `lie`; variant sets classify each as `CODE_LOCAL`, `PROMPT_INJECTED`, or `NO_ANCHOR`. `aggregate_runs()` computes success_rate, mean_retries, mean tokens, stale_detections, and trajectory_delta_note (improvement/regression/tie/no_baseline) vs a baseline aggregate. `sweep_source()` and `sweep_variant_sources()` perform heuristic stale-anchor detection using `# CONTRACT:` / `# ANCHOR:` comment patterns and 7 contradiction signal pairs with an 8-line lookahead. 97 tests in `tests/test_grace_eval_schema.py` + `tests/test_grace_eval_sweep.py`. Future slices: CLI entry point (`mapify grace-eval run/report`), real token-log replay, model-call dispatcher, HTML report viewer.

---

Expand Down
42 changes: 42 additions & 0 deletions src/mapify_cli/grace_eval/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
"""grace_eval — GRACE semantic code-contract anchor evaluation.

Exports the shared data contracts and sweep utilities. This module is a
pure data + deterministic-logic layer for slice 1; no external model calls,
no I/O, no subprocess, no clock access (INV-2/INV-3).
"""

from __future__ import annotations

from mapify_cli.grace_eval.schema import (
VARIANT_NAMES,
CODE_LOCAL_VARIANTS,
PROMPT_INJECTED_VARIANTS,
NO_ANCHOR_VARIANTS,
GraceFixture,
GraceReport,
SweepFinding,
VariantAggregate,
VariantRunRecord,
aggregate_runs,
make_run_id,
)
from mapify_cli.grace_eval.sweep import (
sweep_source,
sweep_variant_sources,
)

__all__ = [
"VARIANT_NAMES",
"CODE_LOCAL_VARIANTS",
"PROMPT_INJECTED_VARIANTS",
"NO_ANCHOR_VARIANTS",
"GraceFixture",
"GraceReport",
"SweepFinding",
"VariantAggregate",
"VariantRunRecord",
"aggregate_runs",
"make_run_id",
"sweep_source",
"sweep_variant_sources",
]
Loading