feat(orchestrator): intent-fidelity gate — "A or better", with percentile

[JustinJLeopard]

What — "the intent is A or better", with percentile

Adds a post-execution intent-fidelity gate. Today summary.status is decided
purely by task counts: every task done → complete. But a plan can decompose a
goal into tasks that all pass while drifting from the real intent, so "all tasks
done" masquerades as "achieved your goal". This gate closes that gap.

For each run it judges whether the OUTCOME achieved the user's original intent and
scores it as a 0-100 percentile with a letter grade and a verdict against a
configurable "A" bar (JUSTAI_INTENT_BAR, default 90):

• met — fidelity ≥ bar (the intended outcome "A" was achieved)
• exceeded — genuinely better than the intent ("or better")
• missed — fell short

Honest downgrade (the point)

A task-complete run judged to have missed the intent is downgraded
complete → partial. The gate only downgrades — it never upgrades a partial
run, so a real task failure is never hidden. Honesty over performance, both ways.

Design / safety

• justai/intent_fidelity.py — score_fidelity(): primary LiteLLM judge, with a
  deterministic completion-based heuristic fallback that is honest about its
  own limits (no LLM → it does NOT fabricate a confident "missed" from fuzzy
  tokens; it scores from completion and marks source="heuristic").
• Best-effort: any scoring error degrades to the heuristic and never fails a
  run (try/except in the orchestrator stage).
• Flag JUSTAI_FIDELITY_GATE (default on) restores prior task-count-only
  semantics when off.
• When the model proxy is down, fidelity == completion → no status change vs
  prior behavior. The only behavior change is when a live judge flags a
  task-complete run as intent-missed — exactly the intended honesty win.

Tests

12 new (tests/test_intent_fidelity.py): percentile→grade mapping, heuristic
fallback, mocked LLM judge (missed/met/exceeded), fallback-on-error, and the
synthesizer downgrade integration. Full suite: 407 passed, +12 new, 0 new
failures (7 failures on this branch are pre-existing README-drift and a local
backend-unavailable E2E smoke — verified identical on clean HEAD with this work
stashed).

⚠ Behavior-semantics note for Justin

This changes when a run reports "complete". You approved building it ("A or
better, good idea. With percentile."). Flagging it here per the git-ops rubric so
the semantics change is explicit; squash-merge target is demo-build.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
justai-demo	Ready	Preview	Aug 9, 2026 1:26pm

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[Copilot]

Pull request overview

Introduces an intent-fidelity gate that scores post-execution outcome fidelity (0–100 percentile + grade) and honestly downgrades task-complete runs from complete → partial when the outcome misses the original intent, while remaining best-effort (non-fatal) with a heuristic fallback.

Changes:

• Add justai/intent_fidelity.py with LiteLLM judge + deterministic completion-based heuristic fallback and verdict/grade mapping.
• Thread fidelity results through orchestrator.run() → synthesizer.synthesize() → RunSummary/OrchestrationResult, including downgrade logic.
• Add comprehensive unit + integration tests for scoring, fallback behavior, and the downgrade semantics.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_orchestrator_pipeline.py	Mocks fidelity scoring in the orchestrator pipeline tests and asserts fidelity fields are surfaced on results.
tests/test_intent_fidelity.py	Adds unit + integration tests for grading, LLM judge path (mocked), heuristic fallback, and synthesizer downgrade behavior.
justai/synthesizer.py	Extends RunSummary with fidelity fields and applies downgrade-on-missed-intent when tasks were otherwise complete.
justai/orchestrator.py	Adds a gated Stage 6 that computes intent fidelity (best-effort) and passes it into synthesis; surfaces fidelity fields in return object/trace metadata.
justai/intent_fidelity.py	New module implementing fidelity scoring, verdicts, grades, LiteLLM call, and heuristic fallback.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JustinJLeopard]

Desktop Codex exact-head review at 3aa7c157b2d8e860273bfe9cfe87aef501a441d4: HOLD / request changes. The new tests are useful, but the gate still permits the false-completion class it is meant to close.

1. The judge-failure path is fail-open. score_fidelity converts an unavailable/invalid judge into completion ratio; all tasks done becomes 100 / met / a_or_better=True. The synthesizer then keeps complete, while RunSummary, OrchestrationResult, formatted output, and stored memory do not preserve source="heuristic". Programmatic consumers therefore cannot distinguish semantic verification from the old task-count behavior. Represent judge failure as explicit unverified/UNKNOWN and do not claim intent-met/complete from a completion-only fallback.
2. Validate the judge response before constructing a verdict. At _make, valid JSON { "fidelity": "NaN" } clamps to 100 / met under Python’s min/max behavior, and { "fidelity": 99, "better_than_intent": "false" } becomes EXCEEDED because non-empty strings are truthy. Require a finite numeric score in range and an actual JSON boolean; schema failures must become unverified, not success.
3. The current local profile and PR config: llama.cpp local profile for the pipeline #19 make this head fail at import. Both contain JUSTAI_INTENT_BAR=90 # ...; JustAi’s custom .env loader preserves the inline comment, while this module parses it with float(...) at import. Exact reproduction raises ValueError before the gate try/except. Fix the profile/loader and make threshold parsing bounded and explicit, then run a combined feat(orchestrator): intent-fidelity gate — "A or better", with percentile #18+config: llama.cpp local profile for the pipeline #19 config probe.
4. The LLM sees self-reported task strings truncated to 300 characters, not independently materialized outcome evidence, and those strings are an untrusted prompt-injection surface. Ground the verdict in verification receipts/artifacts (with an explicit unknown path when evidence is absent). Also call the 0–100 value a judge score unless/until it is calibrated against a reference distribution; it is not presently a percentile.

Networkless differential testing at this exact head produced 407 passes with the same 7 known failures/14 subtests as the base, but GitHub currently has only Vercel checks—no Python CI. Tests/source merge, activation, and live acceptance remain separate gates.