Skip to content

Implement meter-found semantic ambiguity queue from factorized-β runs #76

Description

@usurobor

Filing role: κ (records operator directive, 2026-07-05).
Successor to: the factorized-β wave (#73) — which returned a terminal FAIL (A1/A2/A3 missed; B1/B2/B3 held).
Line: defect-harvesting / calibration — NOT consistency-improvement. See docs/beta/governance/DEFECT-HARVESTING.md and docs/beta/governance/METER-LOOP-DECISION.md.

Why (the sharpened result)

The factorized-β experiment removed three freedoms — the engine chose the same β loci for every witness, the LLM answered only supports | contradicts | insufficient, and the engine (not the LLM) counted and aggregated. A3 still failed: independent witnesses disagree on the local semantic relation itself — not what to inspect, not how to count, not how to aggregate. That localizes the residual variance to genuine semantic judgment over under-specified citation relations (a source line carries multiple claims; a target span partially supports; supports/insufficient and contradicts/not-supported are threshold calls), plus ordinary model non-determinism.

Conclusion (operator, 2026-07-05): we can improve operational reliability but probably cannot make same-route LLM semantic judgment intrinsically stable enough to be a scalar authority. The fix is to change what the LLM is allowed to decide and treat remaining disagreement as a measured property of the artifact — the LLM becomes a defect proposer + ambiguity detector, and a steward / cross-route calibration adjudicates. This is a third, stronger negative for consistency-improvement: even fixed-locus local adjudication did not cross the gate. No v3.2.5 prompt/schema tweak.

Scope

  • Parse the factorized-β per-locus votes from a run's artifacts (.tsc/fb/… inventory + responses).
  • Compute agreement / entropy per locus.
  • Enqueue loci below an agreement threshold into a semantic-ambiguity queue (proposed .tsc/meter/ambiguity-queue.jsonl).
  • Triage each into content-defect | ambiguous | witness-error | clean.
  • Create gold-label calibration records for adjudicated loci (source span, target span, expected verdict, rationale, known tempting wrong reading).
  • Report defect-yield + ambiguity metrics (confirmed defects, ambiguous-locus rate, disagreement rate, false-positive rate, rediscovery-after-fix) — not Coh-consistency.

Ambiguity record (proposed)

{
  "locus_id": "beta.link.0017",
  "target": "repo",
  "kind": "citation_bears_claim",
  "source_path": "...", "source_span": "...",
  "target_path": "...", "target_span": "...",
  "votes": { "supports": 1, "insufficient": 2, "contradicts": 0 },
  "agreement": 0.67,
  "triage_status": "ambiguous | content-defect | witness-error | clean",
  "resolution": null
}

Directions this enables (not all in this issue)

  • A. Gold-labeled β calibration corpus — future routes measured against adjudicated labels, not against themselves. Same-route agreement = health signal; gold-label agreement = reliability signal.
  • B. Cross-route witnesses — the same loci through a second provider/model and/or a steward; use disagreement to locate ambiguity, not to average into "truth" (the METER-LOOP-DECISION replacement line).
  • C. Ambiguity-aware scoring — classify at triage (decidable | ambiguous-source-claim | ambiguous-target-support | needs-domain-judgment); only decidable, gold-labeled loci bear scalar standing; ambiguous loci feed a clarity-debt score, not witness inconsistency.
  • D. Content discipline for citations — a citation-bearing claim must be locally checkable (one claim per source sentence; target anchor directly supports; background links marked as background). Reduces true ambiguity; it's artifact hygiene, not meter-tuning.

Out of scope

Coh-consistency improvement; any v3.2.5 prompt/schema tweak; standing promotion from same-route agreement; re-running the terminal factorized-β experiment. A/B/C/D above are separate downstream issues; this one builds the queue + triage + metrics foundation.

Guardrail

This does not reopen the stopped meter loop. It measures and harvests; it does not optimize Coh-consistency. Metrics are defect-yield / ambiguity, never Coh.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium prioritycddApplied to every issue that runs a CDD cycle

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions