From cd3369872021976c04b01871075964424223963e Mon Sep 17 00:00:00 2001 From: rysweet Date: Sun, 26 Jul 2026 01:58:28 +0000 Subject: [PATCH 1/5] docs(coverage): make the coverage goal's done-gate a deterministic boolean MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The recurring 'audit Simard's test coverage and raise it to 70%' goal kept stalling (self-diagnosing GENUINELY-STUCK) not because 70% is ambiguous — it is trivially measurable — but because the done-gate had been reframed from a one-command boolean into an open-ended per-module audit series with no whole-goal terminator, then wrapped in steward-identity/recursion-guard scaffolding that checked the scaffolding instead of the number. Per the 2026-07-26 operator alignment meeting ('70% coverage is clear enough — cargo llvm-cov, compare to 70, done'), gate on the number: - add scripts/coverage-gate.sh: runs cargo llvm-cov --summary-only --json, reads the TOTAL line-%, exits 0 (DONE, >=70%) / 1 (NOT DONE, prints gap) / 2 (could-not-verify). No steward identity, no recursion guard. - COVERAGE_AUDIT.md: rewrite the done-criteria (Sec 2) to the deterministic boolean; demote the former per-module procedure (Sec 3) to 'how to raise the number when short'; mark the charter ACTIVE/ratified. - COVERAGE_BASELINE.md: the per-group ledger is now a map for choosing what to test next, not the done-gate. No CI behavior changes; Sec 4 (no hard CI coverage gate) still holds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Specs/COVERAGE_AUDIT.md | 148 ++++++++++++++++-------------- docs/testing/COVERAGE_BASELINE.md | 10 +- scripts/coverage-gate.sh | 68 ++++++++++++++ 3 files changed, 156 insertions(+), 70 deletions(-) create mode 100755 scripts/coverage-gate.sh diff --git a/Specs/COVERAGE_AUDIT.md b/Specs/COVERAGE_AUDIT.md index 7dd6c5fb8..f867d277f 100644 --- a/Specs/COVERAGE_AUDIT.md +++ b/Specs/COVERAGE_AUDIT.md @@ -3,10 +3,14 @@ ## Status - **Created**: 2026-07-16 -- **State**: PROPOSED — awaiting owner/PM-architect ratification. The - disambiguation (§1), the measurable done-criteria (§2), and the - deterministic next-target procedure (§3) are actionable immediately; they - do not change any code or CI behavior. +- **Updated**: 2026-07-26 — done-gate simplified to a single deterministic + boolean (whole-repo line coverage ≥ 70%, via `scripts/coverage-gate.sh`), + replacing the former per-module audit series. Ratified by the operator in + the 2026-07-26 alignment meeting ("70% coverage *is* clear enough — + `cargo llvm-cov`, compare to 70, done"). +- **State**: ACTIVE. §1 (scope), §2 (the deterministic done-gate), and §3 + (how to raise the number when short) are in force. This charter changes no + CI behavior; §4 still holds (no hard CI coverage gate). - **Consolidates goal slugs**: `audit-simard-test-coverage`, `raise-coverage-to-70`, `improve-amplihack-test-coverage`. - **Companion ledger**: [`docs/testing/COVERAGE_BASELINE.md`](../docs/testing/COVERAGE_BASELINE.md) @@ -23,6 +27,11 @@ gets a single answer to three questions that were previously unanswered: 2. **When is it DONE, measured how?** (§2) 3. **What is the concrete next step if it is not yet done?** (§3) +> **2026-07-26 update.** The done-gate (question 2) is now a single +> deterministic boolean: whole-repo line coverage ≥ 70%, measured by +> `scripts/coverage-gate.sh`. The per-group ledger below is retained as a +> **map for choosing what to test next** (question 3), not as the done-gate. + ## Why this charter exists now The goal has cycled repeatedly and, most recently, an OODA cycle diagnosed @@ -77,73 +86,78 @@ defensible "already done" verdict). - **Out of scope — a workspace-wide hard coverage gate.** See §4. > If the goal text says "raise it to 70%" with no further qualification, -> read it as: *raise each attacked Simard module to ≥ 70% aggregate line -> coverage, one bounded increment per PR, tracked in the companion ledger* — -> **not** as a single workspace-wide percentage enforced by CI. - -## 2. Measurable done-criteria - -The unit of measurement is **aggregate line coverage of a target group** -(a single file or a `src//` directory), produced by: +> read it literally: *raise the repository's whole-crate aggregate line +> coverage to ≥ 70%*, measured by `cargo llvm-cov` (§2). This is a single +> deterministic number, not a per-module audit series. The per-group ledger +> in §5's companion file remains a **useful map for choosing what to test +> next**, but it is no longer the done-gate — the whole-repo total is. +> +> This is **not** a workspace-wide *hard coverage gate enforced by CI* (§4); +> it is the completion criterion for the recurring goal, evaluated on demand +> by running one command. + +## 2. Measurable done-criteria — one deterministic boolean + +The goal is DONE when the repository's **whole-crate aggregate line coverage +is ≥ 70%**. That is the entire criterion. It is measured by one command and +evaluated by a comparison — a boolean, not a judgement call: ```bash -cargo llvm-cov --no-fail-fast --summary-only -# or, scoped to one library module: -cargo llvm-cov --lib --summary-only -- +scripts/coverage-gate.sh # threshold defaults to 70 +# which measures the same way CI does (.github/workflows/coverage.yml): +cargo llvm-cov --no-fail-fast --workspace --lib --bins \ + --ignore-filename-regex 'tests?/' --summary-only --json \ + | jq '.data[0].totals.lines.percent' +# → compare the printed total to 70 ``` -A target group **clears** the bar when its aggregate line coverage is -**≥ 70%**. Individual files inside a group may sit below 70% when the -uncovered paths are client-/runtime-dependent (require a live cognitive -memory or state-root), provided (a) the *group* aggregate clears 70% and -(b) the exception is recorded with a one-line justification in the ledger. -This mirrors the already-accepted `simard_ooda_step.rs` exception in the -companion ledger (group aggregate 76.07% with one 60.36% file). - -The **audit goal as a whole is DONE** when *all* of the following hold: - -- [ ] Every group listed in the companion ledger's tables shows a landed - post-lift aggregate ≥ 70% (or a recorded, justified exception). -- [ ] The "Other groups" backlog table in the ledger is empty (every - remaining tracked group has either landed or been explicitly deferred - with justification). -- [ ] The deterministic scan in §3 finds no un-ledgered `src/` file that is - both **high-risk** (per the §3 risk list) and below 70% with more than - 50 executable lines. - -When those hold, the goal is marked DONE and its slugs are tombstoned via -`simard goal remove`; a future resurfacing is resolved by linking this -charter and re-tombstoning, not by opening another planning cycle. - -## 3. Deterministic next-target procedure - -Any cycle that picks up this goal runs this procedure and always ends with -either a concrete target file **or** a defensible DONE verdict — never -`GENUINELY-STUCK`: - -1. **Measure.** Run `cargo llvm-cov --no-fail-fast --summary-only` (or the - scoped `--lib` form for speed) and capture the per-file line-% table. - The raw table *is* the evidence a cycle was previously missing. -2. **Filter.** Keep only files under `src/` with **> 50 executable lines** - and **< 70% line coverage** that are not already recorded as a justified - exception in the ledger. -3. **Rank by risk, then by gap.** Prioritise files on the safety/critical - path first — the loop and safety surfaces named in #1735 - (`engineer_loop`, `ooda`/brain, merge/overseer authority, - cognitive-memory, safe-update, `git_guardrails`) — then by absolute - coverage gap (lowest line-% first). -4. **Pick the top candidate** and open a single bounded PR that raises that - one group to ≥ 70% with **hermetic** tests (no network, no sleeps, no - live runtime; use `InMemory*` stores and a `TempDir` `SIMARD_STATE_ROOT` - per `docs/testing/hermetic-tests.md`). -5. **Record.** Add or update the group's row in the companion ledger with - the before/after aggregate, the reproduce command, and any justified - sub-70% file exception. -6. **If step 2 yields an empty set**, the audit is DONE per §2 — record that - verdict (with the measured table as evidence) and tombstone the slug. - -One PR attacks **one** group. Bounded increments keep each PR reviewable and -keep the merge-ready bar achievable, exactly as the landed per-group PRs did. +`scripts/coverage-gate.sh` runs the measurement, reads the `TOTAL` line-%, +and: + +- prints the measured total and the verdict, +- exits **0** when coverage ≥ 70% (**DONE**), +- exits **1** when coverage < 70% (**NOT DONE**, and prints the exact gap), +- exits **2** only if the measurement itself could not run (could-not-verify). + +**When the gate exits 0, the goal is DONE:** mark it complete and tombstone +its slugs via `simard goal remove`. A future resurfacing is resolved by +re-running the gate and re-tombstoning, not by opening another planning cycle. + +There is deliberately **no** steward-identity gate, **no** recursion guard, +and **no** manual per-module audit charter standing between the measurement +and the verdict. Those layers exist to protect judgement calls; whether a +number clears 70 is not a judgement call. Running the command answers it. + +> **Why this replaced the old per-module audit series.** This goal cycled +> repeatedly and self-diagnosed `GENUINELY-STUCK` not because 70% is +> ambiguous — it is trivially measurable — but because the done-gate had been +> reframed into an open-ended "attack each module to ≥ 70%, one PR at a time" +> series with no whole-goal terminator, then wrapped in scaffolding that +> checked the scaffolding instead of the number. The fix is to gate on the +> number. + +## 3. If the gate says NOT DONE: how to make progress + +The whole-repo total in §2 is the done-gate. It does **not** tell you *which* +tests to write to move the number, so use the per-group ledger as a map: + +1. **Measure per file.** Run `cargo llvm-cov --no-fail-fast --summary-only` + and capture the per-file line-% table. +2. **Filter.** Keep files under `src/` with **> 50 executable lines** and + **< 70% line coverage**. +3. **Rank by risk, then by gap.** Prioritise the safety/critical path first — + the loop and safety surfaces named in #1735 (`engineer_loop`, `ooda`/brain, + merge/overseer authority, cognitive-memory, safe-update, `git_guardrails`) — + then by absolute coverage gap (lowest line-% first). +4. **Pick the top candidate** and open a single bounded PR that adds + **hermetic** tests (no network, no sleeps, no live runtime; use + `InMemory*` stores and a `TempDir` `SIMARD_STATE_ROOT` per + `docs/testing/hermetic-tests.md`). +5. **Re-run the gate.** `scripts/coverage-gate.sh`. If it now exits 0, the + goal is DONE. + +This is guidance for *raising* the number when it is short, not a second +done-gate. One PR still attacks one bounded area to stay reviewable. ## 4. Explicitly NOT in this charter diff --git a/docs/testing/COVERAGE_BASELINE.md b/docs/testing/COVERAGE_BASELINE.md index 27cfeae48..a6cee49ba 100644 --- a/docs/testing/COVERAGE_BASELINE.md +++ b/docs/testing/COVERAGE_BASELINE.md @@ -12,9 +12,13 @@ > their per-crate targets as Simard work. This document records the most recent line-coverage baseline for each Cargo -target group and links each group to the issue that drove it to the **per-group -≥ 70% aggregate line-coverage target** (not a single workspace-wide -percentage). Update this file whenever a coverage-targeted PR lands. +target group. Since 2026-07-26 the recurring goal's **done-gate is the +whole-repo aggregate line coverage ≥ 70%** (measured by +`scripts/coverage-gate.sh`; see the charter's §2). This per-group ledger is +retained as the **map for choosing what to test next** when that total is +short — it is no longer itself the done-gate. Each group links to the issue +that drove it toward the ≥ 70% aggregate. Update this file whenever a +coverage-targeted PR lands. The numbers below come from: diff --git a/scripts/coverage-gate.sh b/scripts/coverage-gate.sh new file mode 100755 index 000000000..88576fa57 --- /dev/null +++ b/scripts/coverage-gate.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# +# coverage-gate.sh — the deterministic done-gate for the recurring +# "audit Simard's test coverage and raise it to 70%" goal. +# +# Usage: +# scripts/coverage-gate.sh [threshold] # threshold defaults to 70 +# +# It runs ONE measurement and answers ONE boolean question: +# +# is the whole-repo aggregate line coverage >= % ? +# +# - prints the measured total and the verdict (DONE / NOT DONE + the gap) +# - exits 0 when DONE (coverage >= threshold), 1 when NOT DONE +# - exits 2 on a measurement/tooling error (could-not-verify) +# +# This is a boolean, not a judgement call: there is no steward-identity gate, +# no recursion guard, and no manual per-module audit charter standing between +# the measurement and the verdict. See Specs/COVERAGE_AUDIT.md. +# +# Requires: cargo-llvm-cov, jq. +set -euo pipefail + +threshold="${1:-70}" + +if ! command -v jq >/dev/null 2>&1; then + echo "coverage-gate: jq is required but not installed" >&2 + exit 2 +fi +if ! cargo llvm-cov --version >/dev/null 2>&1; then + echo "coverage-gate: cargo-llvm-cov is required but not installed" >&2 + echo " install with: cargo install cargo-llvm-cov" >&2 + exit 2 +fi + +echo "coverage-gate: measuring whole-repo line coverage (cargo llvm-cov)..." >&2 + +# Measure the same way CI does (.github/workflows/coverage.yml): library + +# binary unit tests, with test files excluded from the denominator. The slow, +# real-subprocess integration tests under tests/ are deliberately NOT run here +# — they take tens of minutes and are prone to being reaped under load, which +# is exactly what made a "just measure it" gate feel unreachable before. +summary_json="$(cargo llvm-cov --no-fail-fast --workspace --lib --bins \ + --ignore-filename-regex 'tests?/' --summary-only --json)" || { + echo "coverage-gate: cargo llvm-cov failed — could not verify coverage" >&2 + exit 2 +} + +total="$(printf '%s' "$summary_json" | jq -r '.data[0].totals.lines.percent')" +if [[ -z "$total" || "$total" == "null" ]]; then + echo "coverage-gate: could not read total line coverage from llvm-cov JSON" >&2 + exit 2 +fi + +# Integer-scaled comparison keeps the gate free of floating-point shell math. +total_scaled="$(printf '%.4f' "$total" | tr -d '.')" +threshold_scaled="$(printf '%.4f' "$threshold" | tr -d '.')" + +printf 'coverage-gate: total line coverage = %.2f%% (threshold %s%%)\n' "$total" "$threshold" + +if (( total_scaled >= threshold_scaled )); then + printf 'coverage-gate: DONE — %.2f%% >= %s%%. Close the goal (simard goal remove).\n' "$total" "$threshold" + exit 0 +fi + +gap="$(printf '%s %s' "$threshold" "$total" | awk '{printf "%.2f", $1 - $2}')" +printf 'coverage-gate: NOT DONE — gap is %s pts. Add hermetic tests to the lowest-coverage src/ files and re-run.\n' "$gap" +exit 1 From a91c2174b723a95fa55a53990c8466e9be91e41b Mon Sep 17 00:00:00 2001 From: rysweet Date: Sun, 26 Jul 2026 02:05:28 +0000 Subject: [PATCH 2/5] =?UTF-8?q?docs(coverage):=20record=2084.36%=20gate=20?= =?UTF-8?q?result=20=E2=80=94=20goal=20is=20DONE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran scripts/coverage-gate.sh: whole-repo line coverage is 84.36% (9393 unit tests, 0 failures), well above the 70% threshold. The recurring coverage goal has been over the bar for a long time; it only appeared stuck because the old done-gate evaluated per-module audit scaffolding instead of this number. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Specs/COVERAGE_AUDIT.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/Specs/COVERAGE_AUDIT.md b/Specs/COVERAGE_AUDIT.md index f867d277f..b4d3e605d 100644 --- a/Specs/COVERAGE_AUDIT.md +++ b/Specs/COVERAGE_AUDIT.md @@ -11,6 +11,11 @@ - **State**: ACTIVE. §1 (scope), §2 (the deterministic done-gate), and §3 (how to raise the number when short) are in force. This charter changes no CI behavior; §4 still holds (no hard CI coverage gate). +- **Latest gate result**: 2026-07-26 — `scripts/coverage-gate.sh` measured + **84.36%** whole-repo line coverage (9393 unit tests, 0 failures), i.e. + **DONE** (≥ 70%). The goal has in fact been over the bar for a long time; + it only ever appeared "stuck" because the done-gate checked scaffolding + instead of this number. Close its slugs with `simard goal remove`. - **Consolidates goal slugs**: `audit-simard-test-coverage`, `raise-coverage-to-70`, `improve-amplihack-test-coverage`. - **Companion ledger**: [`docs/testing/COVERAGE_BASELINE.md`](../docs/testing/COVERAGE_BASELINE.md) From 347e8d0077b947789930cb7b635ba28e967f7103 Mon Sep 17 00:00:00 2001 From: rysweet Date: Sun, 26 Jul 2026 02:23:17 +0000 Subject: [PATCH 3/5] =?UTF-8?q?docs(coverage):=20record=20escalation-triag?= =?UTF-8?q?e=20evidence=20=E2=80=94=20blocker=20payload=20+=20Signal=20sen?= =?UTF-8?q?t?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the escalation-triage runbook trail for the blocked coverage goal: - Retrieve and record the raw typed-blocker payload behind outcome 019f6c08-d053-7d93-89bf-f1f86aee408c (OODA safeguard: no shippable progress for 3 cycles, why=GENUINELY-STUCK, evidence=[(none)]), from the persisted goal-board snapshot in cognitive memory. - Translate it to plain English and pin the root cause (done-gate certified scaffolding, not the coverage number). - Note the plain-English Signal message actually delivered to the operator via the signal-cli JSON-RPC daemon (send SUCCESS, ts 1785032545738), with no raw markers surfaced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Specs/COVERAGE_AUDIT.md | 56 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/Specs/COVERAGE_AUDIT.md b/Specs/COVERAGE_AUDIT.md index b4d3e605d..a77f3833d 100644 --- a/Specs/COVERAGE_AUDIT.md +++ b/Specs/COVERAGE_AUDIT.md @@ -195,3 +195,59 @@ done-gate. One PR still attacks one bounded area to stay reviewable. | #2150 / #2151 | CLOSED | Bash CI gate — rejected by owner (informs §4) | | `docs/testing/COVERAGE_BASELINE.md` | landed | The companion per-group coverage ledger | | `Specs/TDD_ADOPTION.md` | RATIFIED | Sibling charter for the recurring `adopt-tdd` goal; same "durable artifact stops the resurfacing" pattern | + +## 6. Escalation-triage record (2026-07-26) + +Follows `prompt_assets/simard/overseer/escalation_triage.md` for the blocked +goal `audit-simard-s-test-coverage-and-raise-it-to-70-4d27c91a`. Recorded here +as the durable internal audit trail; the operator only ever saw the +plain-English Signal message (last row). + +| Field | Value | +|---|---| +| Goal id | `audit-simard-s-test-coverage-and-raise-it-to-70-4d27c91a` | +| Blocker outcome id | `019f6c08-d053-7d93-89bf-f1f86aee408c` | +| Goal `status.Blocked` (pointer) | `typed blocker recorded in outcome 019f6c08-d053-7d93-89bf-f1f86aee408c` | +| Reason marker | `health-review:blocked-goal` | + +**Raw typed-blocker payload (internal only — never surfaced to the operator):** + +``` +🔒 [OODA-SAFEGUARD] OODA goal made no shippable progress for 3 consecutive +no-action cycles; why=GENUINELY-STUCK evidence=[(none)] +``` + +Retrieved from the persisted goal-board snapshot in the daemon's cognitive +memory (`self-deploy-memory.2026-07-25T20-27-59Z.json`, concept +`goal-board:snapshot`), keyed to the outcome id above. + +**Plain-English translation.** The safety brake tripped: the goal ran several +cycles in a row without shipping anything the system could point to (the +evidence list was empty), so it flagged itself as stuck. It was never stuck on +the coverage number — it was stuck because the finish check could not *observe* +completion. + +**Root cause.** The done-gate certified scaffolding (steward-identity / +recursion-guard wrapping) instead of the one thing that actually decides the +goal: whole-repo line coverage. With nothing measurable to certify, every cycle +produced "no shippable progress" and eventually "genuinely stuck", even though +coverage had been well over 70% the whole time. + +**Decision:** `rewrite-done-gate` (done-gate was unmeasurable) **+** +`complete-delivered-goal` (the underlying work was already delivered). + +**Action taken.** +- Rewrote the done-gate to a single deterministic boolean — whole-repo line + coverage ≥ 70%, measured by `scripts/coverage-gate.sh` (§2). +- Ran the gate: **84.36%** whole-repo line coverage (9,393 unit tests, 0 + failures) → **DONE**. Recorded in §"Latest gate result". + +**Signal sent to operator (plain English, no markers):** delivered via the +signal-cli JSON-RPC daemon on 2026-07-26 (send `type: SUCCESS`, timestamp +`1785032545738`). Message told the operator, in plain English, that the goal +kept stalling because its automatic "is it finished?" check looked at the wrong +thing, that the check was rewritten to measure coverage and pass at ≥ 70%, that +coverage is already 84% so the goal is complete, and that nothing is needed +from them. + +**Escalate to a human?** No — the block was course-corrected agentically. From be7b915acc0aa73ea26a70db45ca074e131c2419 Mon Sep 17 00:00:00 2001 From: rysweet Date: Sun, 26 Jul 2026 12:05:05 +0000 Subject: [PATCH 4/5] =?UTF-8?q?docs(investigation):=20record=20escalation-?= =?UTF-8?q?triage=20for=20blocked=20WS2=20int8/PQ=20goal=20=E2=80=94=20com?= =?UTF-8?q?plete-delivered-goal=20+=20Signal=20sent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the escalation-triage runbook trail for the blocked goal fix-agent-kgpacks-rs-issue-17-ws2-int8-pq-embed-7f5afcca, following prompt_assets/simard/overseer/escalation_triage.md. - Re-verified the seed premise against live GitHub state: it claimed the WS1 #16 eval baseline was still open with nothing delivered. Not so — agent-kgpacks-rs#16 is CLOSED by merged PR #41, and the goal's own tracking issue #17 is CLOSED by merged PR #40 (which satisfied all three of #17's acceptance criteria via the issue's sanctioned 'disabled codec + spike report' branch). - Pinned the root cause: a stale ordering signal, not a live block — the work had already shipped. - Decision: complete-delivered-goal (both #16 and #17 delivered by merged PRs; the seed's ordering question was moot, so no human decision needed). - Recorded the exact plain-English Signal message delivered to the operator via the signal-cli JSON-RPC daemon (send SUCCESS, ts 1785067435101, 2026-07-26 12:03:55 UTC), with no raw diagnostic markers surfaced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...ion-triage-ws2-int8-pq-embed-2026-07-26.md | 119 ++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/investigation/escalation-triage-ws2-int8-pq-embed-2026-07-26.md diff --git a/docs/investigation/escalation-triage-ws2-int8-pq-embed-2026-07-26.md b/docs/investigation/escalation-triage-ws2-int8-pq-embed-2026-07-26.md new file mode 100644 index 000000000..7d1699167 --- /dev/null +++ b/docs/investigation/escalation-triage-ws2-int8-pq-embed-2026-07-26.md @@ -0,0 +1,119 @@ +# Escalation-triage record — blocked goal `fix-agent-kgpacks-rs-issue-17-ws2-int8-pq-embed-7f5afcca` (2026-07-26) + +Follows `prompt_assets/simard/overseer/escalation_triage.md` for the blocked +goal that tracked the WS2 int8/PQ embedding-quantization spike. This is the +durable internal audit trail; the operator only ever saw the plain-English +Signal message (see §5). No raw diagnostic markers were surfaced to the +operator. + +## 1. Blocked-goal context (internal only — never surfaced) + +| Field | Value | +|---|---| +| Goal id | `fix-agent-kgpacks-rs-issue-17-ws2-int8-pq-embed-7f5afcca` | +| Tracking issue | `rysweet/agent-kgpacks-rs#17` — "WS2: int8/PQ embedding quantization spike, gated on eval recall parity" | +| Upstream dependency | `rysweet/agent-kgpacks-rs#16` — "WS1: Full-pack CVE eval validation + extended real 2024/2025 eval questions" | +| Reason marker (raw, internal) | `health-review:upstream-dependency-block` | +| Internal diagnostic WHY (raw, internal) | WS2 #17 done-gate gated on eval recall parity → depends on WS1 #16 eval baseline; at block time #16 open, no PR, no landed baseline → gate unmeasurable; engineer self-emitted `record_blocker` (Cycle 6, no churn). | + +The seed problem handed to triage asserted that #16 was *still open with nothing +delivered*, so #17's completion could not be measured. Triage re-verified that +premise against live GitHub state before acting. + +## 2. Verified ground truth (live GitHub state, `rysweet/agent-kgpacks-rs`) + +| Item | State | Closed / Merged | Delivered by | +|---|---|---|---| +| Issue **#16** (WS1 eval baseline) | **CLOSED** | 2026-07-06T20:16:25Z | merged PR **#41** — "WS1: Full-pack CVE eval validation + real 2024/2025 eval questions (#16)" (`Closes #16`) | +| Issue **#17** (WS2 int8/PQ spike) | **CLOSED** | 2026-07-07T19:19:47Z | merged PR **#40** — "WS2: int8 embedding quantization codec spike, disabled pending #16 parity (Closes #17)" | +| PR **#41** | **MERGED** | 2026-07-06T20:16:24Z | Ports the full-pack CVE eval surface to `kgpacks-eval`, commits a real 2024/2025 CVE corpus + a CI-guarded eval report artifact. | +| PR **#40** | **MERGED** | 2026-07-07T19:19:46Z | Implements the int8 scalar-quantization codec in `kgpacks-embeddings` + a spike report; adoption flag left `false` pending parity. Satisfies all three of #17's acceptance criteria via the issue's own "disabled + report" branch. | + +**Key correction to the seed premise.** The seed said the dependency was open and +undelivered. It is not: **both** issues have shipped via merged PRs. PR #40's own +description states it "satisfies all three of #17's acceptance criteria via the +*disabled + report* branch" — i.e. #17 instructed that, when a real recall-parity +number cannot yet be measured, ship the codec + spike findings with the feature +DISABLED. That is exactly what landed and closed #17. Separately, PR #41 landed +the WS1 #16 eval baseline the seed said was missing. + +## 3. Plain-English translation of the block + +The compression experiment looked stuck because its automatic "is it finished?" +check was pointed at a comparison — "does search stay just as good after +compression?" — that could only be run once a separate groundwork task (the +quality baseline) had landed. While that groundwork looked outstanding, the +finish check had nothing to measure, so the task kept re-cycling without shipping +and eventually flagged itself as waiting on a decision. + +## 4. Root cause and decision + +**Root cause.** The block was a stale ordering signal, not a live one. At the +moment triage inspected it, the underlying work had *already been delivered*: the +WS2 int8 quantization spike shipped (and closed #17) via the issue's sanctioned +"disabled codec + spike report" path, and the WS1 #16 eval baseline the gate +depended on had also shipped. The goal was therefore complete-in-fact while still +being carried as blocked. + +**Decision (per `escalation_triage.md` "HOW TO DECIDE"):** +`complete-delivered-goal`. + +Rationale for choosing this over the other two paths: +- **Not `rewrite-done-gate`:** rewriting the finish check to be machine-checkable + is unnecessary because the work is already done and its tracking issue is + already observably `CLOSED` by a `MERGED` PR — a signal the done-gate can read + directly. There is nothing left to certify. +- **Not `ask-operator-one-question`:** the seed's proposed operator question + (land #16 first vs. relax #17's recall-parity dependency) is moot. Both + outcomes already occurred — #16's baseline landed AND #17 shipped via its + disabled-and-report branch — so no human ordering/scope decision remains. + +## 5. Action taken + +- Confirmed via live GitHub state (§2) that goal `…issue-17-ws2-int8-pq-embed…` + is delivered: issue #17 `CLOSED` by merged PR #40, and its dependency #16 + `CLOSED` by merged PR #41. +- Marked the outcome as **complete-delivered-goal**; the goal should be moved out + of the blocked set — its tracking issue is `CLOSED` by a `MERGED` PR, which is + the machine-observable completion signal. +- **Sent one jargon-free Signal message to the operator** (plain English, no + markers) via the signal-cli JSON-RPC daemon. Send `type: SUCCESS`, timestamp + `1785067435101` (2026-07-26 12:03:55 UTC). + +**Exact Signal message delivered to the operator (verbatim, no raw markers):** + +> Update on the embedding-compression task (the experiment to shrink the search +> index by storing each vector as small whole numbers instead of full decimals). +> It had looked stuck because it was waiting on a separate groundwork task — the +> quality baseline that checks whether search results stay just as good after +> compression. Good news: both are already finished and shipped. The groundwork +> baseline was completed and merged, and the compression task itself was +> completed and merged too — the compression code and its written findings +> landed, with the feature left switched off by default until the quality check +> confirms it's safe, exactly as the task asked for. So this task is actually +> done, not blocked. I've marked it complete. Nothing is needed from you. + +The message contains no `OODA-SAFEGUARD`, `UNCLEAR-CRITERIA`, `GENUINELY-STUCK`, +`why=`, `evidence=[`, 🔒, `record_blocker`, `health-review:upstream-dependency-block`, +`int8`, `PQ`, `recall parity`, `WS1`/`WS2`, or issue/PR numbers — every internal +marker was translated to plain English. + +## 6. `escalation_triage.md` OUTPUT contract + +```json +{ + "problem": "The embedding-compression experiment kept looking stuck because its automatic 'is it finished?' check depended on a separate quality baseline that appeared not to have landed yet, so completion could not be measured.", + "next_step": "Confirm the current state of the compression work and its dependency; if both have already shipped, mark the goal complete instead of leaving it blocked.", + "root_cause": "Stale ordering signal, not a live block: the int8/PQ quantization spike had already shipped (closing its tracking issue via a merged PR) using the issue's own 'ship the codec disabled + a spike report when parity can't yet be measured' branch, and the quality-baseline dependency had also already shipped via a merged PR.", + "decision": "complete-delivered-goal", + "action_taken": "Verified against live GitHub state that issue #17 is CLOSED by merged PR #40 and its dependency #16 is CLOSED by merged PR #41; marked the goal complete-delivered and sent one plain-English Signal message to the operator (send SUCCESS, timestamp 1785067435101).", + "escalate": null +} +``` + +## 7. Escalate to a human? + +No. The block was course-corrected agentically. The seed's ordering/relaxation +question was resolved autonomously because both possible outcomes had already +occurred (dependency baseline landed AND the spike shipped via its disabled + +report path), so no decision remained for a human to make. From 4785201dd073f888ea36ad0479496c0960e93e29 Mon Sep 17 00:00:00 2001 From: rysweet Date: Sun, 26 Jul 2026 21:43:45 +0000 Subject: [PATCH 5/5] fix(coverage-gate): force base-10 in scaled comparison to avoid octal misparse MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scaled coverage values below 1% carry a leading zero (e.g. 0.58% -> '05800'). Bash arithmetic would read these as octal: silently mis-comparing all-octal-digit values and erroring on any digit >= 8 — contradicting the gate's deterministic boolean contract. Prefix both operands with 10# to force base-10. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/coverage-gate.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/coverage-gate.sh b/scripts/coverage-gate.sh index 88576fa57..8fa0cc090 100755 --- a/scripts/coverage-gate.sh +++ b/scripts/coverage-gate.sh @@ -53,12 +53,15 @@ if [[ -z "$total" || "$total" == "null" ]]; then fi # Integer-scaled comparison keeps the gate free of floating-point shell math. +# The `10#` prefixes force base-10: a scaled value below 1% carries a leading +# zero (e.g. 0.58% -> "05800"), which bash would otherwise read as octal — +# silently mis-comparing all-octal-digit values and erroring on any digit >= 8. total_scaled="$(printf '%.4f' "$total" | tr -d '.')" threshold_scaled="$(printf '%.4f' "$threshold" | tr -d '.')" printf 'coverage-gate: total line coverage = %.2f%% (threshold %s%%)\n' "$total" "$threshold" -if (( total_scaled >= threshold_scaled )); then +if (( 10#$total_scaled >= 10#$threshold_scaled )); then printf 'coverage-gate: DONE — %.2f%% >= %s%%. Close the goal (simard goal remove).\n' "$total" "$threshold" exit 0 fi