Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 37 additions & 11 deletions docs/concepts/gap-scan-backoff-dedup.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,18 @@ description: >
fixed-window dedup was insufficient, how exponential backoff rate-limits
without ever permanently silencing a genuinely recurring gap, how this makes
the Overseer ACT on a gap once rather than observe it forever (meta bugs
#4255 / #4126), and the planned cross-process open-issue check (future work).
last_updated: 2026-07-17
#4255 / #4126), and the stable, content-addressed gap signature — the
foundation that a future durable cross-process open-issue check will build on
to survive daemon restarts.
last_updated: 2026-07-25
review_schedule: as-needed
owner: simard
doc_type: concept
status: reference
related:
- ../reference/overseer-backoff-gate-api.md
- ../reference/overseer-gap-durable-dedup.md
- ../howto/configure-gap-durable-dedup.md
- ../howto/configure-overseer-gap-scan-backoff.md
- ../reference/overseer-workstream-gap-scan.md
- ../reference/overseer-recipe-launch-idempotency.md
Expand Down Expand Up @@ -84,8 +88,10 @@ The fix
([BackoffGate reference](../reference/overseer-backoff-gate-api.md)) is
deliberately additive — it adds a new primitive rather than mutating the
existing `WhisperGate`, so every current caller is untouched. An in-process
exponential-backoff gate guards the gap-cover act path, with a cross-process
open-issue equivalence check planned as a follow-on (future work).
exponential-backoff gate guards the gap-cover act path. A stable,
content-addressed gap signature (shipped here) lays the foundation for a
durable cross-process open-issue equivalence check — the follow-on that will let
the guarantee survive daemon restarts.

### Exponential BackoffGate (in-process) — implemented

Expand All @@ -111,12 +117,30 @@ covering issue per distinct gap for the life of the process.
### Open-issue equivalence check (cross-process) — future work

The BackoffGate is in-memory, so a daemon restart forgets its state and a cold
gate could re-file a duplicate that is already open on GitHub. A planned
follow-on would, before launching a gap-cover recipe, do a **best-effort** GitHub
query (reusing the existing `stewardship::dedup` helpers) for an already-open
**equivalent** issue and skip the launch if one exists (failing toward surfacing
on any API error). This cross-process layer is **not** part of #4186; it is
documented here as intended direction only.
gate could re-file a duplicate that is already open on GitHub. The deeper cause
of the observed `[stewardship] workstream_gap:*` flood (e.g. #4671, #4680,
#4685; OODA-stuck #4689) was that the gap signature was keyed
per **run** (`originating-run: overseer-<hash>`), so the in-process gate key
churned every run and any GitHub-side search would never match across runs.

This change fixes the **root cause** by making the signature a **stable,
content-addressed slug** (derived from trusted identifiers, not the run id), so
the in-process gate now collapses a recurring gap to a single notification
**within a running daemon**, and the slug is a valid, restart-safe join key.
That stable signature is the **prerequisite** for a durable cross-process check.

The durable check itself is **future work**: before notifying/filing, the
Overseer would run a GitHub query (reusing the existing `stewardship::dedup`
helpers and `find_existing` on the `stewardship-signature:` body marker) for an
already-open **equivalent** issue and reuse/skip if one exists, failing loud on
any `gh` error. The gap-notification path (`act_flag_workstream_gaps`) does
**not** perform this query today — it applies the in-process `WhisperGate` and
notifies the operator. The proven durable pattern lives on the sibling
stewardship filing seam (`stewardship::process_orchestrator_run`); wiring it
onto the gap path is the follow-on this stable signature enables, tracked in
[#4717](https://github.com/rysweet/Simard/issues/4717). See the
[gap-filing dedup reference](../reference/overseer-gap-durable-dedup.md)
and the [how-to](../howto/configure-gap-durable-dedup.md).

## How this makes the Overseer ACT (not just observe)

Expand Down Expand Up @@ -146,7 +170,9 @@ protect.

- **One open covering issue per distinct gap within a process.** The in-process
gate holds duplicate coverage plans across ticks; the cross-process case
(restarts) is the planned open-issue check (future work).
(restarts / multiple daemons) is not yet closed — the stable signature shipped
here is the foundation for the future durable
[open-issue check](../reference/overseer-gap-durable-dedup.md).
- **Never permanently silent.** The window is capped and resets after silence; a
genuinely recurring gap always re-surfaces.
- **Additive / non-breaking.** A new `BackoffGate` primitive; existing
Expand Down
162 changes: 162 additions & 0 deletions docs/howto/configure-gap-durable-dedup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
---
title: Configure and verify gap dedup
description: >
How to operate and verify the Overseer's stable-signature gap dedup for
workstream-gap notifications — confirm a recurring gap is deduped to one
operator notification within a running daemon, read the
overseer::gap_scan flagged/suppressed logs, understand that a daemon restart
resets the in-process gate, and check the bounded GapCategory taxonomy that
keeps gap signatures stable and dedupable.
last_updated: 2026-07-25
review_schedule: as-needed
owner: simard
doc_type: howto
related:
- ../reference/overseer-gap-durable-dedup.md
- ../reference/overseer-workstream-gap-scan.md
- ../reference/overseer-backoff-gate-api.md
- ../concepts/gap-scan-backoff-dedup.md
- ./review-overseer-workstream-gaps.md
- ./configure-overseer-gap-scan-backoff.md
- ./file-stewardship-issues-from-orchestrator-runs.md
---

# Configure and verify gap dedup

The Overseer flags uncovered backlog work — uncovered goals, high-signal open
issues, and unaddressed telemetry anomalies — on its recurring gap-scan and
**notifies the operator** (email + Signal) about it. This rail makes the gap's
dedup signature a **stable, content-addressed** slug (instead of a per-run hash),
so a recurring gap is deduped to **one notification within a running daemon**
rather than re-notified every tick. This is the root-cause fix behind the
near-duplicate `[stewardship] workstream_gap:*` noise (observed on e.g. #4671,
#4680, #4685).

For the data model, signature grammar, and guarantees, see the
[gap-filing dedup reference](../reference/overseer-gap-durable-dedup.md).

> **Scope.** The gap-notification path dedupes via the **in-process**
> `WhisperGate` and notifies the operator; it does **not** create GitHub issues
> and is **not restart-safe on its own** — a daemon restart resets the gate. A
> durable, GitHub-sourced cross-process check is scoped as follow-on work
> ([#4717](https://github.com/rysweet/Simard/issues/4717)) and is
> **not** wired on this path yet; see the reference doc's *Future work* section.

## What changed for operators

- **Before:** the gap signature was derived per run
(`originating-run: overseer-<hash>`), so every restart/re-run minted a fresh
key and the in-process gate could not collapse a recurring gap → the scan
re-notified a near-duplicate gap every tick.
- **Now:** the signature is a **stable, content-addressed** slug, so the
in-process gate collapses a recurring gap to **one notification per dedup
window** for the life of the daemon.
- **Restart behaviour:** the in-process gate is memory-resident, so a restart
resets it and the first post-restart tick may re-notify. Cross-restart dedup
awaits the durable check (future work).

There is **nothing to turn on** — stable-signature dedup is always active on the
gap path whenever the Overseer runs. The existing knobs still apply:

| Env var | What it does | Default |
|---|---|---|
| `SIMARD_OVERSEER_GAP_SCAN` | Falsey (`0`/`false`/`no`/`off`) turns the gap-scan off entirely | on |
| `SIMARD_OVERSEER_GAP_SCAN_EVERY_N` | Run the scan every *Nth* tick | `1` |

See [configure the gap-scan backoff](./configure-overseer-gap-scan-backoff.md)
for the in-process dedup window.

## Prerequisites

- The acting Overseer is enabled (`SIMARD_OVERSEER_ENABLED` unset or truthy).
- An operator notifier is configured (email and/or Signal) so gap notifications
have somewhere to go.

## Verify: a recurring gap is deduped within a run

This is the core acceptance check for the workstream-gap dedup rail.

1. With a standing, uncovered gap present, let the Overseer run one gap-scan tick
and confirm it notifies **once** (a `flagged>=1` info line):

```text
INFO overseer::gap_scan flagged=1 suppressed=0
overseer recorded uncovered backlog work and notified the operator
```

2. On the **next** tick within the dedup window, confirm the **same** gap is
suppressed rather than re-notified — the count moves to `suppressed`:

```text
DEBUG overseer::gap_scan flagged=0 suppressed=1
overseer gap-scan: every observed gap is within the dedup window (suppressed)
```

3. `flagged=0 suppressed=1` for the recurring gap is the proof the stable
signature deduped it. (A daemon restart between steps 1 and 2 resets the gate
and may re-notify — that is expected until the durable check lands.)

## Read the logs

The gap path emits structured `tracing` + OTel only (no `print!`/`println!`), on
`target: "overseer::gap_scan"`:

| Field | Meaning |
|---|---|
| `flagged` | Fresh gaps notified this tick |
| `suppressed` | Gaps dropped by the in-process gate or by a malformed signature |
| `dispatched` / `all_sent` | Operator-notification delivery status |

## Malformed signatures are dropped (injection defense)

A gap whose signature is not a valid restricted slug
(`^[a-z0-9][a-z0-9:_#.\-/]{0,200}$`) is **dropped at the filing seam** and counted
as suppressed — it never reaches an operator notification:

```text
WARN overseer::gap_scan category="goal"
overseer gap-scan: dropping a gap with a malformed dedup signature
(outside the bounded taxonomy)
```

This is deliberate: signatures come from trusted identifiers only, so a malformed
one signals a bug or an injection attempt, not a real gap.

## The bounded taxonomy (why signatures are now stable)

Duplicates used to slip through because free-form titles drifted between ticks.
Each gap resolves to a bounded `GapCategory` variant with a stable slug that
anchors the signature:

| Gap kind | `GapCategory` | Signature prefix |
|---|---|---|
| Uncovered p1/p2 goal | `GoalUncovered` | `goal:<goal_id>` |
| High-signal open issue | `IssueUncovered` | `issue:<repo>#<n>` |
| Unaddressed anomaly | `AnomalyUnaddressed` | `anomaly:<slug>` |

`GapCategory` is a closed enum of exactly these three kinds, so a gap's signature
is stable across ticks. The fix did not add kinds — it made the signature a
stable, content-addressed slug (instead of a per-run hash) so the in-process gate
recognises the same gap across ticks. This change is additive.

## Common pitfalls

- **A duplicate notification appeared right after a restart.** Expected — the
in-process gate is reset on restart. Cross-restart dedup is future work.
- **Two notifications for the "same" gap.** Confirm the gaps carry the **same**
signature. If the signatures differ, the gap resolved to two distinct keys
(e.g. two different goal ids) — correct behaviour, not a dedup miss.
- **A gap I expected was never notified.** Check for a `dropping a gap with a
malformed dedup signature` WARN — a signature outside the bounded taxonomy is
dropped by design.

## See also

- [Gap-filing dedup reference](../reference/overseer-gap-durable-dedup.md)
— signature grammar, the in-process flow, and the scoped durable follow-on.
- [Review the Overseer's workstream gaps](./review-overseer-workstream-gaps.md)
— where the gaps surface and how to respond.
- [Gap-scan dedup & exponential backoff](../concepts/gap-scan-backoff-dedup.md)
— the in-process gate this stable signature feeds.
- [File stewardship issues from orchestrator runs](./file-stewardship-issues-from-orchestrator-runs.md)
— the sibling loop whose durable dedup flow the future gap check would mirror.
2 changes: 2 additions & 0 deletions docs/howto/review-overseer-workstream-gaps.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ owner: simard
doc_type: howto
related:
- ../reference/overseer-workstream-gap-scan.md
- ../reference/overseer-gap-durable-dedup.md
- ../howto/configure-gap-durable-dedup.md
- ./watch-overseer-activity.md
- ../reference/overseer-activity-feed.md
- ../design/overseer.md
Expand Down
2 changes: 2 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,8 @@ Bare `simard` prints the unified help text instead of attempting a hidden enviro
- [Canary gate isolation and self-deploy convergence reference](./reference/canary-gate-convergence.md) - The #4440 root-cause repair that acts on those diagnostics so a healthy candidate self-deploys and `DeployDrift` returns to 0: per-gate `self_relaunch::gate` tracing/OTel spans in `verify_canary`, the additive `RelaunchConfig.canary_env` narrow allow-list plus `scrub_gate_env` that supplies a gate's legitimately-missing signal without weakening fail-closed semantics, and the self-deploy loop advancing past the stuck target SHA. See the [convergence runbook](./howto/converge-a-stuck-red-canary-self-deploy.md).
- [How to review the Overseer's workstream gaps](./howto/review-overseer-workstream-gaps.md) - Read, act on, and tune the Overseer's recurring "what workstreams are we missing?" gap-scan — the uncovered high-priority goals, high-signal issues, and unaddressed anomalies it flags each tick, where the deduped notification appears, and the `SIMARD_OVERSEER_GAP_SCAN` knobs (#2630).
- [Overseer workstream gap-scan reference](./reference/overseer-workstream-gap-scan.md) - The additive Observe→Orient→Act gap-scan: the `Signal::WorkstreamGap`/`GapItem`/`ProblemKind::WorkstreamCoverage` model, the coverage-set detection contract, the deduped NotifyOperator act path, the `SIMARD_OVERSEER_GAP_SCAN` configuration, and the additive `OverseerTickReport.workstream_gaps_detected` counter (#2630).
- [Overseer gap-filing dedup reference](./reference/overseer-gap-durable-dedup.md) - The stable, content-addressed gap signature that stops the Overseer re-notifying a near-duplicate gap every tick within a running daemon: the bounded `GapCategory` taxonomy (`GoalUncovered`/`IssueUncovered`/`AnomalyUnaddressed`), the slug-validated `stewardship-signature:` key that replaces the old per-run hash, the in-process `WhisperGate` dedup + operator notification on the gap path, and the injection-defense slug validation enforced at the filing seam. The durable, restart-safe GitHub-side open-issue check is scoped as the follow-on this signature enables.
- [How to configure and verify gap dedup](./howto/configure-gap-durable-dedup.md) - Confirm a recurring gap is deduped to one operator notification within a running daemon, read the `overseer::gap_scan` `flagged`/`suppressed` logs, understand that a daemon restart resets the in-process gate, and check the bounded taxonomy that keeps gap signatures stable and dedupable.
- [Overseer recipe-launch idempotency reference](./reference/overseer-recipe-launch-idempotency.md) - The launcher-level rail that makes `AmplihackRecipeRunner::spawn` idempotent per task signature: the pure `recipe_signature` normalization (`target_repo` + `task_description`, trim/lowercase/whitespace-collapse, `\u{1F}` separator), the reap-then-dedup order, the fail-visible `overseer::recipe` suppressed-launch warning, the shared-handle `probe` semantics, and the injectable `ChildSpawner`/`SpawnedChild` test seam — so a still-blocked signature no longer spawns a byte-identical `smart-orchestrator` every tick (#4125).
- [Overseer root-cause ("WHY") API reference](./reference/overseer-root-cause-why-api.md) - The `RootCause`/`CauseCandidate`/`Confidence`/`Likelihood`/`CauseSource` model and additive `Problem.why` field, the pure `root_cause::analyze` analyzer with `PriorOccurrence`/`root_cause_signature`, the `Remediation`/`RemediationClass` classification on `PlannedIntervention`, the `Overseer::with_memory` recall+store seam, the `decide_blocked_goal` recurrence routing, the `goal_blocked_with_why` WHY notification constructor, the `ProblemEntry` activity-feed rows, and the extended `OverseerTickReport`/`OverseerTotals` counters (#2635).
- [How to configure and observe the Overseer root-cause principle](./howto/configure-overseer-root-cause-why.md) - Read the WHY and the root-cause/symptom label in the `overseer::root_cause` traces, the `OverseerTickReport` counters, the activity-feed `problem_entries` rows, and the `goal-blocked` notification; understand the always-on/no-opt-out contract and the graceful memory-recall degrade; and verify the feature end-to-end with injected fakes (#2635).
Expand Down
Loading
Loading