Enrich in Search batches instead of one request per entity

.env.example

@@ -64,22 +64,27 @@ GITHUB_MODE=live
 # The always-on poller attempts once and defers to its next cycle regardless.
 # SOURCE_LOCK_WAIT_SECONDS=30
 
-# Inputs to the one authoritative allowance formula (§10):
+# Inputs to the one authoritative core allowance formula (§10, as amended by
+# Appendix G):
 #
 #   poll_attempt_allowance = ceil(3600 / POLL_INTERVAL_SECONDS)
 #                            x MAX_PAGES_PER_POLL x ENABLED_LIVE_SOURCE_COUNT
-#   enrichment_allowance   = rate_limit - RATE_LIMIT_RESERVE - poll_attempt_allowance
+#   feasible iff poll_attempt_allowance + RATE_LIMIT_RESERVE
+#                + CORE_DETAIL_FALLBACK_ALLOWANCE <= rate_limit
 #
 # With these defaults: ceil(3600/300) x 1 x 1 = 12 poll attempts/hour, and
-# 60 - 8 - 12 = 40 enrichment attempts/hour. Startup fails if
-# poll_attempt_allowance + RATE_LIMIT_RESERVE reaches the limit, which would
-# leave no capacity for the enrichment Story 3 requires.
+# 12 poll + 40 detail fallback + 8 reserve = exactly 60. Normal-path enrichment runs
+# on the separate per-minute *search* budget below, not on core; the 40 covers the
+# ~2% of entities Search does not return. Startup fails when the three core lanes
+# together exceed the limit, and a deeper poll clamps the fallback lane rather than
+# over-committing it.
 #
 # Both are formula inputs and both are obeyed at runtime. POLL_INTERVAL_SECONDS is
 # the cadence: a source polled at T is due again at T + this, and an unforced run
 # before then is deferred rather than made. MAX_PAGES_PER_POLL caps how many
-# `Link`-followed pages one poll may fetch — raising it trades enrichment
-# allowance for capture depth, so at 3 the split moves from 12/40 to 36/16.
+# `Link`-followed pages one poll may fetch — raising it clamps the detail-fallback
+# lane down to what is left (at 5 pages the formula is infeasible and boot refuses);
+# it no longer cuts batch enrichment, which lives on the search budget.
 #
 # The poll allowance is enforced on top of the cadence: every
 # `docker compose run --rm ingest` spends one poll attempt per page, so at these
@@ -97,49 +102,121 @@ GITHUB_MODE=live
 # ENABLED_LIVE_SOURCE_COUNT=1
 # RATE_LIMIT_RESERVE=8
 
-# How the enrichment allowance is split between the two entity classes (§10):
+# How the core detail-fallback allowance is split between the two entity classes
+# (§10; since Appendix G this share governs the detail lane only — Search batch
+# lanes rotate by the weights in the staged section below):
 #
 #   actor_guarantee      = floor(enrichment_allowance x ACTOR_ENRICHMENT_SHARE)
 #   repository_guarantee = enrichment_allowance - actor_guarantee
 #
-# With these defaults: 40 x 0.50 = 20 actor and 20 repository attempts an hour. The
-# remainder always goes to repository, because the formula floors one side and
-# subtracts for the other — so the two always add up to the whole allowance.
-#
-# The split exists because one observed live page held ~89 distinct actors against
-# ~92 distinct repositories: repository candidates alone exceed the entire hourly
-# allowance, so a queue ordered purely by recency would starve actor enrichment to
-# zero indefinitely.
+# With these defaults: 4 x 0.50 = 2 actor and 2 repository detail-fallback
+# attempts an hour. The remainder always goes to repository, because the formula
+# floors one side and subtracts for the other — so the two always add up to the
+# whole allowance.
 #
 # It is a guarantee, not a cap. A class may borrow the other's unused capacity when
-# the other class has no *currently eligible* candidate — not merely no rows — so a
+# the other class has no *currently claimable* detail candidate — not merely no rows — so a
 # quiet hour for repositories is spent on actors rather than wasted. Both ends of
 # the range are legal: 0.0 means "repositories first, actors during the quiet
 # periods". Below 0 or above 1 is rejected at startup, because either would put one
 # guarantee above the class allowance.
 # ACTOR_ENRICHMENT_SHARE=0.50
 
-# How long an entity stays worth enriching, and how long an enriched one stays
-# fresh (§10):
-#
-#   ENRICHMENT_ELIGIBILITY_WINDOW_SECONDS - a candidate whose most recent distinct
-#     push event is older than this transitions to skipped_budget. This is what
-#     bounds the backlog: at ~2,000 candidates an hour against 40 available
-#     requests, without it the pending queue grows forever. A skip is not
-#     permanent — a genuinely new push event referencing the entity reactivates it,
-#     while a duplicate replay never does.
-#   ACTOR_REFRESH_TTL_SECONDS / REPOSITORY_REFRESH_TTL_SECONDS - how long an
-#     enriched entity is reused before it is re-fetched. Never-enriched candidates
-#     always take priority over refreshes, so raising these buys coverage of new
-#     entities rather than freshness of old ones.
-#
-# All three must be greater than zero. A zero window skips every candidate on
-# sight; a zero TTL turns the freshness cache off entirely. "Never refresh" is a
-# large number, not zero.
-# ENRICHMENT_ELIGIBILITY_WINDOW_SECONDS=3600
+# How long before an enriched entity becomes refresh-eligible (§10, Appendix G):
+#
+#   ACTOR_REFRESH_TTL_SECONDS / REPOSITORY_REFRESH_TTL_SECONDS - the minimum time an
+#     enriched entity is reused before it may be re-fetched. Never-enriched entity rows
+#     are a durable FIFO backlog: quota exhaustion defers them to a later window and
+#     never terminates them. Refreshes ride the same Search batch path: a batch fills
+#     from its own class's backlog first, tops up spare slots with TTL-stale refresh
+#     candidates only when its own backlog is exhausted and the other class has none
+#     claimable, and a refresh-only batch runs only when neither class has backlog —
+#     so raising these values affects freshness only after the backlog drains.
+#
+# Both must be greater than zero. A zero TTL turns the freshness cache off entirely.
+# "Never refresh" is a large number, not zero.
 # ACTOR_REFRESH_TTL_SECONDS=86400
 # REPOSITORY_REFRESH_TTL_SECONDS=86400
 
+# ---------------------------------------------------------------------------
+# Staged batch enrichment (plan Appendix G)
+# ---------------------------------------------------------------------------
+
+# The bounded core lane for individual detail fetches. Only batch items that came
+# back missing, renamed, identity-mismatched, or contract-invalid reach it, at the
+# stored payload-provided URL. It is the third term of the core feasibility rule
+# above and can never take the polling allocation. Zero is legal and turns the
+# fallback off.
+# CORE_DETAIL_FALLBACK_ALLOWANCE=40
+
+# The per-minute Search budget — a separate GitHub rate-limit resource from core.
+# The ceiling is the observed unauthenticated header limit (10/minute); the
+# reserve is never spent, leaving 8 spendable Search requests a minute. Windows
+# are 60 seconds and roll from response headers, or without headers after a
+# minute of inactivity.
+# SEARCH_REQUEST_CEILING=10
+# SEARCH_SAFETY_RESERVE=2
+
+# How many repeated exact user:/repo: qualifiers one Search request carries.
+# Capped at 10 — the probe-verified batching form; qualifiers are never joined
+# with OR, which GitHub answers with a 422.
+# SEARCH_BATCH_SIZE=10
+
+# Minimum seconds between Search requests, spread inside the per-minute window
+# rather than burst at its top. 0 disables pacing — the fixture walkthrough uses
+# that so batches run back to back. Must stay below
+# ENRICHMENT_CYCLE_BUDGET_SECONDS.
+# SEARCH_PACING_SECONDS=6
+
+# Must be exactly 1 while the global request gate serializes every outbound
+# request; the variable exists so the constraint is stated rather than implied.
+# SEARCH_WORKER_CONCURRENCY=1
+
+# How the batch lanes rotate. Weights, not shares: with 1/1 the cycle alternates
+# actor and repository batches, and a lane with no claimable work yields its slot
+# to the other.
+# ACTOR_ENRICHMENT_WEIGHT=1
+# REPOSITORY_ENRICHMENT_WEIGHT=1
+
+# The detail-fallback retry ladder. A retryable detail failure re-rests in
+# detail_pending with backoff until this many attempts, then the entity goes
+# terminal; a 404/410 goes terminal immediately. Batch retries are unbounded —
+# a batch failure says nothing entity-specific.
+# DETAIL_FALLBACK_MAX_ATTEMPTS=3
+
+# The durable claim lease. A batch or detail claim owns its rows until released
+# or until this many seconds pass; a crashed worker's lease expires by
+# arithmetic and the rows are reclaimed. Must exceed the worst-case single-fetch
+# time — (retries+1) x (redirects+1) x (45s gate wait + open + read) = 585s at
+# the HTTP defaults — or a slow live fetch would be reclaimed while in flight.
+# ENRICHMENT_LEASE_SECONDS=600
+
+# Enrichment retry backoff, jittered: base doubles per attempt and is capped at
+# max. Base must not exceed max.
+# ENRICHMENT_RETRY_BASE_SECONDS=60
+# ENRICHMENT_RETRY_MAX_SECONDS=3600
+
+# One enrichment cycle's time budget. The cycle stops claiming new work at this
+# boundary so it finishes inside the 60-second dispatch tick; must be below 60
+# and above SEARCH_PACING_SECONDS.
+# ENRICHMENT_CYCLE_BUDGET_SECONDS=55
+
+# Refresh eligibility beyond the TTLs: only entities seen pushing within this
+# window are refreshed at all, so quota is never spent re-fetching an entity
+# nothing references any more. Default is seven days.
+# REFRESH_ACTIVE_WITHIN_SECONDS=604800
+
+# The window behind /status's throughput and batch-quality blocks — arrivals,
+# completions, fill ratios, and backlog slope are measured over this many
+# seconds. Reporting only, like the coverage window below.
+# ENRICHMENT_METRICS_WINDOW_SECONDS=3600
+
+# How much sample the catch-up verdict requires before it will say anything.
+# Below this, /status reports catch_up.state = insufficient_sample rather than
+# guessing; at or above it, the verdict is keeping_up or not_keeping_up from
+# measured rates. There is deliberately no ETA anywhere.
+# CATCH_UP_MIN_SAMPLE_SECONDS=900
+
 # How far back GET /status looks when it computes §11's three coverage
 # percentages. Unlike every other knob in this file it changes only what the
 # system *reports* — nothing schedules, reserves, defers or skips on it, so two
@@ -148,8 +225,7 @@ GITHUB_MODE=live
 #
 # The window is measured on push_events.created_at, the instant this application
 # persisted the row, not on GitHub's occurred_at. Coverage grades this
-# application's enrichment pipeline, and that pipeline's own eligibility rule is
-# already COALESCE(last_seen_at, created_at) — one clock, not two. It also keeps
+# application's enrichment pipeline and uses one local clock throughout. It also keeps
 # the offline reviewer path meaningful: the fixture corpus pins its event
 # timestamps to a fixed date, so an occurred_at basis would report null coverage
 # to anyone running the walkthrough after that date.

.github/workflows/ci.yml

@@ -118,6 +118,10 @@ jobs:
       - name: Smoke test the one-shot enrichment command
         env:
           GITHUB_MODE: fixture
+          # The one-shot never sleeps pacing out; at the default 6-second search
+          # pacing the second batch would be reported as a pacing deferral instead
+          # of running, and the smoke would prove one lane instead of the chain.
+          SEARCH_PACING_SECONDS: "0"
         run: bin/enrich --limit 6
 
       # Solid Queue's own validator over config/queue.yml and config/recurring.yml: a

CLAUDE.md

@@ -8,7 +8,10 @@ plan wins.
 
 1. Read `IMPLEMENTATION_PLAN.md` (repository root). It is the frozen execution
    plan; its pre-implementation revision history lives in Git and in its
-   Appendices A–D, and Appendix E records how the build diverged from it.
+   Appendices A–D, Appendix E records how the build diverged from it, Appendix F
+   supersedes the enrichment load-shedding policy with a durable backlog, and
+   Appendix G supersedes Appendix F's per-entity service model with derivation-first
+   staged batch enrichment.
 2. Read `docs/DESIGN_BRIEF.md` and the ADRs under `docs/adr/`.
 3. Do not change architectural direction, add infrastructure, or add dependencies
    without first updating the plan and stating the tradeoff.
@@ -55,7 +58,7 @@ acquire `SourceLock` — they take only the request gate.
 
 Repeated observation and job execution are expected. State only the two proved ingestion
 invariants: a duplicate GitHub event ID cannot create another `push_events` row, and that
-duplicate cannot register entity activity or reactivate a `skipped_budget` entity.
+duplicate cannot register entity activity.
 Executions, ingestion runs, quarantine occurrence counts, budget debits, and logs may
 repeat or change. Recovery before commit is conditional on the event remaining in a later
 sliding-feed response. Never claim or code against exactly-once execution or universal
@@ -64,9 +67,9 @@ idempotency of persisted state.
 ### Duplicate-event invariants (plan §7)
 
 - `push_events` inserts use `ON CONFLICT (github_event_id) DO NOTHING RETURNING id`.
-- Entity activity fields (`last_seen_at`, `latest_event_at`, reactivation) update
-  only when `RETURNING` produced a row. Duplicate replays may refresh identity
-  fields but must never reactivate a `skipped_budget` entity.
+- Entity activity fields (`last_seen_at`, `latest_event_at`) update only when
+  `RETURNING` produced a row. Duplicate replays may refresh identity fields but must
+  never register new entity activity.
 - Quarantine identity is `payload_fingerprint` alone: SHA-256 of compact UTF-8
   JSON with recursively sorted object keys. One algorithm, no alternates.
 
@@ -76,6 +79,22 @@ Enrichment fetches only validated URLs: HTTPS, host exactly `api.github.com`, no
 userinfo, no non-default port, no IP literals, bounded re-validated redirects.
 Fixture mode fails closed — never a live fallback.
 
+### Durable staged enrichment backlog (plan §10, Appendices F–G)
+
+Entity rows are durable work, coalesced by stable GitHub ID, selected FIFO by
+`created_at ASC, id ASC`. The normal path is GitHub Search batches of up to
+`SEARCH_BATCH_SIZE` repeated exact `user:`/`repo:` qualifiers — never joined with `OR` —
+on the minute-scoped search ledger (ceiling 10, reserve 2, 6-second pacing). The
+payload-URL detail fallback serves only missing/renamed/mismatched/contract-invalid batch
+items and is bounded by `CORE_DETAIL_FALLBACK_ALLOWANCE` (40/hour); it never takes the
+polling allocation. Quota, pacing, reserve, and fairness denials defer work — they never
+terminate an entity. Batch results apply only on a stable-ID match. Observations are
+append-only; a refresh repoints the projection and never overwrites retained evidence.
+Refresh composition: a batch fills from its own class's never-enriched backlog first,
+tops up spare slots with TTL-stale refresh candidates only when its own backlog is
+exhausted and the other class has none claimable, and refresh-only batches run only when
+neither class has backlog.
+
 ## Database changes
 
 Schema changes go through migrations with intentional indexes, constraints where