diff --git a/.env.example b/.env.example index c9747b5..62d420c 100644 --- a/.env.example +++ b/.env.example @@ -109,7 +109,7 @@ GITHUB_MODE=live # actor_guarantee = floor(enrichment_allowance x ACTOR_ENRICHMENT_SHARE) # repository_guarantee = enrichment_allowance - actor_guarantee # -# With these defaults: 4 x 0.50 = 2 actor and 2 repository detail-fallback +# With these defaults: 40 x 0.50 = 20 actor and 20 repository detail-fallback # attempts an hour. The remainder always goes to repository, because the formula # floors one side and subtracts for the other — so the two always add up to the # whole allowance. diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md index 3577a38..226db41 100644 --- a/IMPLEMENTATION_PLAN.md +++ b/IMPLEMENTATION_PLAN.md @@ -871,7 +871,7 @@ With defaults: ```text ceil(3600 / 300) × 1 × 1 = 12 poll attempts/hour -12 poll + 4 detail fallback + 8 reserve = 24 ≤ 60 +12 poll + 40 detail fallback + 8 reserve = 60 ``` | Allocation | Default | @@ -929,7 +929,7 @@ poll_class_blocked_until = poll_used >= poll_allowance ? reset_at : nil enrichment_class_blocked_until = enrichment_used >= enrichment_allowance ? reset_at : nil ``` -So the detail-fallback lane exhausting its 4 attempts never stops polling, and polling exhausting its 12 never stops enrichment — batch enrichment does not even spend core. Actor/repository share exhaustion lives inside `BudgetLedger.reserve!(:actor | :repository)` and never touches the global block; search-window exhaustion lives in the search ledger and blocks only search. A routine future `X-RateLimit-Reset` on a successful response never defers anything. +So the detail-fallback lane exhausting its 40 attempts never stops polling, and polling exhausting its 12 never stops enrichment — batch enrichment does not even spend core. Actor/repository share exhaustion lives inside `BudgetLedger.reserve!(:actor | :repository)` and never touches the global block; search-window exhaustion lives in the search ledger and blocks only search. A routine future `X-RateLimit-Reset` on a successful response never defers anything. **Secondary rate limits are global.** They are IP-scoped, and they can arise on *any* live request — including enrichment, which has no source row. On any secondary-limit response: set `global_blocked_until` from `Retry-After` (or ≥ 1 minute with exponential backoff when the header is absent), also update the request-specific source or entity retry state, and stop all live requests until the block expires. @@ -1652,8 +1652,8 @@ with a limit of 10; and joining exact qualifiers with `OR` produced HTTP 422. payload-provided `api_url` through the core ledger's `CORE_DETAIL_FALLBACK_ALLOWANCE` (40/hour). The fallback never constructs a URL from an identifier and never touches the polling allocation. -- **Dual ledgers.** `github_api_budget` (core: 12 poll + 4 detail fallback + 8 reserve - ≤ 60, remainder deliberately unspent) and `github_search_budget` (per-minute search) +- **Dual ledgers.** `github_api_budget` (core: 12 poll + 40 detail fallback + 8 reserve + = 60) and `github_search_budget` (per-minute search) are reconciled independently against their own `x-ratelimit-resource` headers. The global request gate still serializes all outbound requests. - **A useful-data completion contract per entity.** Completion is an explicit, queryable @@ -1747,6 +1747,6 @@ What supersedes what: Appendix F's durable-backlog invariants (rows survive quot windows; denial defers; FIFO by `created_at, id`; no quota terminal state) carry forward unchanged. Its one-request-per-entity service model, its "40 backlog-enrichment requests" core split, its refresh-suppression phrasing, and its refusal to publish any service rate -are superseded by the staged batch path, the 12 + 4 + 8 core formula with the search +are superseded by the staged batch path, the 12 + 40 + 8 core formula with the search budget beside it, the refresh composition rule, and the measured catch-up block. ADR 0013 records the decision and its rejected alternatives. diff --git a/README.md b/README.md index 29f683c..a29d698 100644 --- a/README.md +++ b/README.md @@ -80,7 +80,7 @@ What it does, running: **Enrichment is a durable, staged, batch-served backlog.** The normal path resolves up to ten entities per GitHub Search request on Search's own per-minute budget (10 ceiling, 2 reserved, 6-second pacing); only items a batch could not settle fall back to individual -payload-URL fetches inside a bounded core allowance of 4 per hour. The core ledger keeps +payload-URL fetches inside a bounded core allowance of 40 per hour. The core ledger keeps 12 requests for polling, 8 in reserve, and the rest for the detail-fallback lane. Entity rows remain actionable until enrichment satisfies the useful-data contract or an entity-specific terminal outcome is established; quota exhaustion, pacing, and reserve @@ -632,7 +632,7 @@ Reports persisted state only, as nine top-level blocks in a fixed order: `captur which one is binding. - **`ledger`** — the core hourly ledger: window status, `poll` used/allowance, **`detail_fallback`** used/allowance (renamed from `enrichment` — it budgets the - bounded detail lane, default 4/hour), and the `actor_requests`/`repository_requests` + bounded detail lane, default 40/hour), and the `actor_requests`/`repository_requests` share pairs. - **`search_ledger`** — the per-minute Search ledger: `present`, the observed resource/limit/remaining/reset, the configured `request_ceiling` and `reserve`, the @@ -1026,9 +1026,9 @@ and the JSON stream, so a leading brace is exactly what separates them. Boot, then a poll that created four events and quarantined three: ```text -{"timestamp":"2026-07-31T15:50:49.677Z","level":"info","service":"github-push-ingestor","environment":"development","event":"config.budget_resolved","mode":"fixture","poll_interval_seconds":300,"max_pages_per_poll":1,"enabled_live_source_count":1,"worst_case_reservations_per_poll":9,"limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":4,"actor_guarantee":2,"repository_guarantee":2} +{"timestamp":"2026-07-31T15:50:49.677Z","level":"info","service":"github-push-ingestor","environment":"development","event":"config.budget_resolved","mode":"fixture","poll_interval_seconds":300,"max_pages_per_poll":1,"enabled_live_source_count":1,"worst_case_reservations_per_poll":9,"limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":40,"actor_guarantee":20,"repository_guarantee":20} {"timestamp":"2026-07-31T15:50:49.896Z","level":"info","service":"github-push-ingestor","environment":"development","event":"ingestion.run_started","run_id":"099d562d-1261-488d-9003-cb0c443cdb55","event_source_id":1,"source_type":"github_fixture_events","github_mode":"fixture","forced":false,"lock_wait_ms":2.2} -{"timestamp":"2026-07-31T15:50:49.924Z","level":"info","service":"github-push-ingestor","environment":"development","event":"budget.window_initialized","limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":4,"actor_guarantee":2,"repository_guarantee":2,"rate_limit_resource":"core","rate_limit_limit":60,"rate_limit_remaining":59,"rate_limit_used":1,"rate_limit_reset_at":"2026-07-31T16:50:49Z","poll_used":1} +{"timestamp":"2026-07-31T15:50:49.924Z","level":"info","service":"github-push-ingestor","environment":"development","event":"budget.window_initialized","limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":40,"actor_guarantee":20,"repository_guarantee":20,"rate_limit_resource":"core","rate_limit_limit":60,"rate_limit_remaining":59,"rate_limit_used":1,"rate_limit_reset_at":"2026-07-31T16:50:49Z","poll_used":1} {"timestamp":"2026-07-31T15:50:49.965Z","level":"info","service":"github-push-ingestor","environment":"development","event":"ingestion.event_quarantined","run_id":"099d562d-1261-488d-9003-cb0c443cdb55","github_event_id":"58000000006","event_type":"PushEvent","error_code":"invalid_field_format","error_message":"payload.head is \"not-a-valid-object-name\", not 40 or 64 hexadecimal characters","payload_fingerprint":"a8ad67ca97a4c48049f5fa447d5d88ae10c58c514e0129546e18b5ff22368020"} {"timestamp":"2026-07-31T15:50:49.973Z","level":"info","service":"github-push-ingestor","environment":"development","event":"ingestion.run_completed","run_id":"099d562d-1261-488d-9003-cb0c443cdb55","event_source_id":1,"duration_ms":100.7,"next_poll_at":"2026-07-31T15:55:49Z","consecutive_failures":0,"run_status":"completed","classification":"ok","stop_reason":"no_next_link","pages_fetched":1,"events_received":8,"push_events_seen":6,"events_created":4,"duplicates_skipped":0,"events_quarantined":3,"events_ignored":1,"events_failed":0} {"timestamp":"2026-07-31T15:50:50.024Z","level":"info","service":"github-push-ingestor","environment":"development","event":"enrichment.dispatched","cycle_enqueued":1,"reason":"ingestion"} @@ -1257,7 +1257,7 @@ Two different things that both stop requests, deliberately kept apart: limit set it; it stops **everything**. `budget.global_block_set` and `budget.global_block_cleared` mark the edges. - **Class exhaustion** is *derived* from the counters and writes nothing. When polling has - spent its twelve, the detail lane carries on; when the detail lane has spent its four, + spent its twelve, the detail lane carries on; when the detail lane has spent its forty, polling carries on — and batch enrichment, on its own resource, notices neither. `budget.class_exhausted` fires once per class per window, and `budget.share_exhausted` once per fairness share. @@ -1560,8 +1560,8 @@ Each dispatch call enqueues at most one `EnrichmentCycleJob` regardless of backl entity rows, not queued jobs, are the backlog. Each cycle runs batch lanes then detail lanes inside its 55-second budget, every claim under a lease. Steady state at the defaults: twelve polls an hour on core, up to eight paced Search -batches a minute serving as many as ten entities each, and at most four detail fallbacks -an hour split 2/2 with borrowing. Within each class the oldest never-enriched entity is +batches a minute serving as many as ten entities each, and at most forty detail fallbacks +an hour split 20/20 with borrowing. Within each class the oldest never-enriched entity is batched first; refresh candidates ride along only under the composition rule — own backlog exhausted, none claimable in the other class. diff --git a/app/services/github/enrichment/cycle_runner.rb b/app/services/github/enrichment/cycle_runner.rb index 77fef69..7c9e22d 100644 --- a/app/services/github/enrichment/cycle_runner.rb +++ b/app/services/github/enrichment/cycle_runner.rb @@ -36,7 +36,7 @@ def to_log = to_h.compact # lane has nothing claimable. For search batches the borrow is purely a # scheduling fact — the search ledger has no per-lane caps and its actor_used / # repository_used counters exist for observability. For detail requests the flag - # travels to the core ledger, which enforces the 2/2 share split under its row + # travels to the core ledger, which enforces the 20/20 share split under its row # lock exactly as before. class LaneSchedule def initialize(actor_weight:, repository_weight:) diff --git a/config/ci.rb b/config/ci.rb index c855158..b321763 100644 --- a/config/ci.rb +++ b/config/ci.rb @@ -17,7 +17,8 @@ # that step has already initialized the rate-limit window and persisted the corpus's # stub entities, so this one exercises the whole §12 chain — poll, persist, stub, # enrich — across two real processes, offline and deterministically. - step "Tests: One-shot enrichment smoke", "env RAILS_ENV=test GITHUB_MODE=fixture bin/enrich" + step "Tests: One-shot enrichment smoke", + "env RAILS_ENV=test GITHUB_MODE=fixture SEARCH_PACING_SECONDS=0 bin/enrich --limit 6" # Solid Queue's own validator over config/queue.yml and config/recurring.yml. Starts no # process; catches an unparseable schedule or a task naming a class that does not exist. diff --git a/config/initializers/github.rb b/config/initializers/github.rb index 7ec0732..7f5d315 100644 --- a/config/initializers/github.rb +++ b/config/initializers/github.rb @@ -11,7 +11,7 @@ # and no schema — it is arithmetic over the environment — so `bin/rails db:prepare`, # `rails runner`, CI's schema load, and a container starting before the `setup` # service completes are all unaffected. Every variable has a working default, and the -# default set is feasible (12 + 8 = 20 < 60), so a clean checkout never raises. +# default set is feasible (12 + 40 + 8 = 60), so a clean checkout never raises. # # That property is why the *configured* source count is what is validated here and the # *observed* one is not: ADR 0004 puts deriving ENABLED_LIVE_SOURCE_COUNT from diff --git a/docs/DESIGN_BRIEF.md b/docs/DESIGN_BRIEF.md index 1c06f28..3912482 100644 --- a/docs/DESIGN_BRIEF.md +++ b/docs/DESIGN_BRIEF.md @@ -76,7 +76,7 @@ feasible ⇔ poll_allowance + RATE_LIMIT_RESERVE + CORE_DETAIL_FALLBACK_ALLOWANCE ≤ rate_limit ``` -With defaults: 12 poll attempts, 4 detail-fallback attempts split 2/2 by +With defaults: 12 poll attempts, 40 detail-fallback attempts split 20/20 by `ACTOR_ENRICHMENT_SHARE`, and a reserve of 8 — the three lanes fill the limit exactly. Normal-path enrichment spends the **search** resource instead — a separate singleton ledger over GitHub's per-minute Search window (ceiling 10, reserve 2, 6-second pacing), diff --git a/docs/adr/0007-enrichment-fairness-shares-and-borrowing.md b/docs/adr/0007-enrichment-fairness-shares-and-borrowing.md index 30cfcb8..0e35b63 100644 --- a/docs/adr/0007-enrichment-fairness-shares-and-borrowing.md +++ b/docs/adr/0007-enrichment-fairness-shares-and-borrowing.md @@ -189,7 +189,7 @@ Search batches the normal enrichment path, so this ADR's share arithmetic — `floor(enrichment_allowance × ACTOR_ENRICHMENT_SHARE)`, borrowing on the caller's word, `:share_exhausted` as a denial — now governs only the bounded core **detail-fallback** lane, whose allowance is `CORE_DETAIL_FALLBACK_ALLOWANCE` (default 40, so the guarantees -default to 2/2). The batch lanes are balanced differently: a weighted rotation +default to 20/20). The batch lanes are balanced differently: a weighted rotation (`ACTOR_ENRICHMENT_WEIGHT` / `REPOSITORY_ENRICHMENT_WEIGHT`, defaults 1/1) over whole Search requests, with a lane that has nothing claimable yielding its slot — batch capacity is per-request rather than per-entity, so a per-entity share would misdescribe diff --git a/docs/adr/0013-derivation-first-staged-batch-enrichment.md b/docs/adr/0013-derivation-first-staged-batch-enrichment.md index 979478e..f01bed6 100644 --- a/docs/adr/0013-derivation-first-staged-batch-enrichment.md +++ b/docs/adr/0013-derivation-first-staged-batch-enrichment.md @@ -46,8 +46,8 @@ Adopt **derivation-first, lossless staged batch enrichment** (plan Appendix G; i payload-provided `api_url`, through the core ledger's `CORE_DETAIL_FALLBACK_ALLOWANCE` (40/hour). No identifier is ever turned into a constructed detail URL; the polling allocation is never touched. -4. **Dual ledgers.** `github_api_budget` (core, hourly: 12 poll + 4 detail + 8 reserve - ≤ 60, remainder deliberately unspent) and `github_search_budget` (search, per-minute: +4. **Dual ledgers.** `github_api_budget` (core, hourly: 12 poll + 40 detail + 8 reserve + = 60) and `github_search_budget` (search, per-minute: ceiling 10, reserve 2, 6-second pacing, header-less window roll) — each reconciled only against headers naming its own resource, both behind the one global request gate. 5. **Observations and projections split.** Every raw item is an append-only @@ -76,7 +76,9 @@ What this buys: (8 spendable search requests/minute × batches of 10) — stated as a **capacity hypothesis**, since misses, fallback, retries, and pacing all subtract from it. - Core polling is better protected than before: normal-path enrichment no longer - competes on core at all, and the detail lane is capped at 4 rather than 40. + competes on core at all, and what core enrichment may still spend is an explicit + `CORE_DETAIL_FALLBACK_ALLOWANCE` cap rather than whatever the poll allowance and + reserve happen to leave over. - Every enrichment claim is auditable from durable state: what was asked, what came back, what validated, what was applied, and which raw evidence supports the current projection. diff --git a/spec/services/github/allowances_spec.rb b/spec/services/github/allowances_spec.rb index a7f10b0..42a3930 100644 --- a/spec/services/github/allowances_spec.rb +++ b/spec/services/github/allowances_spec.rb @@ -161,7 +161,7 @@ def configuration(**overrides) end # The single strongest argument for deriving the guarantees rather than storing them: - # members computed at .derive would have frozen the pre-clamp 2/2 into the clamped + # members computed at .derive would have frozen the pre-clamp 20/20 into the clamped # copy, and the ledger would enforce guarantees the clamped allowance cannot fund. it "recomputes the guarantees from the clamped allowance rather than from the derived one" do derived = described_class.derive(configuration: configuration, limit: 15) diff --git a/spec/services/github/budget_ledger_shared_ip_spec.rb b/spec/services/github/budget_ledger_shared_ip_spec.rb index 5c396b9..11752c8 100644 --- a/spec/services/github/budget_ledger_shared_ip_spec.rb +++ b/spec/services/github/budget_ledger_shared_ip_spec.rb @@ -76,9 +76,10 @@ def snapshot(reset_at: window_reset, observed_at: frozen_time, **overrides) describe "an observed limit that changes mid-window" do # ADR 0004: "allowances are re-derived at window rollover and initialization, not # mid-window... the price of keeping the change atomic with the counter reset." - # A limit of 22 cannot fund the full 12 + 4 + 8 commitment, so a re-derivation here - # would have clamped the detail allowance to 2 — which is how staying at 4 proves - # nothing was re-derived. + # A limit of 22 cannot fund the 12 + 4 + 8 this window stores — the helper's + # deliberately small detail allowance, not the 40 a real window derives — so a + # re-derivation here would have clamped the detail allowance to 2, which is how + # staying at 4 proves nothing was re-derived. it "stores the new limit without re-deriving the allowances under it" do active_window ledger.reconcile!(snapshot("x-ratelimit-limit" => "22", "x-ratelimit-remaining" => "20"), diff --git a/spec/services/github/budget_ledger_spec.rb b/spec/services/github/budget_ledger_spec.rb index 5e3ffa2..eb760af 100644 --- a/spec/services/github/budget_ledger_spec.rb +++ b/spec/services/github/budget_ledger_spec.rb @@ -84,8 +84,8 @@ def snapshot(**overrides) end describe "class isolation (plan §10)" do - # §10: detail fallback exhausting its four attempts never stops polling, and polling - # exhausting its twelve never stops the fallback. + # §10: detail fallback exhaustion never stops polling, and polling exhaustion never + # stops the fallback. This context uses the helper's deliberately small allowance of 4. it "denies polling once its allowance is spent, without touching enrichment" do active_window(poll_used: 12) diff --git a/spec/services/github/enrichment/admission_spec.rb b/spec/services/github/enrichment/admission_spec.rb index 6ac24f0..dd7732c 100644 --- a/spec/services/github/enrichment/admission_spec.rb +++ b/spec/services/github/enrichment/admission_spec.rb @@ -132,8 +132,8 @@ expect(admission.detail(now: frozen_time)).to be_granted end - # enrichment_allowance now means CORE_DETAIL_FALLBACK_ALLOWANCE (=4), and the class - # block is derived from the counters — spent allowance defers to the window reset. + # enrichment_allowance now means CORE_DETAIL_FALLBACK_ALLOWANCE. This example uses the + # helper's deliberately small value of 4; spent allowance defers to the window reset. it "denies :class_exhausted once the detail-fallback allowance is spent" do active_budget_window(enrichment_used: 4) diff --git a/spec/support/budget_helpers.rb b/spec/support/budget_helpers.rb index f8276a6..2b4bcca 100644 --- a/spec/support/budget_helpers.rb +++ b/spec/support/budget_helpers.rb @@ -1,9 +1,8 @@ # Ledger rows in the state most reservations actually happen in. # # Shared between the ledgers' own specs and the executors', so "what an active window -# looks like" is defined once. Core values match the plan's Appendix G defaults: -# 12 poll + 4 detail fallback + 8 reserve on the hourly core resource; the search -# window is the per-minute 10-ceiling / 2-reserve pair. +# looks like" is defined once. The deliberately small core detail allowance keeps boundary +# examples concise; the search window uses the default per-minute 10-ceiling / 2-reserve pair. module BudgetHelpers def active_budget_window(now: frozen_time, **overrides) Github::BudgetLedger.new.bootstrap!(now: now)