Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ GITHUB_MODE=live
# actor_guarantee = floor(enrichment_allowance x ACTOR_ENRICHMENT_SHARE)
# repository_guarantee = enrichment_allowance - actor_guarantee
#
# With these defaults: 4 x 0.50 = 2 actor and 2 repository detail-fallback
# With these defaults: 40 x 0.50 = 20 actor and 20 repository detail-fallback
# attempts an hour. The remainder always goes to repository, because the formula
# floors one side and subtracts for the other — so the two always add up to the
# whole allowance.
Expand Down
10 changes: 5 additions & 5 deletions IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -871,7 +871,7 @@ With defaults:

```text
ceil(3600 / 300) × 1 × 1 = 12 poll attempts/hour
12 poll + 4 detail fallback + 8 reserve = 24 ≤ 60
12 poll + 40 detail fallback + 8 reserve = 60
```

| Allocation | Default |
Expand Down Expand Up @@ -929,7 +929,7 @@ poll_class_blocked_until = poll_used >= poll_allowance ? reset_at : nil
enrichment_class_blocked_until = enrichment_used >= enrichment_allowance ? reset_at : nil
```

So the detail-fallback lane exhausting its 4 attempts never stops polling, and polling exhausting its 12 never stops enrichment — batch enrichment does not even spend core. Actor/repository share exhaustion lives inside `BudgetLedger.reserve!(:actor | :repository)` and never touches the global block; search-window exhaustion lives in the search ledger and blocks only search. A routine future `X-RateLimit-Reset` on a successful response never defers anything.
So the detail-fallback lane exhausting its 40 attempts never stops polling, and polling exhausting its 12 never stops enrichment — batch enrichment does not even spend core. Actor/repository share exhaustion lives inside `BudgetLedger.reserve!(:actor | :repository)` and never touches the global block; search-window exhaustion lives in the search ledger and blocks only search. A routine future `X-RateLimit-Reset` on a successful response never defers anything.

**Secondary rate limits are global.** They are IP-scoped, and they can arise on *any* live request — including enrichment, which has no source row. On any secondary-limit response: set `global_blocked_until` from `Retry-After` (or ≥ 1 minute with exponential backoff when the header is absent), also update the request-specific source or entity retry state, and stop all live requests until the block expires.

Expand Down Expand Up @@ -1652,8 +1652,8 @@ with a limit of 10; and joining exact qualifiers with `OR` produced HTTP 422.
payload-provided `api_url` through the core ledger's
`CORE_DETAIL_FALLBACK_ALLOWANCE` (40/hour). The fallback never constructs a URL from an
identifier and never touches the polling allocation.
- **Dual ledgers.** `github_api_budget` (core: 12 poll + 4 detail fallback + 8 reserve
60, remainder deliberately unspent) and `github_search_budget` (per-minute search)
- **Dual ledgers.** `github_api_budget` (core: 12 poll + 40 detail fallback + 8 reserve
= 60) and `github_search_budget` (per-minute search)
are reconciled independently against their own `x-ratelimit-resource` headers. The
global request gate still serializes all outbound requests.
- **A useful-data completion contract per entity.** Completion is an explicit, queryable
Expand Down Expand Up @@ -1747,6 +1747,6 @@ What supersedes what: Appendix F's durable-backlog invariants (rows survive quot
windows; denial defers; FIFO by `created_at, id`; no quota terminal state) carry forward
unchanged. Its one-request-per-entity service model, its "40 backlog-enrichment requests"
core split, its refresh-suppression phrasing, and its refusal to publish any service rate
are superseded by the staged batch path, the 12 + 4 + 8 core formula with the search
are superseded by the staged batch path, the 12 + 40 + 8 core formula with the search
budget beside it, the refresh composition rule, and the measured catch-up block. ADR 0013
records the decision and its rejected alternatives.
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ What it does, running:
**Enrichment is a durable, staged, batch-served backlog.** The normal path resolves up to
ten entities per GitHub Search request on Search's own per-minute budget (10 ceiling, 2
reserved, 6-second pacing); only items a batch could not settle fall back to individual
payload-URL fetches inside a bounded core allowance of 4 per hour. The core ledger keeps
payload-URL fetches inside a bounded core allowance of 40 per hour. The core ledger keeps
12 requests for polling, 8 in reserve, and the rest for the detail-fallback lane. Entity
rows remain actionable until enrichment satisfies the useful-data contract or an
entity-specific terminal outcome is established; quota exhaustion, pacing, and reserve
Expand Down Expand Up @@ -632,7 +632,7 @@ Reports persisted state only, as nine top-level blocks in a fixed order: `captur
which one is binding.
- **`ledger`** — the core hourly ledger: window status, `poll` used/allowance,
**`detail_fallback`** used/allowance (renamed from `enrichment` — it budgets the
bounded detail lane, default 4/hour), and the `actor_requests`/`repository_requests`
bounded detail lane, default 40/hour), and the `actor_requests`/`repository_requests`
share pairs.
- **`search_ledger`** — the per-minute Search ledger: `present`, the observed
resource/limit/remaining/reset, the configured `request_ceiling` and `reserve`, the
Expand Down Expand Up @@ -1026,9 +1026,9 @@ and the JSON stream, so a leading brace is exactly what separates them.
Boot, then a poll that created four events and quarantined three:

```text
{"timestamp":"2026-07-31T15:50:49.677Z","level":"info","service":"github-push-ingestor","environment":"development","event":"config.budget_resolved","mode":"fixture","poll_interval_seconds":300,"max_pages_per_poll":1,"enabled_live_source_count":1,"worst_case_reservations_per_poll":9,"limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":4,"actor_guarantee":2,"repository_guarantee":2}
{"timestamp":"2026-07-31T15:50:49.677Z","level":"info","service":"github-push-ingestor","environment":"development","event":"config.budget_resolved","mode":"fixture","poll_interval_seconds":300,"max_pages_per_poll":1,"enabled_live_source_count":1,"worst_case_reservations_per_poll":9,"limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":40,"actor_guarantee":20,"repository_guarantee":20}
{"timestamp":"2026-07-31T15:50:49.896Z","level":"info","service":"github-push-ingestor","environment":"development","event":"ingestion.run_started","run_id":"099d562d-1261-488d-9003-cb0c443cdb55","event_source_id":1,"source_type":"github_fixture_events","github_mode":"fixture","forced":false,"lock_wait_ms":2.2}
{"timestamp":"2026-07-31T15:50:49.924Z","level":"info","service":"github-push-ingestor","environment":"development","event":"budget.window_initialized","limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":4,"actor_guarantee":2,"repository_guarantee":2,"rate_limit_resource":"core","rate_limit_limit":60,"rate_limit_remaining":59,"rate_limit_used":1,"rate_limit_reset_at":"2026-07-31T16:50:49Z","poll_used":1}
{"timestamp":"2026-07-31T15:50:49.924Z","level":"info","service":"github-push-ingestor","environment":"development","event":"budget.window_initialized","limit":60,"reserve":8,"poll_allowance":12,"enrichment_allowance":40,"actor_guarantee":20,"repository_guarantee":20,"rate_limit_resource":"core","rate_limit_limit":60,"rate_limit_remaining":59,"rate_limit_used":1,"rate_limit_reset_at":"2026-07-31T16:50:49Z","poll_used":1}
{"timestamp":"2026-07-31T15:50:49.965Z","level":"info","service":"github-push-ingestor","environment":"development","event":"ingestion.event_quarantined","run_id":"099d562d-1261-488d-9003-cb0c443cdb55","github_event_id":"58000000006","event_type":"PushEvent","error_code":"invalid_field_format","error_message":"payload.head is \"not-a-valid-object-name\", not 40 or 64 hexadecimal characters","payload_fingerprint":"a8ad67ca97a4c48049f5fa447d5d88ae10c58c514e0129546e18b5ff22368020"}
{"timestamp":"2026-07-31T15:50:49.973Z","level":"info","service":"github-push-ingestor","environment":"development","event":"ingestion.run_completed","run_id":"099d562d-1261-488d-9003-cb0c443cdb55","event_source_id":1,"duration_ms":100.7,"next_poll_at":"2026-07-31T15:55:49Z","consecutive_failures":0,"run_status":"completed","classification":"ok","stop_reason":"no_next_link","pages_fetched":1,"events_received":8,"push_events_seen":6,"events_created":4,"duplicates_skipped":0,"events_quarantined":3,"events_ignored":1,"events_failed":0}
{"timestamp":"2026-07-31T15:50:50.024Z","level":"info","service":"github-push-ingestor","environment":"development","event":"enrichment.dispatched","cycle_enqueued":1,"reason":"ingestion"}
Expand Down Expand Up @@ -1257,7 +1257,7 @@ Two different things that both stop requests, deliberately kept apart:
limit set it; it stops **everything**. `budget.global_block_set` and
`budget.global_block_cleared` mark the edges.
- **Class exhaustion** is *derived* from the counters and writes nothing. When polling has
spent its twelve, the detail lane carries on; when the detail lane has spent its four,
spent its twelve, the detail lane carries on; when the detail lane has spent its forty,
polling carries on — and batch enrichment, on its own resource, notices neither.
`budget.class_exhausted` fires once per class per window, and
`budget.share_exhausted` once per fairness share.
Expand Down Expand Up @@ -1560,8 +1560,8 @@ Each dispatch call enqueues at most one `EnrichmentCycleJob` regardless of backl
entity rows, not queued jobs, are the backlog. Each cycle runs batch lanes then detail
lanes inside its 55-second budget, every claim under a lease.
Steady state at the defaults: twelve polls an hour on core, up to eight paced Search
batches a minute serving as many as ten entities each, and at most four detail fallbacks
an hour split 2/2 with borrowing. Within each class the oldest never-enriched entity is
batches a minute serving as many as ten entities each, and at most forty detail fallbacks
an hour split 20/20 with borrowing. Within each class the oldest never-enriched entity is
batched first; refresh candidates ride along only under the composition rule — own
backlog exhausted, none claimable in the other class.

Expand Down
2 changes: 1 addition & 1 deletion app/services/github/enrichment/cycle_runner.rb
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def to_log = to_h.compact
# lane has nothing claimable. For search batches the borrow is purely a
# scheduling fact — the search ledger has no per-lane caps and its actor_used /
# repository_used counters exist for observability. For detail requests the flag
# travels to the core ledger, which enforces the 2/2 share split under its row
# travels to the core ledger, which enforces the 20/20 share split under its row
# lock exactly as before.
class LaneSchedule
def initialize(actor_weight:, repository_weight:)
Expand Down
3 changes: 2 additions & 1 deletion config/ci.rb
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@
# that step has already initialized the rate-limit window and persisted the corpus's
# stub entities, so this one exercises the whole §12 chain — poll, persist, stub,
# enrich — across two real processes, offline and deterministically.
step "Tests: One-shot enrichment smoke", "env RAILS_ENV=test GITHUB_MODE=fixture bin/enrich"
step "Tests: One-shot enrichment smoke",
"env RAILS_ENV=test GITHUB_MODE=fixture SEARCH_PACING_SECONDS=0 bin/enrich --limit 6"

# Solid Queue's own validator over config/queue.yml and config/recurring.yml. Starts no
# process; catches an unparseable schedule or a task naming a class that does not exist.
Expand Down
2 changes: 1 addition & 1 deletion config/initializers/github.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
# and no schema — it is arithmetic over the environment — so `bin/rails db:prepare`,
# `rails runner`, CI's schema load, and a container starting before the `setup`
# service completes are all unaffected. Every variable has a working default, and the
# default set is feasible (12 + 8 = 20 < 60), so a clean checkout never raises.
# default set is feasible (12 + 40 + 8 = 60), so a clean checkout never raises.
#
# That property is why the *configured* source count is what is validated here and the
# *observed* one is not: ADR 0004 puts deriving ENABLED_LIVE_SOURCE_COUNT from
Expand Down
2 changes: 1 addition & 1 deletion docs/DESIGN_BRIEF.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ feasible ⇔ poll_allowance + RATE_LIMIT_RESERVE
+ CORE_DETAIL_FALLBACK_ALLOWANCE ≤ rate_limit
```

With defaults: 12 poll attempts, 4 detail-fallback attempts split 2/2 by
With defaults: 12 poll attempts, 40 detail-fallback attempts split 20/20 by
`ACTOR_ENRICHMENT_SHARE`, and a reserve of 8 — the three lanes fill the limit exactly.
Normal-path enrichment spends the **search** resource instead — a separate singleton
ledger over GitHub's per-minute Search window (ceiling 10, reserve 2, 6-second pacing),
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0007-enrichment-fairness-shares-and-borrowing.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ Search batches the normal enrichment path, so this ADR's share arithmetic —
`floor(enrichment_allowance × ACTOR_ENRICHMENT_SHARE)`, borrowing on the caller's word,
`:share_exhausted` as a denial — now governs only the bounded core **detail-fallback**
lane, whose allowance is `CORE_DETAIL_FALLBACK_ALLOWANCE` (default 40, so the guarantees
default to 2/2). The batch lanes are balanced differently: a weighted rotation
default to 20/20). The batch lanes are balanced differently: a weighted rotation
(`ACTOR_ENRICHMENT_WEIGHT` / `REPOSITORY_ENRICHMENT_WEIGHT`, defaults 1/1) over whole
Search requests, with a lane that has nothing claimable yielding its slot — batch
capacity is per-request rather than per-entity, so a per-entity share would misdescribe
Expand Down
8 changes: 5 additions & 3 deletions docs/adr/0013-derivation-first-staged-batch-enrichment.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ Adopt **derivation-first, lossless staged batch enrichment** (plan Appendix G; i
payload-provided `api_url`, through the core ledger's
`CORE_DETAIL_FALLBACK_ALLOWANCE` (40/hour). No identifier is ever turned into a
constructed detail URL; the polling allocation is never touched.
4. **Dual ledgers.** `github_api_budget` (core, hourly: 12 poll + 4 detail + 8 reserve
60, remainder deliberately unspent) and `github_search_budget` (search, per-minute:
4. **Dual ledgers.** `github_api_budget` (core, hourly: 12 poll + 40 detail + 8 reserve
= 60) and `github_search_budget` (search, per-minute:
ceiling 10, reserve 2, 6-second pacing, header-less window roll) — each reconciled
only against headers naming its own resource, both behind the one global request gate.
5. **Observations and projections split.** Every raw item is an append-only
Expand Down Expand Up @@ -76,7 +76,9 @@ What this buys:
(8 spendable search requests/minute × batches of 10) — stated as a **capacity
hypothesis**, since misses, fallback, retries, and pacing all subtract from it.
- Core polling is better protected than before: normal-path enrichment no longer
competes on core at all, and the detail lane is capped at 4 rather than 40.
competes on core at all, and what core enrichment may still spend is an explicit
`CORE_DETAIL_FALLBACK_ALLOWANCE` cap rather than whatever the poll allowance and
reserve happen to leave over.
- Every enrichment claim is auditable from durable state: what was asked, what came
back, what validated, what was applied, and which raw evidence supports the current
projection.
Expand Down
2 changes: 1 addition & 1 deletion spec/services/github/allowances_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ def configuration(**overrides)
end

# The single strongest argument for deriving the guarantees rather than storing them:
# members computed at .derive would have frozen the pre-clamp 2/2 into the clamped
# members computed at .derive would have frozen the pre-clamp 20/20 into the clamped
# copy, and the ledger would enforce guarantees the clamped allowance cannot fund.
it "recomputes the guarantees from the clamped allowance rather than from the derived one" do
derived = described_class.derive(configuration: configuration, limit: 15)
Expand Down
7 changes: 4 additions & 3 deletions spec/services/github/budget_ledger_shared_ip_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,10 @@ def snapshot(reset_at: window_reset, observed_at: frozen_time, **overrides)
describe "an observed limit that changes mid-window" do
# ADR 0004: "allowances are re-derived at window rollover and initialization, not
# mid-window... the price of keeping the change atomic with the counter reset."
# A limit of 22 cannot fund the full 12 + 4 + 8 commitment, so a re-derivation here
# would have clamped the detail allowance to 2 — which is how staying at 4 proves
# nothing was re-derived.
# A limit of 22 cannot fund the 12 + 4 + 8 this window stores — the helper's
# deliberately small detail allowance, not the 40 a real window derives — so a
# re-derivation here would have clamped the detail allowance to 2, which is how
# staying at 4 proves nothing was re-derived.
it "stores the new limit without re-deriving the allowances under it" do
active_window
ledger.reconcile!(snapshot("x-ratelimit-limit" => "22", "x-ratelimit-remaining" => "20"),
Expand Down
4 changes: 2 additions & 2 deletions spec/services/github/budget_ledger_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,8 @@ def snapshot(**overrides)
end

describe "class isolation (plan §10)" do
# §10: detail fallback exhausting its four attempts never stops polling, and polling
# exhausting its twelve never stops the fallback.
# §10: detail fallback exhaustion never stops polling, and polling exhaustion never
# stops the fallback. This context uses the helper's deliberately small allowance of 4.
it "denies polling once its allowance is spent, without touching enrichment" do
active_window(poll_used: 12)

Expand Down
4 changes: 2 additions & 2 deletions spec/services/github/enrichment/admission_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,8 @@
expect(admission.detail(now: frozen_time)).to be_granted
end

# enrichment_allowance now means CORE_DETAIL_FALLBACK_ALLOWANCE (=4), and the class
# block is derived from the counters — spent allowance defers to the window reset.
# enrichment_allowance now means CORE_DETAIL_FALLBACK_ALLOWANCE. This example uses the
# helper's deliberately small value of 4; spent allowance defers to the window reset.
it "denies :class_exhausted once the detail-fallback allowance is spent" do
active_budget_window(enrichment_used: 4)

Expand Down
5 changes: 2 additions & 3 deletions spec/support/budget_helpers.rb
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Ledger rows in the state most reservations actually happen in.
#
# Shared between the ledgers' own specs and the executors', so "what an active window
# looks like" is defined once. Core values match the plan's Appendix G defaults:
# 12 poll + 4 detail fallback + 8 reserve on the hourly core resource; the search
# window is the per-minute 10-ceiling / 2-reserve pair.
# looks like" is defined once. The deliberately small core detail allowance keeps boundary
# examples concise; the search window uses the default per-minute 10-ceiling / 2-reserve pair.
module BudgetHelpers
def active_budget_window(now: frozen_time, **overrides)
Github::BudgetLedger.new.bootstrap!(now: now)
Expand Down
Loading