Implement derivation-first staged batch enrichment without authentication

[batbrainy]

Context

This issue extends the durable backlog work in #44 and is intended to be implemented on the existing draft PR #43.

The current one-request-per-entity enrichment model cannot keep up with the observed cold actor-and-repository demand under GitHub's unauthenticated core quota. Making that queue durable prevents loss, but durability alone does not make the backlog drain.

Authentication is explicitly out of scope and must not be introduced. Polling must remain protected because the public-events feed is a finite recent window; enrichment may not pause or starve ingestion.

Decision

Adopt derivation-first, lossless staged enrichment:

1. Persist every accepted push event and its event-native actor/repository identity transactionally.
2. Derive every field that can be computed from stored data without a network call.
3. Coalesce duplicate enrichment demand by stable GitHub entity ID.
4. Use unauthenticated GitHub Search requests as the normal batch path:
   • actors: repeated exact user: qualifiers;
   • repositories: repeated exact repo: qualifiers.
5. Validate returned items against stable GitHub IDs before applying them.
6. Persist every successful enrichment item and its provenance before projecting selected fields.
7. Send only missing, renamed, incomplete, or malformed batch results to individual payload-provided detail URLs.
8. Treat quota exhaustion and temporary failures as deferral only. Work remains durable and retryable.
9. Refresh active entities through the same staged/batched path.

Do not add a skipped_budget terminal outcome. Quota is a scheduling constraint, not an entity result.

Useful-data completion contract

The service is an activity monitor, not a complete GitHub profile directory. Completion means the following explicit, queryable contract—not every field GitHub can return.

Entity	Event-native / derived immediately	Required batch enrichment
Actor	GitHub ID, login/display login, avatar URL, API URL	account type and the complete raw actor search item
Repository	GitHub ID, full name, API URL; derive short name and owner login locally	description, primary language, owner GitHub ID, fork status, archived status, default branch, GitHub creation time, and the complete raw repository search item

Actor profile name, company, location, bio, and follower counts are not part of required completion because they require per-user detail calls and are not necessary to analyze push activity or contributor behavior. They may be added later as explicitly optional hydration.

Nullable fields remain valid when GitHub returns them as null. “Complete” means a valid response was durably observed and the contract was evaluated, not that every nullable value is populated.

Durable observations and projections

Preserve the original evidence as well as the convenient current view.

Add an append-only enrichment observation/audit model (exact naming may follow repository conventions) that records at least:

• entity kind and stable GitHub ID;
• source: event, search, or detail;
• observation time;
• complete raw item JSON;
• payload fingerprint;
• request/batch correlation and validation outcome.

For batch requests, also retain request metadata, requested identifiers, response status, total_count, incomplete_results, and the observed rate-limit headers. It is acceptable to store individual raw result items plus batch-envelope metadata rather than duplicating the whole response body.

Actor/repository tables remain the latest queryable projection and reference their latest successful observation/source/time. A refresh must not overwrite the only retained raw response. Identical payload content may be deduplicated by fingerprint only if distinct observation times remain representable.

The original raw push event remains authoritative and is never discarded by enrichment state changes.

Stages and outcomes

At minimum, make these conditions observable:

• event-native data persisted;
• local derivation complete;
• actor/repository batch pending;
• batch request in flight/leased;
• batch result applied;
• detail fallback pending;
• retry scheduled;
• required contract complete;
• deterministic entity-specific terminal outcome.

A terminal outcome is allowed only for an entity-specific fact such as a confirmed unavailable/deleted entity after the retry policy is exhausted. It must retain the event data, observations, reason, and timestamps. Quota exhaustion, service restart, worker contention, and transient GitHub failures are never terminal outcomes.

One hundred percent of captured entities must remain represented until they reach either the useful-data contract or a documented entity-specific terminal outcome.

Quota scheduler

Track Search and core quota as separate ledgers/windows using GitHub response headers.

Defaults should be conservative and all of the following must be configurable and published by /status:

• search request ceiling and safety reserve;
• search batch size;
• search pacing/concurrency;
• actor/repository fairness weights;
• core detail-fallback reserve;
• retry/backoff and stale-lease timings;
• refresh age and eligibility policy.

The normal batch worker uses only the expendable Search budget after the safety reserve. Actor and repository lanes receive a fair share and may borrow unused capacity from each other.

Polling retains priority in the core budget. Individual detail fallback may consume only its explicit core reserve/allowance and may never silently take the polling allocation. Keep the existing global outbound-request safety/serialization guarantees while accounting for the two rate-limit resources independently.

A conservative starting point, subject to implementation validation, is 10 Search requests/minute observed, 2/minute reserved, and batches of up to 10 exact entities. That yields a theoretical ceiling of 4,800 returned entity items/hour before misses and fallback—above the approximately 2,172–2,280 cold entities/hour seen in the short pressure sample. This is a capacity hypothesis, not proof of catch-up.

Catch-up requirement

Durability and catch-up are separate acceptance gates.

The implementation must expose and demonstrate:

• actor/repository arrivals per interval;
• batch attempts, requested item count, returned item count, and fill ratio;
• missing/invalid/incomplete results;
• detail fallbacks and their outcomes;
• completions per interval;
• pending count and oldest pending age by stage/entity kind;
• Search/core quota usage and reserve;
• backlog slope.

Under a sustained representative run, the measured completion rate must exceed the measured arrival rate and the backlog must show a negative slope while draining. If that is not true, the service must report that it is not keeping up; it must not claim eventual catch-up. Include test duration, inputs, rates, fallback percentage, and before/after backlog measurements in PR #43.

Required implementation work

• Replace per-entity normal-path detail fetching with actor and repository Search batch planners/workers.
• Build exact repeated-qualifier queries; do not join qualifiers with OR.
• Map results by stable GitHub ID, not result order or mutable login/name alone.
• Add durable leases/idempotency so crashes and concurrent workers cannot lose or double-apply work.
• Add partial-result, renamed-repository, missing-result, and incomplete_results fallback handling.
• Add the observation/audit persistence and latest projections described above.
• Remove all skipped_budget state, counters, documentation, and tests.
• Add separate Search/core quota accounting and tunable scheduler settings.
• Expand /status with the stage, throughput, batch-quality, quota, and catch-up metrics above.
• Update the implementation plan, README, configuration reference, status contract, and an ADR that records this decision and the payload-URL fallback amendment.
• Update PR Enrich in Search batches instead of one request per entity #43's description with the final design, commands, test results, and live unauthenticated evidence.

Tests and live verification

Automated coverage must include:

• event-native completion and local derivation without network calls;
• duplicate-demand coalescing;
• actor and repository batching;
• stable-ID validation;
• partial, empty, malformed, renamed, and incomplete batch responses;
• nullable contract fields;
• fallback admission and bounded core consumption;
• Search/core reserve enforcement and lane fairness/borrowing;
• quota deferral across windows;
• retry, stale lease, restart, and concurrent-worker behavior;
• append-only raw observations and projection refresh;
• the absence of any quota-based terminal/drop state;
• /status backlog slope and per-stage metrics;
• configuration validation.

Run a real unauthenticated test against GitHub with no credentials present. At minimum, prove both actor and repository batch paths, capture the Search rate-limit resource/limit headers, exercise a missing-or-renamed fallback case, and show persisted observations/projections and pending work surviving a quota boundary or controlled restart. Record sanitized commands and output in PR #43.

The exploratory probe already established these assumptions and should be repeated through the implemented path:

• repeated exact user: qualifiers returned all five requested actors;
• repeated exact repo: qualifiers returned nine of ten requested repositories with incomplete_results=false;
• the missing facebook/react identifier redirects to react/react, demonstrating why stable-ID validation and fallback are required;
• both responses reported the search rate-limit resource with a limit of 10;
• joining exact qualifiers with OR produced HTTP 422 and is not the supported batching form.

Out of scope

• any authenticated GitHub token or credential;
• complete user-profile hydration, including actor profile name;
• object-storage work for avatars/raw events;
• pausing polling to make room for enrichment;
• sampling, dropping, or marking work terminal because budget was unavailable.

Delivery

Implement this on top of branch agent/durable-enrichment-backlog and add it to draft PR #43. Preserve the durable-backlog work from #44 while replacing its one-request-per-entity capacity assumption with this staged batch design.