diff --git a/docs/DESIGN_BRIEF.md b/docs/DESIGN_BRIEF.md index 2c7fe75..af214de 100644 --- a/docs/DESIGN_BRIEF.md +++ b/docs/DESIGN_BRIEF.md @@ -2,19 +2,18 @@ This Rails 8.1 API service samples GitHub's public Events API, stores `PushEvent` records and their raw payloads in PostgreSQL, and enriches referenced actors and -repositories without a token. The README is the runbook; this brief explains the -decisions. Detailed arguments live in the [ADRs](adr/), and execution history lives in -[`IMPLEMENTATION_PLAN.md`](../IMPLEMENTATION_PLAN.md). +repositories without a token. The README is the runbook; the [ADRs](adr/) and +[`IMPLEMENTATION_PLAN.md`](../IMPLEMENTATION_PLAN.md) hold the detailed arguments and +execution history. ## Business problem and constraint The assignment asks for durable ingestion, enrichment, and restart safety. The difficult constraint is the source: `/events` is a sliding window with documented delivery latency, -while unauthenticated callers share 60 requests per hour per outbound IP. An event can -leave the window before this service observes it, and enrichment demand can exceed the -remaining budget by orders of magnitude. The product is therefore an observable, -bounded sampler, not a mirror. It makes no guarantee of complete upstream capture or -complete enrichment. +and unauthenticated callers share 60 requests per hour per outbound IP — an event can +leave the window unobserved, and enrichment demand can exceed the remaining budget by +orders of magnitude. The product is therefore an observable, bounded sampler, not a +mirror, with no guarantee of complete upstream capture or complete enrichment. ## Architecture, data, and durability @@ -37,30 +36,30 @@ flowchart LR ``` Every outbound call passes through `Github::RequestExecutor`. A global session advisory -lock serializes request reservation and execution, so the singleton budget ledger cannot -race itself. Polling also holds a per-source advisory lock for its whole cycle; the only -valid order is source lock, then request gate. Enrichment takes only the gate. Web routes -have no path to the executor, so `/health/*` and `/status` cannot spend GitHub budget. +lock serializes reservation and execution so the singleton budget ledger cannot race +itself; polling also holds a per-source advisory lock for its whole cycle (the only valid +order is source lock, then gate), and enrichment takes only the gate. Web routes have no +path to the executor, so `/health/*` and `/status` cannot spend budget. PostgreSQL is the system of record. Seven business tables hold source/run state, `push_events`, shared actor and repository state, quarantined payloads, and the global -budget ledger; Solid Queue uses a separate database in the same server. Raw payloads use -`jsonb`, preserving JSON meaning rather than byte layout. Quarantine identity is a -canonical-payload SHA-256 because malformed data may have no usable GitHub event ID. +budget ledger; Solid Queue uses a second database in the same server. Raw payloads are +`jsonb` — JSON meaning, not byte layout — and quarantine identity is a canonical-payload +SHA-256, since malformed data may lack a usable event ID. Acceptance occurs when a `push_events` row commits. Inserts use `ON CONFLICT (github_event_id) DO NOTHING RETURNING id`. A duplicate observation cannot create a second event row, and entity activity or `skipped_budget` reactivation occurs only when `RETURNING` yields a new row. A replay may still refresh permitted identity fields. -These are deliberately narrow invariants: executions, ingestion runs, quarantine -occurrence counts, budget debits, and logs can repeat or change. +These invariants are deliberately narrow: executions, runs, quarantine counts, budget +debits, and logs can repeat. Work committed before a crash remains durable. Advisory locks disappear with their sessions, entity leases expire by timestamp, and a reconciler rebuilds missing enrichment -work from committed entity rows, making cross-database enqueue a hint rather than the -durability boundary. Work lost before commit is recoverable only if the event remains in a -later response from the sliding feed; advancing out of that window is an acknowledged loss -mode. This is not exactly-once execution ([ADR 0005](adr/0005-at-least-once-with-idempotent-writes.md), +work from committed entity rows — the cross-database enqueue is a hint, not the durability +boundary. Work lost before commit is recoverable only while the event remains in a later +feed response; leaving the window is an acknowledged loss mode. This is not +exactly-once execution ([ADR 0005](adr/0005-at-least-once-with-idempotent-writes.md), [ADR 0008](adr/0008-post-commit-enqueue-and-entity-scoped-reconciliation.md)). ## Request budget and the `304` finding @@ -75,54 +74,50 @@ actor_guarantee = floor(enrichment_allowance × ACTOR_ENRICHMENT_SHARE) repository_guarantee = enrichment_allowance - actor_guarantee ``` -With the defaults, polling receives 12 attempts, the reserve is 8, and enrichment receives -40 attempts split 20/20. Startup rejects configurations leaving no enrichment allowance. -The source count comes from enabled in-service rows at window initialization, with the -configured count as a boot-time fallback ([ADR 0004](adr/0004-class-aware-budget-ledger.md), +With defaults: 12 poll attempts, a reserve of 8, and 40 enrichment attempts split 20/20; +startup rejects configurations that leave no enrichment allowance, and the live source +count is read from in-service rows at window initialization +([ADR 0004](adr/0004-class-aware-budget-ledger.md), [ADR 0009](adr/0009-runtime-source-allocation-and-shared-ip-observability.md)). -GitHub's endpoint documentation broadly describes `304` responses as free, while its REST -best-practices guidance scopes that behavior to correctly authorized requests. A dated -unauthenticated probe observed `x-ratelimit-used` increase across a `304`, so this system -debits every unauthenticated conditional request. ETags still save bandwidth but not -budget. Treating a charged response as free risks exhausting the shared window; treating a -free one as charged costs only local opportunity. The [probe transcript](evidence/2026-07-30-unauthenticated-304-quota-probe.md) -records the evidence. +GitHub's endpoint documentation broadly describes `304` responses as free, but its REST +best-practices guidance scopes that to correctly authorized requests — and a dated +unauthenticated [probe](evidence/2026-07-30-unauthenticated-304-quota-probe.md) observed +`x-ratelimit-used` increase across a `304`. This system therefore debits every conditional +request: ETags save bandwidth, not budget. The asymmetry decides it — a wrongly-free `304` +can exhaust the shared window, while a wrongly-charged one costs only local opportunity. ## Bounded, fair, and safe enrichment -One observed page contained about 90 distinct actors and 90 repositories: roughly 180 cold -lookups competing for 40 hourly attempts. Candidates therefore age from `pending` to the -terminal `skipped_budget` state after an eligibility window. Only a newly inserted push -event can reactivate a skipped entity. This bounds actionable backlog while `/status` -reports pending, skipped, and sampled proportions. +One observed page held roughly 180 distinct actors and repositories — cold lookups +competing for 40 hourly attempts. Candidates therefore age from `pending` to the +terminal `skipped_budget` state after an eligibility window, and only a newly inserted +push event can reactivate a skipped entity. This bounds actionable backlog, and +`/status` reports the sampled proportions. Each entity class receives a guaranteed share. A class may borrow beyond it only when the other has no currently eligible candidate; the ledger independently refuses invalid -reservations. Never-enriched candidates precede stale refreshes, and selection plus leases -prevents concurrent duplicate work. This keeps a repository-heavy feed from starving actor -enrichment without wasting an idle share ([ADR 0007](adr/0007-enrichment-fairness-shares-and-borrowing.md)). - -Payload, pagination, and redirect URLs are attacker-influenceable. The SSRF boundary allows -only HTTPS URLs whose host is exactly `api.github.com`, with no userinfo, non-default port, -or IP literal. Redirects are bounded, revalidated, and separately debited. Fixture mode -fails closed on an unknown URL and never falls back to the network +reservations. Never-enriched candidates precede stale refreshes, and selection plus +leases prevent concurrent duplicate work, so a repository-heavy feed cannot starve actor +enrichment ([ADR 0007](adr/0007-enrichment-fairness-shares-and-borrowing.md)). + +Payload, pagination, and redirect URLs are attacker-influenceable, so the SSRF boundary +allows only HTTPS URLs whose host is exactly `api.github.com` — no userinfo, non-default +port, or IP literal; redirects are bounded, revalidated, and separately debited, and +fixture mode fails closed rather than falling back to the network ([ADR 0003](adr/0003-event-source-and-transport-seams.md)). ## Tradeoffs, omissions, and scaling -| Decision | Deliberate cost | -|---|---| -| `jsonb` semantic retention | Whitespace, object-key order, and duplicate keys are lost | -| Session advisory locks | Lock ownership needs dedicated observability | -| At-least-once job execution | Duplicate work and non-event side effects may repeat | -| Solid Queue over Kafka | PostgreSQL bounds queue throughput and fan-out | +Each decision has a deliberate, recorded cost: `jsonb` semantic retention loses +whitespace, object-key order, and duplicate keys; session advisory locks need dedicated +lock-ownership observability; at-least-once execution can repeat duplicate work and +non-event side effects; and Solid Queue over Kafka lets PostgreSQL bound queue +throughput and fan-out. Authentication, object storage, extra entity APIs, Kafka, and a frontend were omitted to -keep the submission focused on correctness at the stated quota. The current bottleneck is -upstream allowance, not local throughput. A larger authenticated budget could materially -increase feasible coverage, but would not establish complete capture or enrichment: feed -window loss, failures, demand, and shared-budget effects still apply. At higher sustained -throughput, queue capacity, database contention, and source partitioning should be measured -before introducing a broker. The API version remains pinned to `2022-11-28`; any upgrade -must revalidate payloads, redirects, headers, and `304` accounting first. +keep the submission focused on correctness at the stated quota. The bottleneck is +upstream allowance, not local throughput: an authenticated budget would raise coverage +but not make capture or enrichment complete. Queue capacity and database contention +should be measured before any broker, and the pinned `2022-11-28` API version +revalidated before upgrade. diff --git a/docs/SUBMISSION_CHECKLIST.md b/docs/SUBMISSION_CHECKLIST.md index 96a0eec..9d8c443 100644 --- a/docs/SUBMISSION_CHECKLIST.md +++ b/docs/SUBMISSION_CHECKLIST.md @@ -3,14 +3,19 @@ Repository: https://github.com/batbrainy/github-push-ingestor This is a reusable runbook, not a record of one run. Keep the boxes unchecked in the -repository. The external findings report records the default-branch SHA, UTC date, command, -exit code, salient output, duration, and one classification for every gate: pass, repository -defect, environment issue, or documentation mismatch. - -Run every gate against the default branch after the final hardening change merges, from a -fresh clone — never against a working tree. A working tree can hide an untracked `.env` or -`config/master.key`, while Compose's fixed project name can make even a fresh clone reuse a -stale image or the globally named `github-push-ingestor_pgdata` volume. +repository. The external findings report — most recently +[`docs/evidence/2026-08-01-post-merge-verification.md`](evidence/2026-08-01-post-merge-verification.md) +— records the verified refs, UTC date, command/assertion synopses, salient output, duration, +and one classification for each reported gate or gate group: pass, repository defect, +environment issue, or documentation mismatch. The exact commands remain in this checklist. + +Run every application, runtime, and repository-history gate against the default branch +after the final hardening change merges, from a fresh clone — never against a modified +working tree. A documentation-only gate for content not yet merged may use the final PR +blob when the report records that blob separately and does not attribute it to the runtime +checkout. A modified runtime tree can hide an untracked `.env` or `config/master.key`, +while Compose's fixed project name can make even a fresh clone reuse a stale image or the +globally named `github-push-ingestor_pgdata` volume. --- diff --git a/docs/evidence/2026-07-31-container-kill-recovery.md b/docs/evidence/2026-07-31-container-kill-recovery.md index 2724592..8a4129f 100644 --- a/docs/evidence/2026-07-31-container-kill-recovery.md +++ b/docs/evidence/2026-07-31-container-kill-recovery.md @@ -12,7 +12,9 @@ Status: Historical first-party observation; superseded for submission gating > recreation. A recreated worker could therefore default to live mode and spend GitHub > budget. The transcript is preserved verbatim below as a historical observation. Re-run the > corrected script against the final default-branch SHA and record that result in the -> external findings report. +> external findings report. That re-run is now recorded: +> [`2026-08-01-post-merge-verification.md`](2026-08-01-post-merge-verification.md) gate 1.18 +> ran the corrected script against `88e2260c` — 45 checks passed, 0 failed. ## Why this verification exists diff --git a/docs/evidence/2026-08-01-post-merge-verification.md b/docs/evidence/2026-08-01-post-merge-verification.md new file mode 100644 index 0000000..c1e9500 --- /dev/null +++ b/docs/evidence/2026-08-01-post-merge-verification.md @@ -0,0 +1,93 @@ +# Post-merge verification — external findings report + +Date: 2026-08-01 (UTC) + +Status: Verification of runtime behavior and repository-history gates at the default-branch +SHA below, plus documentation-only gates against the PR #42 content identified separately + +```text +Runtime checkout: 88e2260c7f20fea06cdacb88d62132caaed1fc14 (default branch) +Design-brief source: blob af214de7a2fc5c1ee3716ec7ff1ee2e73df21b06 + (introduced by PR commit 513834c4199fa5abea206bf5b379126167607f47) +Clone: fresh `git clone` into a newly created temporary parent directory +Run window (UTC): 2026-08-01T14:22:35Z → 14:30:35Z (8m00s, single uninterrupted run) +Docker version: 28.3.0 (Docker Desktop) +Compose version: v2.38.1-desktop.1 +Host: Darwin 25.5.0 arm64 (macOS, Apple Silicon) +Local toolchain: no host Ruby, PostgreSQL, or psql used at any point +``` + +The runtime gates below ran in the order listed, in one scripted pass against the clean +runtime checkout; a failing assertion would have aborted the run at that gate. The render +gate used the exact design-brief blob identified above, which was not part of that default- +branch checkout. The command/assertion cells are synopses keyed to the exact commands in +the checklist. Classification vocabulary is the checklist's: pass, repository defect, +environment issue, or documentation mismatch. + +## Section 1 — authoritative clean-checkout verification + +| Gate | Command / assertion | Salient output | Duration | Result | +|---|---|---|---|---| +| 1.1 | clone, `git rev-parse HEAD`, clean tree, no `.env`/`master.key` | HEAD `88e2260c`; porcelain empty | 1s | pass | +| 1.2 | `down -v --remove-orphans`; volume inspection must fail | `github-push-ingestor_pgdata` absent | 1s | pass | +| 1.3 | `docker compose build --no-cache --pull` | cold image built | 70s | pass | +| 1.4 | `docker image rm …-app:latest`; `up --build -d`; `ps --all` | second cold path recreated the image | 13s | pass | +| 1.5 | topology | exactly `db` healthy, `setup` exited 0, `web` healthy, `worker` running; no tools service | <1s | pass | +| 1.6 | `curl -fsS …/health/ready` | `{"status":"ok"}` | 3s | pass | +| 1.7 | live worker reaches GitHub, no token | `ingestion.run_completed`: 1 page, 100 events received, 97 push events created, 3 ignored, `next_poll_at` +5m; budget row debited | 61s | pass | +| 1.8 | fixture boundary: `down -v`, volume absent, `GITHUB_MODE=fixture up --build -d db setup web` | offline stack ready | 13s | pass | +| 1.9 | fixture ingest, captured via `tee` | `4 created / 3 quarantined / 1 ignored` | 3s | pass | +| 1.10 | `enrich --limit 6` | `complete 2 / permanent_failure 1` per entity class | 4s | pass | +| 1.11 | SQL state | exactly `4 / 3 / 3 / 3 / 3` | 1s | pass | +| 1.12 | replay after 60s poll floor, `ingest --force` | `Duplicates skipped: 4`; no `enrichment.reactivated` in captured output | 65s | pass | +| 1.13–1.17 | three suite runs (`run --rm --build test`, `run --rm test`, fixed seed 4242) with dev-DB observations before/after | 1,734 + 10 examples, 0 failures, three consecutive times; dev `push_events`, `solid_queue_jobs`, and `setup` container finish time unchanged | 87s | pass | +| 1.18 | empty volume, offline full stack, `GITHUB_MODE=fixture script/verify_recovery.sh --confirm` | 45 checks PASS, 0 FAIL; "Every check above passed."; worker `GITHUB_MODE=fixture` after | 134s | pass | +| 1.19 | `push_events` across `docker compose restart` | equal | 2s | pass | +| 1.20 | `push_events` across `down --remove-orphans` / `up --build -d` | equal | 12s | pass | + +## Section 4.5 — endpoint budget isolation + +`md5(row_to_json(github_api_budget))` and all four request counters were captured, the +three endpoints (`/health/live`, `/health/ready`, `/status`) were each called three times, +and the hash and counters were byte-identical afterward. The worker had never started in +this stack. Result: **pass**. + +## Section 5 — design-brief render + +`docs/DESIGN_BRIEF.md` at blob +`af214de7a2fc5c1ee3716ec7ff1ee2e73df21b06` (the content introduced by PR #42, not the +base runtime checkout) rendered to PDF at US Letter, 0.75-inch `@page` margins, +11pt/1.35 sans-serif, browser default body margin zeroed, GFM conversion via `marked`, +Mermaid diagram rendered, printed with +headless Chrome. `pdfinfo`: **2 pages, 612 × 792 pts (letter)**; visual inspection found no +clipping, overlap, or malformed layout; the final paragraph ends on page 2. The QA PDF is +not committed. Result: **pass** (page count is renderer-dependent; parameters above +reproduce it). + +## Sections 6 and 7 — final repository review + +The security tools and history scans used the runtime checkout; the documentation scans +included the PR content described above. + +- `bin/brakeman --quiet --no-pager --exit-on-warn --exit-on-error` in the container: + 0 security warnings, 0 errors. `bin/bundler-audit` in the container: advisory database + cloned fresh, no vulnerabilities. Result: **pass**. +- Secret scans — working tree token grep, `git log -p --all` token grep, and + history-filename grep for `.env`/`master.key`: no matches. Result: **pass**. +- Stale-documentation grep: no matches. Result: **pass**. +- Forbidden-claim scan: 24 hits in the recorded pass, while this report was still untracked. + A follow-up scan of committed PR revision `513834c` found 25; the additional hit is this + report's own negation. Every hit is a negation, a stated limitation, or the scan's own + vocabulary. No affirmative claim of singular execution, exhaustive capture, or exhaustive + enrichment. Result: **pass**. + +## What this run does not show + +- One machine, one date, one operating system (macOS/arm64 with Docker Desktop). It does + not demonstrate other hosts or architectures beyond what CI covers. +- The live phase observed one successful unauthenticated poll and its budget debit — a + reachability and accounting check, not sustained live operation. +- Runtime behavior is attested at `88e2260c`; the render result applies only to the exact + design-brief blob recorded above. All changes after the runtime SHA are documentation + only, so every runtime command a reviewer runs executes the verified application code + unchanged.