feat(webhook): provider-agnostic receiver (push → warm), GitHub phase 1#70
Merged
Conversation
Self-hosters have no automatic warming today — they must call /sync or write a CI Action. Design a built-in webhook receiver on ripclone-server: verify the provider signature, normalize the payload, and enqueue a sync on the existing build queue. Provider-agnostic via a WebhookProvider trait (GitHub first, then GitLab/Gitea). Documents how this converges with the managed cloud at the build queue + per-job credential (#55), so self-host gets identical warm-on-push. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a built-in webhook endpoint so a provider push auto-enqueues a sync,
reusing the existing build queue. No CI Action, no glue — the same
warm-on-push the managed cloud gives you.
- `webhook` module: `WebhookProvider` trait (`verify` over the RAW body,
`parse` → `CanonicalEvent`) + `WebhookConfig` (per-provider secret +
optional allowlist). Structured so GitLab/Gitea are later trait impls.
- GitHub adapter: `X-Hub-Signature-256` HMAC-SHA256 over the raw body
(constant-time compare via `subtle`), `X-GitHub-Event` routing; parses
push / branch-delete / ping.
- `POST /webhooks/{provider}` in server.rs, under `rate_limited` but NOT
`auth_middleware` (the HMAC is the auth). Reads raw bytes before JSON,
looks up the ProviderInstance, verifies, parses, dispatches.
- Factor `enqueue_sync(state, repo, branch, rev, cred)` out of
`sync_repo_inner`; both `/sync` and the webhook call it — no duplicated
build logic. The webhook drops the handle (fire-and-forget warming).
- Branch-delete → `RefStore::delete_branch` (file + S3 + caching impls),
never builds.
- Config: `RIPCLONE_WEBHOOK_SECRET_<provider>`, StaticBroker credential for
private clones, optional `RIPCLONE_WEBHOOK_ALLOWLIST`. No secret ⇒ 503.
Resolved the doc's open questions with the recommended defaults: allow-all
allowlist + loud startup log; always warm the default branch, other branches
only if already tracked.
Tests: signature verify (valid/invalid/missing), GitHub parse, enqueue
invoked on push, branch-delete cleanup, allowlist gating, no-secret ⇒ 503,
tracked/untracked non-default branch. `fmt` + `clippy -D warnings` + full
release test suite green.
GitLab (`X-Gitlab-Token`) and Gitea (`X-Gitea-Signature`) are follow-ups
behind the same trait.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
From two independent adversarial reviews of the receiver: Security/correctness: - Cap webhook body at 25 MiB (MAX_WEBHOOK_BODY_BYTES) instead of the global 256 MiB. The HMAC can only be checked after the whole body is buffered, so an unauthenticated caller must not be able to make the server hold 256 MiB before the 401. Oversized → 413. - Validate the payload-derived branch (validate_git_rev) in both the push and delete handlers before it reaches the queue/git. Contained before (storage keys are slugged, git re-validates), but makes the trust boundary explicit and skips a doomed enqueue. - Document that branch-delete cleanup intentionally skips the push allowlist (a non-allowlisted repo was never warmed, so delete is a safe no-op). Testability + coverage: - Extract `parse_secret` / `parse_allowlist` from `from_env` and unit-test them: empty secret ⇒ no secret (fail closed, no empty HMAC key), allowlist trimming/empty-dropping. - GitHub verify: valid-hex-but-wrong-length signature (the exact ct_eq length-mismatch branch the comment claims is safe) + correct-length wrong bytes. - Handler tamper test: sign body A with the right secret, deliver body B ⇒ 401 (proves verification is over the raw received bytes). - Coalescing: two identical signed pushes ⇒ exactly one queued build. - Tag delete ignored; hostile branch name rejected. - CachingRefStore::delete_branch evicts the cache (not just the file). fmt + clippy -D warnings + full release suite green (lib 202 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mirror Address the two items previously left as low-severity notes: - Branch-delete now applies the same repo allowlist as push, so the receiver acts only on in-scope repos symmetrically (an out-of-scope delete is ignored rather than silently mutating refs). - Default-branch policy no longer depends solely on the payload: when a provider omits `repository.default_branch`, fall back to the local mirror's HEAD (populated by any prior sync), exactly as sync_repo_inner resolves HEAD. GitHub always sends it; this keeps the policy correct for future GitLab/Gitea adapters. A brand-new repo with neither stays untracked until first warmed (fail-safe). Tests: default branch resolved from the mirror when the payload omits it; no-default + no-mirror + untracked stays ignored; delete outside the allowlist leaves refs untouched. fmt + clippy -D warnings + full release suite green (lib 205 passed, 51 test binaries, 0 failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Main shipped its own webhook + polling (#67) while #70 was in review. Both rewrote the same enqueue block and both added a webhook. Reconciled by taking the best of each: - Adopt main's engine wholesale: `trigger_build` (shared fire-and-forget enqueue used by /build, the webhook, and the poll loop), presence-based coalescing, the polling fallback, and `sync_repo_inner` as-is. Dropped my `enqueue_sync` refactor entirely (superseded). - Keep my richer receiver, now wired to `trigger_build`: provider-agnostic `WebhookProvider` trait (GitLab/Gitea-ready), per-provider secrets + allowlist (`WebhookConfig`), default-vs-tracked branch policy, branch-delete cleanup (`RefStore::delete_branch`), constant-time verify over the raw body. - One receiver, two URLs: `/webhooks/{provider}` is canonical; `/v1/webhooks/github` is a back-compat alias into the same handler. Removed main's `github_webhook_handler`/`verify_github_signature`/`GithubPushEvent`. - Config back-compat: `RIPCLONE_WEBHOOK_SECRET` still honored as the github secret; `RIPCLONE_WEBHOOK_WARM_ALL=1` opts into main's warm-every-branch behavior (default stays conservative). Replaced the `webhook_secret` field with `webhook_config`. - Updated main's webhook unit tests + e2e_webhook integration tests to the unified payload shape/responses; kept the polling test. fmt + clippy -D warnings + full release suite green (lib 211 passed, 53 test binaries, 0 failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A built-in webhook receiver so a provider push auto-enqueues a sync (push → warm), reusing the existing build queue. No CI Action, no glue — the same warm-on-push the managed cloud gives you. Phase 1 = GitHub; structured so GitLab/Gitea are later trait impls.
Implements
docs/WEBHOOKS.md(the design doc is the first commit on this branch).Changes
webhookmodule (rust/src/webhook/):WebhookProvidertrait (verifyover raw body,parse→CanonicalEvent { kind, repo, ref_, after, default_branch, private }) +WebhookConfig(per-provider secret + optional allowlist).X-Hub-Signature-256HMAC-SHA256 over the raw body, constant-time compare viasubtle;X-GitHub-Eventrouting; parses push / branch-delete / ping.POST /webhooks/{provider}inserver.rs, registered under therate_limitedlayer but notauth_middleware(the HMAC is the auth). Reads raw bytes before JSON, looks up theProviderInstance, verifies, parses, dispatches.enqueue_sync(...)factored out ofsync_repo_innerand called from both/syncand the webhook — no duplicated build logic. The webhook drops the handle (fire-and-forget; responds 2xx fast).RefStore::delete_branch(file + S3 + caching impls); never builds.RIPCLONE_WEBHOOK_SECRET_<provider>, StaticBroker credential (queue: carry the per-request upstream token to the cross-process worker #55) for private clones, optionalRIPCLONE_WEBHOOK_ALLOWLIST. No secret ⇒ 503, bad signature ⇒ 401.Resolved open questions (recommended defaults)
RIPCLONE_WEBHOOK_ALLOWLIST.{provider}is theProviderInstanceid; the secret is keyed per instance id.Tests
Signature verify (valid / invalid / missing), GitHub parse, enqueue invoked on push, branch-delete cleanup, allowlist gating, no-secret ⇒ 503, tracked vs untracked non-default branch.
cargo fmt --check+cargo clippy --all-targets -- -D warnings+ the full release test suite (cargo test --release --all-targets --locked) are green.Follow-ups
GitLab (
X-Gitlab-Token) and Gitea/Forgejo (X-Gitea-Signature) adapters — each is oneWebhookProviderimpl plus a match arm inwebhook::provider_for.🤖 Generated with Claude Code