Spike: middleware/webhook — receiver with HMAC verification + provider presets

[FumingPower3925]

Context

Nearly every SaaS-integrated backend has to receive webhooks from third-party providers (Stripe, GitHub, Shopify, Slack, Twilio, Discord, Dropbox, DocuSign, Zoom, GitLab, Intercom, …). The Go ecosystem has no unified middleware for this today — teams either hand-roll HMAC verification per provider or depend on a scattering of provider-specific SDKs (stripe-go ships its own, go-github ships its own, each with a different API surface). Ship middleware/webhook to fill that gap.

Outbound webhook sending (durable queue + retry + dead-letter) is explicitly out of scope — that's a service concern, not a per-request middleware, and better served either by a separate standalone package or by pointing users at existing projects (svix, hookdeck, standard-webhooks reference implementations). This spike is receiver-only.

Why now

v1.5.0 already ships the primitives this middleware needs:

• Zero-copy body (Stream.rawBody + H1State.bodyBuf via SetRawBody) — HMAC can run against the raw slice without a second allocation or a parse-then-rebuild round-trip that would break the signature.
• middleware/idempotency (new this release) — webhook providers deliver duplicates as a matter of policy; a Webhook-ID / Idempotency-Key header naturally plugs into the existing key/state-machine.
• middleware/store.KV (new this release) — backs both the idempotency cache and (optionally) the secret-rotation registry via any of the Redis / PG / Memcached adapters.
• middleware/requestid — correlation between webhook delivery attempts and server logs.

The middleware is largely composition over work already done; no new core surface expected.

Scope

In scope

• Core verifier supporting pluggable algorithms:
  • HMAC-SHA256 (most common — Stripe, Slack, Shopify, GitLab, …)
  • HMAC-SHA1 (legacy — GitHub still emits both SHA1 and SHA256)
  • Ed25519 (Discord, modern providers)
  • Constant-time comparison (crypto/subtle.ConstantTimeCompare) everywhere.
• Canonical signing-string builders per-provider (the "what bytes are hashed" piece is the fiddly one — some providers hash timestamp . "." . body, some hash method . "\n" . path . "\n" . timestamp . "\n" . body, some hash the raw body only, some hash a canonicalized URL form). Abstraction: SigningStringFn func(*celeris.Context, rawBody []byte) []byte.
• Freshness window (replay protection): reject signatures whose timestamp is outside ±TolerancePeriod (default 5 min per Stripe's recommendation).
• Secret rotation: accept a SecretsFn func(*celeris.Context) [][]byte that returns one or more keys; signature must verify against any active key (allows rolling to a new secret without downtime). Back-compatible with a single Secret []byte static setter.
• Provider presets (subpackages): Stripe, GitHub, Shopify, Slack, Twilio, GitLab, Discord, DocuSign. Each exports a single New(secret string, opts ...Option) celeris.HandlerFunc that wires the core verifier with that provider's signing-string + header conventions. Target coverage: the top 8 providers cover > 80 % of real-world webhook traffic.
• Response shape on failure: opinionated but override-able. Default: 401 for missing/invalid signature, 400 for malformed timestamp, 403 for stale timestamp. No body disclosure that would leak signature internals to scanners.
• Idempotency integration: optional Idempotency store.KV field → when set, middleware wires middleware/idempotency with the webhook's event-id header (per-provider) as the idempotency key so replays become 200 OK fast-path without re-invoking the handler.
• Body capture: verifier runs before any parser; handler still gets the parsed body via c.BindJSON / c.Body() (which now just returns the already-captured bytes thanks to Stream.rawBody).
• Test helpers in middleware/webhook/webhooktest/: signed-request builders per provider, fake-clock support, preset-roundtrip tests so users can unit-test their webhook handlers without writing signature code.

Out of scope (explicit non-goals)

• Outbound webhook delivery. Durable queue, retry with exponential backoff + jitter, dead-letter routing, per-destination rate limits, subscription registry — those belong in a separate service/package (webhook-sender/? standalone lib?). File as a v1.6.0+ follow-up if adoption signals demand it.
• Webhook forwarding / proxy (dev-time ngrok-style tunnelling). Standalone tools cover this well (cloudflared, ngrok, hookdeck).
• Provider-specific business-logic helpers (e.g. Stripe event-type routing tables, GitHub event-kind enums). Receiver verifies + delivers raw JSON; parsing is the user's problem or a job for the provider's own SDK. Scope creep otherwise.

Design sketch

package webhook

type Config struct {
    Secret          []byte                                         // static key (convenience)
    SecretsFn       func(c *celeris.Context) [][]byte              // dynamic / rotating keys
    SigningString   func(c *celeris.Context, rawBody []byte) []byte
    Algorithm       Algorithm                                      // HMACSHA256 | HMACSHA1 | Ed25519
    SignatureHeader string                                         // e.g. "X-Signature", "Stripe-Signature"
    SignatureParser func(header string) (sig []byte, ts time.Time, err error)
    Tolerance       time.Duration                                  // 0 → default 5 min; -1 → disable
    Now             func() time.Time                               // test injection

    Idempotency     store.KV // optional; wires middleware/idempotency
    EventIDHeader   string   // provider's event-id header name (for idempotency key)

    OnVerifyError   func(c *celeris.Context, err error) error      // override default 401/400/403
}

func New(cfg Config) celeris.HandlerFunc

Per-provider presets live at middleware/webhook/stripe, middleware/webhook/github, etc. Each is a thin preset:

package stripe

func New(secret string, opts ...webhook.Option) celeris.HandlerFunc {
    return webhook.New(webhook.Config{
        Secret:          []byte(secret),
        SigningString:   signStripe,       // timestamp + "." + body
        Algorithm:       webhook.HMACSHA256,
        SignatureHeader: "Stripe-Signature",
        SignatureParser: parseStripeHeader, // t=ts,v1=sig,v1=sig,...
        Tolerance:       5 * time.Minute,
        EventIDHeader:   "Stripe-Idempotency-Key",
    }, opts...)
}

Exit criteria

1. Core middleware/webhook package boots with a handcrafted signing scheme (HMAC-SHA256, header name configurable) and passes a signed-request round-trip test.
2. Provider presets for Stripe, GitHub, Shopify, Slack shipped (4 is enough to prove the preset template; the rest follow mechanically).
3. Freshness window, secret rotation, constant-time compare all covered by unit + fuzz tests.
4. Idempotency wiring: end-to-end test using middleware/idempotency + middleware/store.NewMemoryKV, plus one integration test against Redis via middleware/store/redisstore.
5. Zero heap allocations on the verify-success hot path (strict-alloc budget asserted; tied to the v1.5.0 Stream.rawBody zero-copy contract).
6. middleware/webhook/webhooktest/ ships a signed-request builder per preset + a fake clock, so user-level integration tests are a 3-line setup.
7. README + godoc with one end-to-end example per preset.
8. Benchmark: verify path (HMAC-SHA256 over 4 KiB body + timestamp check) on msr1. Target: ≤ 1 µs / req for the verifier itself, dwarfed by handler cost.

Critical files (expected)

• New submodule: middleware/webhook/{webhook.go, config.go, algorithm.go, errors.go, doc.go, *_test.go}
• New sub-presets: middleware/webhook/{stripe,github,shopify,slack}/preset.go
• New test helpers: middleware/webhook/webhooktest/webhooktest.go
• No core celeris changes expected. If any surface, flag as scope creep.

Related

• Spike: native gRPC support (unary, server/client/bidirectional streaming) #246 Spike: native gRPC — opposite effort shape (core-heavy). This spike validates that not-every-new-surface needs core work, keeping release velocity healthy.
• Spike: native GraphQL middleware (transport, subscriptions, APQ) #247 Spike: GraphQL middleware — same "middleware-only, compose on existing primitives" philosophy.
• middleware/idempotency + middleware/store.KV (both v1.5.0) — primary integration points; this spike is a real-world consumer of both.
• v1.5.0 zero-copy body (Stream.rawBody) — the reason HMAC over the raw bytes is free.

[eben-vranken]

Built a standalone Go webhook verification library recently that covers a lot of the same ground: trusthook.

The provider preset model and scheme abstraction might be worth a look for reference. The benchmarks might also be useful as a baseline: Stripe-style HMAC-SHA256 over a typical body comes in around ~1.1 µs on an AMD Ryzen 5 5600X, which lines up with your ≤ 1 µs target for the verifier itself.