** GitHub API Rate Limit Resilience with Intelligent Backoff and Queue Pausing

[github-actions]

Strategic Improvement

** GitHub API rate limit hits cause pipeline failures and daemon stalls. Current handling is basic. Implement intelligent rate limit monitoring, exponential backoff with jitter, queue pausing when limits approached, and priority routing (critical ops bypass queue). Prevents cascading failures and improves daemon reliability in high-volume scenarios.

Acceptance Criteria

• Rate limit monitor checks X-RateLimit-* headers on every GitHub API call, caches state
• Exponential backoff (2^n seconds with jitter) when rate limit hit, respects Retry-After header
• Queue pausing: daemon stops dispatching new pipelines when remaining quota < threshold (configurable, default 100)
• Priority bypass: critical ops (webhook processing, deployment status) use reserve quota
• Metrics: rate_limit_hits, backoff_delays, queue_pauses, priority_bypasses
• Test suite with mock GitHub API returning 403 rate limit responses
• Dashboard widget showing rate limit status (remaining, reset time, queue paused state)
• Config: rate_limit_threshold, backoff_base, backoff_max, priority_reserve_quota

Context

• Priority: ** P0
• Complexity: ** standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: ** P0: Reliability & Success Rate

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Core reliability improvement for daemon's GitHub API integration; well-defined scope spanning GitHub API layer, daemon dispatch, metrics, and dashboard with clear acceptance criteria.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	30083012283

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (315 lines)

Implementation Plan: GitHub API Rate Limit Resilience

Issue #807 — Intelligent Backoff, Queue Pausing, and Priority Routing (P0)

Context & Current State

Shipwright already has reactive, failure-count rate-limit handling in
scripts/lib/daemon-state.sh:99-130 (gh_rate_limited, gh_record_failure,
gh_record_success) — it counts consecutive gh failures and backs off 30s→300s.
It never reads X-RateLimit-* headers, so it is blind until calls already fail, and
it lives only in the daemon. GitHub API calls are scattered across sw-github-graphql.sh,
sw-github-checks.sh, sw-github-deploy.sh, and dozens of gh api sites — there is no
central wrapper. The daemon has a PAUSE_FLAG mechanism (daemon-state.sh:11) already
used for auth failures, which we reuse for quota-based pausing. A design-pattern brief
already exists at scripts/skills/generated/rate-limit-resilience-patterns.md and its
recommendations are incorporated below.

Minimum viable change: a single shared library (scripts/lib/github-ratelimit.sh)
that (1) proactively reads/caches rate-limit state from headers, (2) implements
exponential backoff with jitter honoring Retry-After, (3) exposes a
gh_rl_should_pause predicate the daemon consults before dispatching, and (4) exposes a
priority-bypass check drawing on reserve quota. Existing scattered gh api calls are
not rewritten wholesale (huge blast radius); instead the daemon dispatch path and the
new module are wired together, and the module offers an opt-in gh_rl_api wrapper that
high-value call sites can adopt incrementally.

---

Files to Modify

Create

• scripts/lib/github-ratelimit.sh — new core module (~320 lines). Header parsing,
  state cache, backoff-with-jitter, pause predicate, priority bypass, metrics.
• scripts/sw-lib-github-ratelimit-test.sh — new test suite (~400 lines) with mock
  gh returning 403 + X-RateLimit-* / Retry-After headers.

Modify

• scripts/lib/daemon-state.sh — delegate the reactive circuit breaker to the new module;
  keep gh_rate_limited/gh_record_failure/gh_record_success as thin back-compat shims
  so existing callers and sw-lib-daemon-state-test.sh keep passing.
• scripts/lib/daemon-poll.sh — before drain/dispatch (near the capacity check at
  ~line 285-334), consult gh_rl_should_pause; set/clear a quota pause reason.
• scripts/lib/daemon-dispatch.sh — in daemon_spawn_pipeline (~line 44), skip spawning
  normal issues while quota-paused unless the issue is a priority op.
• scripts/sw-widgets.sh — add a ratelimit widget (terminal + JSON) reading the cache.
• dashboard/server.ts — surface rate-limit state (remaining, reset, paused) in the
  status payload; add a small frontend card.
• .claude/daemon-config.json (+ init/template defaults in sw-daemon.sh/sw-init.sh) —
  add a rate_limit config block.
• .claude/CLAUDE.md — document config keys, metrics, the new lib and test in the AUTO tables.

✅ Task Checklist

• Task 1: Create github-ratelimit.sh scaffold — VERSION, guards, config accessors, state paths
• Task 2: Implement gh_rl_parse_headers (X-RateLimit-*, Retry-After, missing-header tolerant)
• Task 3: Implement gh_rl_update_cache / gh_rl_get_remaining (atomic write, staleness)
• Task 4: Implement gh_rl_backoff (2^n + jitter, Retry-After override, overflow clamp)
• Task 5: Implement gh_rl_should_pause (threshold predicate, fail-open on unknown)
• Task 6: Implement gh_rl_is_priority (critical_ops + reserve quota)
• Task 7: Implement gh_rl_metric (JSONL + emit_event; 5 metric types)
• Task 8: Implement gh_rl_api wrapper (gh api --include, retry loop, header parse)
• Task 9: Refactor daemon-state.sh to delegate + keep back-compat shims
• Task 10: Wire daemon-poll.sh quota pause/resume (reuse PAUSE_FLAG, reason quota_low)
• Task 11: Wire daemon-dispatch.sh priority bypass on quota pause
• Task 12: Add ratelimit widget to sw-widgets.sh (terminal + JSON)
• Task 13: Add rate-limit status to dashboard/server.ts + frontend card
• Task 14: Add rate_limit config block + update .claude/CLAUDE.md + package.json
• Task 15: Write sw-lib-github-ratelimit-test.sh with mock 403 GitHub responses
• scripts/lib/github-ratelimit.sh created; passes bash -n, set -euo pipefail,
• X-RateLimit-* + Retry-After parsed and cached on API calls via gh_rl_api.
• Exponential backoff (2^n + jitter, backoff_max clamp) honors Retry-After.
• Daemon pauses new dispatch when remaining < threshold (default 100); resumes on reset.
• Critical ops (webhook, deployment, security-alert) bypass via reserve quota.

---

Generated by shipwright pipeline at 2026-07-24T09:43:10Z

[github-actions]

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (125 lines)

The plan's references check out against the codebase (VERSION 3.3.0, the reactive circuit breaker at daemon-state.sh:99-130, the PAUSE_FLAG JSON-with-reason convention already used by disk_low, and the _smart_int/emit_event guards). One useful discovery: PAUSE_FLAG already carries {"paused":true,"reason":...,"resume_after":...} and daemon_poll_issues auto-resumes on resume_after — the design should reuse that rather than invent new resume logic.

Here is the ADR.

---

Design: ** GitHub API Rate Limit Resilience with Intelligent Backoff and Queue Pausing

Context

Shipwright's daemon processes GitHub issues by making gh api calls scattered across sw-github-graphql.sh, sw-github-checks.sh, sw-github-deploy.sh, and dozens of ad-hoc sites. The only rate-limit defense today is a reactive, failure-count circuit breaker in scripts/lib/daemon-state.sh:99-130 (gh_rate_limited/gh_record_failure/gh_record_success): it counts consecutive gh failures and backs off 30s→300s. It is blind to X-RateLimit-* headers, so it only reacts after calls fail, wastes quota discovering the wall, and lives solely in the daemon loop.

Constraints from the codebase that shape the design:

• Bash 3.2 only — no declare -A, readarray, ${var,,}/${var^^}. Integer-only arithmetic for backoff.
• set -euo pipefail everywhere; atomic writes via mktemp + mv; jq --arg for JSON (never interpolation).
• PAUSE_FLAG already exists as ${DAEMON_DIR}/daemon-pause.flag and already stores JSON with a reason field ({"paused":true,"reason":"disk_low"}) and an optional resume_after ISO timestamp that daemon_poll_issues (daemon-poll-github.sh:31-45) auto-clears once elapsed. We must reuse this, not add a competing pause source, and must not clobber an auth/disk_low pause.
• Config chaining via _smart_int (env SW_* → daemon-config.json → default) is the established pattern (compat.sh:337).
• Source guards for emit_event/now_epoch/now_iso are idiomatic (sw-github-graphql.sh:31-40).
• Rewriting all ~40 gh api sites is a P0-inappropriate blast radius against a large existing GitHub test surface.

Decision

Ship a single new shared library scripts/lib/github-ratelimit.sh (VERSION="3.3.0") that is proactive (reads headers), additive (opt-in wrapper), and fail-open (never pauses on unknown state). Wire it into the daemon's dispatch decision path — where pausing actually matters — and leave existing call sites untouched for incremental adoption.

Layering (dependencies point inward, state files are the only shared surface):

daemon-poll.sh / daemon-dispatch.sh   (consumers: decide)
        │ gh_rl_should_pause / gh_rl_is_priority
        ▼
github-ratelimit.sh                   (core: parse, cache, backoff, predicates)
        │ atomic writes
        ▼
~/.shipwright/github-ratelimit.json   (state cache)
~/.shipwright/ratelimit-events.jsonl  (metrics)
        ▲ read-only
sw-widgets.sh / dashboard/server.ts   (presentation)

Core function contracts (bash; space-separated stdout, exit codes are the API):

Function	Contract
gh_rl_parse_headers <hdr_text>	echoes limit remaining reset used retry_after; missing headers → empty fields, always exit 0
gh_rl_update_cache <limit> <rem> <reset> [resource]	atomic mktemp+mv of {limit,remaining,reset,resource,updated_at} via jq -n --arg
gh_rl_get_remaining [resource]	echoes cached remaining; -1 if file missing, corrupt, or reset elapsed (stale)
gh_rl_should_pause	exit 0 (pause) iff 0 <= remaining < threshold AND reset not passed; exit 1 otherwise (-1 unknown → fail-open, no pause); reason on stderr
gh_rl_is_priority <op>	exit 0 iff op ∈ critical_ops AND remaining >= (threshold - priority_reserve_quota)
gh_rl_backoff <attempt> [retry_after]	echoes min(base^attempt + jitter, backoff_max), raised to retry_after when larger; $RANDOM jitter 0..delay/10; exponent clamped to avoid overflow (mirrors daemon-state.sh:122)
gh_rl_api <gh-api-args...>	gh api --include "$@", split header/body at first blank line, parse+cache, retry 403 rate-limit up to circuit_breaker; body on stdout
gh_rl_metric <event> <k=v>...	append JSONL + emit_event "github.ratelimit.<event>"

Data flow. gh_rl_api runs gh api --include, splits the header block from the body at the first blank line, feeds headers to gh_rl_parse_headers → gh_rl_update_cache. On 403 + remaining==0/Retry-After it computes gh_rl_backoff, emits rate_limit_hit, sleeps, retries. The daemon poll calls gh_rl_should_pause (pure cache read — no API call), and on true writes PAUSE_FLAG as {"paused":true,"reason":"quota_low","resume_after":"<reset ISO>"}, letting the existing resume_after machinery auto-resume at reset. Dispatch consults gh_rl_is_priority to let critical_ops bypass the pause using reserve quota.

Error handling / boundaries.

• Missing/malformed headers or corrupt cache → -1 unknown → fail-open (daemon pauses only on known low quota, never on absence of data). This is the single most important failure-mode decision: a false 0 would freeze the daemon permanently.
• gh absent or NO_GITHUB=true / SW_RATE_LIMIT_ENABLED=false → every function is an early-return no-op.
• Pause-reason guard: the resume/clear path acts only when the existing reason == quota_low; it never stomps auth/disk_low pauses.
• Backoff capped at backoff_max so the poll loop never blocks unbounded.

Backpressure state machine: NORMAL → (remaining < threshold) → PAUSED(quota_low, resume_after=reset) → (reset elapsed) → NORMAL. In-flight pipelines finish; only new non-priority spawns are gated.

---

Generated by shipwright pipeline design stage at 2026-07-24T09:46:10Z

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]