fix(loop): auto-retry acp-loop replay on transient upstream rejections (#189)

[ranxianglei]

Problem

GitHub #189: on GLM Coding Plan (zcode.z.ai), acp-loop replay requests get upstream 400 {"code":3007,"msg":"captcha verify failed"} within ~300ms. Old code passed the error straight into the agent session as [acp-proxy: compress loop upstream error 400: ...].

What the log actually proves (fact vs hypothesis)

Facts from the reporter's log (5 events in one day):

• ALL 5 failures are acp-loop re-requests (bili-initiated replays). Zero direct-forward failures.
• Each 400 lands 200–300ms after re-requesting so the model sees the result — <1s after the previous response completed, no human delay.
• The next human-paced turn always succeeds (no real captcha ever served).
• Notably, 2 of 5 events are NOT immediately after a compress — they are re-requests inside ordinary turns (read-only proxy tool) with STABLE context size between the failed pair (input=57578 / input=26925). In both cases the preceding forward with the SAME (already-shrunk) context succeeded. So "context shrinkage" alone explains only 3/5.

Candidate triggers (UNCONFIRMED — needs a controlled experiment):

• A. inter-request interval <1s without human delay (fits all 5 — strongest)
• B. context shrinkage / prefix-cache rewrite (3/5 only)
• C. replay-body content specifics (developer visibility-marker messages etc.)
• D. TLS connection churn (mitm tunnel re-established adjacent to several events)

3007 is NOT in z.ai's public error-code reference (docs.z.ai/api-reference/api-code documents 1xxx–2xxx business codes; no 3xxx) — it's an internal ZCode-plan gateway code, so no authoritative public definition exists.

Fix

Bounded retry with exponential backoff on the REPLAY request only (both the streaming loop src/loop/core.ts and the Responses-API JSON loop src/compress-loop-responses.ts):

• Up to 3 total attempts (BILI_REPLAY_RETRY_MAX; set 1 to restore the legacy fail-fast behavior exactly), backoff base 1500ms doubling per attempt (BILI_REPLAY_RETRY_BASE_MS; 0 = no delay).
• Transient = HTTP 429/5xx, or any other 4xx whose body matches risk-control markers (captcha, verify failed, risk control, 风控, rate limit, too many requests, try again). Plain 4xx fails fast. Network-level failures are NOT retried (avoids stacking the 10-min timeout).
• Clear per-retry log line: upstream rejected replay (HTTP 400 ...); likely provider risk-control — retrying in 1500ms (attempt 1/3); final failure message carries after N attempt(s).

Worst case if the hypotheses are wrong: the same error surfaces ≤4.5s later. Replay calls are idempotent LLM requests; no state mutation. BILI_REPLAY_RETRY_MAX=1 restores old behavior exactly.

NOT doing (yet): smoothing the post-compress transition (split big compress / overlap tail) — larger semantic change; revisit if failures persist despite retries.

Path to hard evidence (controlled experiment, needs a GLM Coding Plan account)

Pure curl against zcode.z.ai/api/v1/zcode-plan/anthropic/v1/messages, no bili involved:

1. Baseline: two medium (~20K-token) requests 3s apart → expect both 200.
2. Timing: large request (~100K+), then IMMEDIATELY (<1s) another large request → 3007? (tests A)
3. Shrinkage: large request, wait 30s, then small request (~5K, different prefix) → 3007? (tests B)
4. Transience: on any 3007, resend the EXACT same body after 1.5s, then 3s → record statuses. A 200 on the second try validates the retry approach.
5. Content: run bili with ACP_DUMP_REQ=1 + debug → the failing replay body is dumped to ~/.local/state/billion-context/dumps/req-*-REREQUEST.json; resend that file standalone via curl after 30s → 3007 again ⇒ body content is the trigger (tests C).

Tests / pre-flight

8 new regression tests (retry-then-success, persistent-captcha exhaustion, fatal-400 fail-fast, 429 retry, classifier matrix, backoff math, max-attempts env, MAX=1 legacy mode) + S3 "upstream 500 mid-loop" test updated for the new behavior. typecheck ✓ · 520/520 tests ✓ · build ✓. CHANGELOG + CONFIGURATION.md updated.