2.9.14 (1/2) — fleet issues + the QA gate that failed open - #1022
Open
emooreatx wants to merge 3 commits into
Open
2.9.14 (1/2) — fleet issues + the QA gate that failed open#1022emooreatx wants to merge 3 commits into
emooreatx wants to merge 3 commits into
Conversation
…935) Whether a class implements is_healthy() is STATIC — it cannot change between polls — but this runs on every /v1/system/health check. On datum running 2.9.13 that was 292 identical lines for `tool` and 146 for `wise_authority` in a 31-minute window, at ~2,900 lines/hour overall: 2.6x the rate of the 2.7.6 flood that 2.4.3 and 2.9.6 fixed. The incident log still rotated faster than the soak it exists to document. Same defect as the AUTH_STEP_INFO flood, in a new place: a constant, correct observation emitted at WARNING on a hot path. Silence is NOT the fix — it is a genuine registration defect and the operator must see it. So it reports ONCE per (service_type, provider class), and the one report carries the RCA: which class, which service, why it can never pass a health check, that this is a registration bug rather than a runtime state, and what to change. Measured: 950 warnings -> 19 across 50 polls (98% reduction); the 19 are one per service type, which is the real defect count. The #943 property is preserved and tested: an unknown provider stays in `available` and never in `healthy`, so it drags the ratio toward degraded rather than padding a quiet 100%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018umjfoyNwpa7BWVmzoayTM
…report once (#935) PRICING_DATA.json carried 10 models across 4 providers and NONE of the models in the live matrix, so every cost calculation on a production agent missed and fell through to a flat global default — 64 identical warnings in a 31-minute window on datum. Adds the live matrix (CLAUDE.md) and the three providers that had no entry at all: google/gemma-4-31B-it and meta-llama/Llama-4-Scout-17B-16E-Instruct on Together, meta-llama/llama-4-scout on OpenRouter, meta-llama/llama-4-scout-17b-16e-instruct on Groq, Qwen/Qwen3.6-35B-A3B on DeepInfra. 4 providers -> 7, 10 models -> 15. PER-PROVIDER DEFAULTS. A single global fallback is wrong in both directions: it understated an unknown Anthropic model by 75x on output (20 vs 1500) and overstates a budget one. The model name may be new; the provider is always known, and it tells us the price tier. `get_fallback_pricing(provider)` now prefers the provider default and falls back to the global only when the provider is unknown too. Units verified before touching anything: values are CENTS per million tokens despite `currency: USD` — confirmed as exactly 100x list price against gpt-4o-mini, gpt-4o, claude-3-opus and claude-3-haiku, a 100x price range. Recorded in metadata.units_note so the next person does not have to re-derive it. Getting this backwards would have mispriced everything by 100x. The warning now reports once per (provider, model) and carries the RCA: that derived costs are ESTIMATES, which file to edit, that values are in cents, and that the gap will not resolve on its own. The added prices are APPROXIMATE list prices as of 2026-08-09 and are marked as such in every description — they need verification against provider catalogs before anyone bills on them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018umjfoyNwpa7BWVmzoayTM
`_has_incidents_occurred` returned False when the incidents log was absent —
no log, therefore no incidents, therefore PASS. The gate was a coin flip on
file existence, and it was observed twice on the SAME commit:
2.9.13 main run : sqlite found the log -> FAILED
postgres "NO INCIDENTS LOG FOUND" -> PASSED
re-run : postgres found the log -> FAILED
sqlite "NO INCIDENTS LOG FOUND" -> PASSED
The legs swapped. Both runs had 100% test success on BOTH backends, with
identical expected test-induced errors (invalid-state transitions, adapter
loads without credentials, SIGTERM teardown). Which leg failed depended only
on which leg's file happened to exist.
The flaky red is not the dangerous half. A leg that passes because the log is
absent certifies a run nobody checked, and there is no way to tell after the
fact which historical greens were real. This is the same defect class as the
71-hour trace stall and the ignored template: an instrument reporting success
for work never performed.
Absent log now fails, and reports CANNOT CERTIFY with the expected path, the
backend, why a pass would be meaningless, and the remedy. It is flagged
separately from "incidents were found" so the summary does not claim
detection that did not happen — the two need different debugging.
A present-but-unchanged log is still a genuine pass; fail-closed must not turn
the steady state red, and a test pins that.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018umjfoyNwpa7BWVmzoayTM
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2.9.14, part 1 — fleet issues from the 2.9.13 upgrade
Three fixes from the manager team's fleet review, plus the QA harness race that made CI itself untrustworthy. The wizard redesign lands in a companion PR; the 2.9.14 version bump rides with that one so two branches don't both bump.
1. The incidents gate failed open — CI has been passing partly by luck
Observed twice on the same commit:
NO INCIDENTS LOG FOUND→ passNO INCIDENTS LOG FOUND→ passThe legs swapped. Both runs: 100% test success on both backends, identical expected test-induced errors (invalid-state transitions, adapter loads without credentials, SIGTERM teardown). Which leg failed depended only on which leg's file existed.
The flaky red isn't the dangerous half. A leg that passes because the log is absent certifies a run nobody checked, and there's no way to tell after the fact which historical greens were real. Same class as the 71-hour trace stall and the silently-ignored template.
Now fails closed, reports
CANNOT CERTIFYwith the expected path, the backend, why a pass would be meaningless, and the remedy — flagged separately from "incidents were found", because the two need different debugging. A present-but-unchanged log is still a genuine pass, and a test pins that so fail-closed doesn't turn the steady state red.2.
#935— provider health noise (measured)exposes no is_healthy()re-reported a static fact on every health poll: 292 lines fortooland 146 forwise_authorityin a 31-minute window on datum, ~2,900/hour overall — 2.6× the 2.7.6 flood that 2.4.3 and 2.9.6 fixed.950 → 19 warnings across 50 polls (98% reduction), the 19 being one per service type — the true defect count. Silence isn't the fix: it's a genuine registration defect, so it reports once per
(service_type, provider class)and the report carries the RCA — which class, why it can never pass a health check, that it's a registration bug rather than a runtime state, and what to change. The#943property (unknown never counts as healthy) is preserved and tested.3.
#935— pricing had zero coverage of what we actually runPRICING_DATA.jsoncarried 10 models across 4 providers and none of the live matrix, so every production cost calculation missed. Now 15 models across 7 providers, adding OpenRouter, Groq and DeepInfra which had no entry at all.Per-provider defaults. One global fallback was wrong in both directions — it understated an unknown Anthropic model by 75× on output (20 vs 1500 cents/M). The model name may be new; the provider is always known and tells us the tier.
Units were verified before anything was touched: values are cents per million tokens despite
currency: USD— confirmed as exactly 100× list price against four models spanning a 100× range, and recorded inmetadata.units_note. Getting that backwards would have mispriced everything by 100×.Also verified, no change needed
#935's original two causes are already fixed —AUTH_STEP_INFO→ debug in 2.4.3, and thekeyattribute in v2.9.6 (2026-06-12). The fleet saw them on processes with 7–11 weeks uptime, started ~2026-06-11. The manager team's controlled comparison confirms it: 2.9.13 datum shows 0 of both; the 2.7.6 controls show 2,696 and 3,600.#1021is substrate-side — the anti-Goodhart guard fires because on a cohabited node the scorer's own key is the agent's key. Working as designed; the defect is that it's attempted at all and logged per-cycle. Belongs inCIRISServer/src/scorer.rs.Tests
216tools tests,148API route tests,66pricing/cost tests — all green.🤖 Generated with Claude Code
https://claude.ai/code/session_018umjfoyNwpa7BWVmzoayTM