Skip to content

0.3.68: resolve the whole config in ONE corpus read, not fifty - #79

Merged
emooreatx merged 1 commit into
mainfrom
fix/config-resolve-one-fetch
Sep 8, 2026
Merged

0.3.68: resolve the whole config in ONE corpus read, not fifty#79
emooreatx merged 1 commit into
mainfrom
fix/config-resolve-one-fetch

Conversation

@emooreatx

Copy link
Copy Markdown
Contributor

Investigating "ciris.ai/status/ is not loading". The page and the API both answer 200, and from the box the API answers in 1.2 ms — but one request in eight took 1.1–2.0 seconds, which through Cloudflare is what a first load feels like.

Where the stall is

Correlating per-second CPU against per-second requests on the live node:

 sec  cpu%_of_core   request_ms
   7        33         1.125     <-- slow
  18        91         0.0087
  19       102         0.0073    <-- 100% CPU, still 7ms
  25       101         0.0023
  30        41         2.023     <-- slow
  35        43         0.926     <-- slow

The slow requests don't land at peak CPU — the 100%-CPU seconds served in 1–8 ms. They land at 33–43%, which is the signature of workers blocked in synchronous SQLite reads rather than starved of CPU: four workers sitting in pread, nothing free to pick up the request.

What they're reading is our own config

graph_config::get_i64(engine, key) reads as a keyed lookup and is a full scan. get_config calls live_config_rows, which lists every attestation this node authored and JSON-parses each envelope to find the dozen whose dimension starts with config:. Per key. No cache.

Config::resolve reads ~50 keys — scalars, per-region URLs, per-provider settings, per-capability members, per-auth targets. So every poll cycle ran ~50 full scans of everything this node has ever signed:

strace -c -f, 25s:   334,399 pread64, all on ciris_engine.db
per-second CPU:      ~20s of each 60s cycle pinned at 100% of a core
corpus state:        trace_events 0 rows, roster empty

And it degrades with every observation we emit, because those rows share the attester key the scan seeks on.

list_configs runs that same scan once and returns the lot. One read per resolve instead of fifty.

The API shape is CIRISServer#557, filed with these measurements. This is the consumer-side fix that doesn't wait for it.

One contract nearly lost in the batching

resolve_auth_targets read the raw value and tested is_empty below — that's what makes status.auth.<id>.url = "" disable that probe, which is documented behaviour. The batched accessor filters empty to None, which would have fallen through to the baked endpoint and turned the probe back on: the inverse of what the operator asked for, and silent.

Snapshot::str_raw keeps empty meaningful where empty means something, and an_empty_auth_url_disables_that_probe fails if that ever inverts again.

115 tests (1 new), fmt, clippy -D warnings.

🤖 Generated with Claude Code

`https://ciris.ai/status/` was slow to load. The page and the API both answer
200, and from the box the API answers in 1.2ms — but one request in eight took
1.1-2.0 SECONDS. Correlating per-second CPU against per-second requests, the
slow ones do not land at peak CPU (100% seconds served in 1-8ms); they land at
33-43%, which is the signature of workers blocked in synchronous SQLite reads
rather than starved of CPU. Four workers in `pread`, nothing free to accept the
request.

What they are reading is our own config. `graph_config::get_i64(engine, key)`
reads as a keyed lookup and is a full scan: `get_config` calls
`live_config_rows`, which lists every attestation this node authored and
JSON-parses each envelope for the dozen whose dimension starts with `config:`.
Per key. No cache.

`Config::resolve` reads ~50 keys — scalars, per-region URLs, per-provider
settings, per-capability members, per-auth targets — so every poll cycle ran ~50
full scans of everything this node has ever signed. Measured: 334,399 `pread64`s
in 25s, all on `ciris_engine.db`, and ~20s of each 60s cycle pinned at 100% of a
core, on a node with no traces and an empty roster. It also degraded with every
observation emitted, since those rows share the attester key the scan seeks on.

`list_configs` runs that scan once and returns the lot. One read per resolve.

The API shape is CIRISServer#557, filed with these measurements; this is the
consumer-side fix that does not wait for it.

**One contract nearly lost in the batching, now pinned by a test.**
`resolve_auth_targets` read the RAW value and tested `is_empty` below, which is
what makes `status.auth.<id>.url = ""` disable that probe — documented
behaviour. The batched accessor filters empty to `None`, which would have fallen
through to the baked endpoint and turned the probe back ON: the inverse of what
the operator asked for, silently. `Snapshot::str_raw` keeps empty meaningful
where empty means something, and `an_empty_auth_url_disables_that_probe` fails
if that inverts again.

115 tests (1 new), fmt, clippy -D warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@emooreatx
emooreatx merged commit ba62979 into main Sep 8, 2026
3 checks passed
@emooreatx
emooreatx deleted the fix/config-resolve-one-fetch branch September 8, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant