0.3.68: resolve the whole config in ONE corpus read, not fifty - #79
Merged
Conversation
`https://ciris.ai/status/` was slow to load. The page and the API both answer 200, and from the box the API answers in 1.2ms — but one request in eight took 1.1-2.0 SECONDS. Correlating per-second CPU against per-second requests, the slow ones do not land at peak CPU (100% seconds served in 1-8ms); they land at 33-43%, which is the signature of workers blocked in synchronous SQLite reads rather than starved of CPU. Four workers in `pread`, nothing free to accept the request. What they are reading is our own config. `graph_config::get_i64(engine, key)` reads as a keyed lookup and is a full scan: `get_config` calls `live_config_rows`, which lists every attestation this node authored and JSON-parses each envelope for the dozen whose dimension starts with `config:`. Per key. No cache. `Config::resolve` reads ~50 keys — scalars, per-region URLs, per-provider settings, per-capability members, per-auth targets — so every poll cycle ran ~50 full scans of everything this node has ever signed. Measured: 334,399 `pread64`s in 25s, all on `ciris_engine.db`, and ~20s of each 60s cycle pinned at 100% of a core, on a node with no traces and an empty roster. It also degraded with every observation emitted, since those rows share the attester key the scan seeks on. `list_configs` runs that scan once and returns the lot. One read per resolve. The API shape is CIRISServer#557, filed with these measurements; this is the consumer-side fix that does not wait for it. **One contract nearly lost in the batching, now pinned by a test.** `resolve_auth_targets` read the RAW value and tested `is_empty` below, which is what makes `status.auth.<id>.url = ""` disable that probe — documented behaviour. The batched accessor filters empty to `None`, which would have fallen through to the baked endpoint and turned the probe back ON: the inverse of what the operator asked for, silently. `Snapshot::str_raw` keeps empty meaningful where empty means something, and `an_empty_auth_url_disables_that_probe` fails if that inverts again. 115 tests (1 new), fmt, clippy -D warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Investigating "ciris.ai/status/ is not loading". The page and the API both answer 200, and from the box the API answers in 1.2 ms — but one request in eight took 1.1–2.0 seconds, which through Cloudflare is what a first load feels like.
Where the stall is
Correlating per-second CPU against per-second requests on the live node:
The slow requests don't land at peak CPU — the 100%-CPU seconds served in 1–8 ms. They land at 33–43%, which is the signature of workers blocked in synchronous SQLite reads rather than starved of CPU: four workers sitting in
pread, nothing free to pick up the request.What they're reading is our own config
graph_config::get_i64(engine, key)reads as a keyed lookup and is a full scan.get_configcallslive_config_rows, which lists every attestation this node authored and JSON-parses each envelope to find the dozen whose dimension starts withconfig:. Per key. No cache.Config::resolvereads ~50 keys — scalars, per-region URLs, per-provider settings, per-capability members, per-auth targets. So every poll cycle ran ~50 full scans of everything this node has ever signed:And it degrades with every observation we emit, because those rows share the attester key the scan seeks on.
list_configsruns that same scan once and returns the lot. One read per resolve instead of fifty.The API shape is CIRISServer#557, filed with these measurements. This is the consumer-side fix that doesn't wait for it.
One contract nearly lost in the batching
resolve_auth_targetsread the raw value and testedis_emptybelow — that's what makesstatus.auth.<id>.url = ""disable that probe, which is documented behaviour. The batched accessor filters empty toNone, which would have fallen through to the baked endpoint and turned the probe back on: the inverse of what the operator asked for, and silent.Snapshot::str_rawkeeps empty meaningful where empty means something, andan_empty_auth_url_disables_that_probefails if that ever inverts again.115 tests (1 new), fmt, clippy
-D warnings.🤖 Generated with Claude Code