feat(sqlite): split queue-wait vs service metrics, slow-op log, configurable read workers

[arielmelendez]

Motivation

On the canary gateways, the core StandaloneSqlite worker intermittently backs
up under concurrent retrieval load — the SQLite Core Worker Latency by Method
panel shows every core-worker method spiking to tens of seconds at once. That
panel is derived from standalone_sqlite_method_duration_seconds, which times
enqueue → completion, so it conflates three very different things: time spent
waiting in the queue, worker execution, and main-event-loop reply scheduling.
During a jam every method inflates to the queue depth, so the panel can't tell
you which query is expensive or where the time actually goes.

Investigation found the individual queries are all indexed and fast (single-digit
ms in isolation), which points at saturation — a single core read worker
(core.read = 1) serializing request volume, plus main-event-loop latency on the
worker message round-trip — rather than a slow query. This PR adds the
instrumentation needed to confirm that and the knob to relieve it.

Changes

Split timing metrics (metrics.ts, standalone-sqlite.ts)

• standalone_sqlite_method_queue_wait_seconds — enqueue → dispatch
• standalone_sqlite_method_service_seconds — dispatch → reply received
• sqlite_queued_ops gauge — enqueued-but-not-yet-dispatched depth per worker/role

Reading them: high queue_wait → worker-pool serialization (raise the read
pool); high service with fast underlying queries → main-event-loop
saturation (more read workers won't help). The existing
standalone_sqlite_method_duration_seconds summary is retained for dashboard
continuity.

Slow-op log

• When queue_wait + service ≥ SQLITE_SLOW_QUERY_LOG_THRESHOLD_MS (default 1000),
  a warn reports method, the queue/service breakdown, current queue depth,
  and a bounded arg summary (summarizeSqliteArgs — scalars pass through, long
  strings/blobs become <bytes:N>/<string:N>, containers collapse; no buffers
  or row payloads reach the logs).

Configurable reader pools

• CORE_SQLITE_READ_WORKER_COUNT (default 1) and DATA_SQLITE_READ_WORKER_COUNT
  (default 2) make the reader pool sizes tunable; WAL supports concurrent
  readers. Writers stay single — SQLite permits only one writer, and writes
  are not the bottleneck.

Defaults preserve current behavior exactly. docs/envs.md and the
docker-compose passthrough are updated.

Testing

• tsc --noEmit: clean
• eslint src test + prettier: clean
• src/database/standalone-sqlite.test.ts: 68/68 pass (queue mechanics intact)

Follow-ups (not in this PR)

• Deadline-drop-at-dispatch load shedding (bound queue latency by dropping work
  whose caller already timed out).
• Worker-internal execution timing, if service proves ambiguous vs main-loop
  scheduling in practice.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 77.28337% with 97 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.75%. Comparing base (fc77b92) to head (4826305).

Files with missing lines	Patch %	Lines
src/config.ts	76.30%	41 Missing ⚠️
src/data/gateways-data-source.ts	59.57%	38 Missing ⚠️
src/database/standalone-sqlite.ts	82.52%	17 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #817      +/-   ##
===========================================
- Coverage    78.77%   78.75%   -0.02%     
===========================================
  Files          133      133              
  Lines        50477    50880     +403     
  Branches      3815     3835      +20     
===========================================
+ Hits         39761    40072     +311     
- Misses       10666    10757      +91     
- Partials        50       51       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds configurable SQLite worker counts and gateway socket timeout/logging settings, introduces new SQLite and gateway timing metrics, and instruments standalone SQLite workers and gateway agents with queue and socket latency logging.

Changes

SQLite worker and gateway observability

Layer / File(s)	Summary
Environment settings and config exports src/config.ts, docker-compose.yaml, docs/envs.md, src/config.test.ts	Adds gateway socket timing and per-host socket-cap config, plus SQLite worker-count and slow-query settings, with matching compose, docs, and config tests.
SQLite and gateway metrics src/metrics.ts	Adds the SQLite queued-ops gauge, split queue-wait/service histograms, and gateway socket acquisition/connect histograms.
Standalone SQLite worker timing src/database/standalone-sqlite.ts	Updates worker pool sizing, stamps queued jobs with timing data, records queue-wait and service durations, and logs slow SQLite operations with summarized arguments.
Gateway agent socket instrumentation src/data/gateways-data-source.ts	Uses the configured idle-socket timeout for gateway agents and instruments request handling to measure socket acquisition and connect timing, emit slow-acquisition warnings, and distinguish reused sockets from newly connected sockets.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• ar-io/ar-io-node#219: Also changes src/database/standalone-sqlite.ts around worker/job dispatch behavior.
• ar-io/ar-io-node#220: Also extends src/metrics.ts with SQLite operation metrics.
• ar-io/ar-io-node#720: Also changes src/data/gateways-data-source.ts agent creation and reuse.

Suggested reviewers: djwhitt

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main SQLite observability and tuning changes in the PR.
Description check	✅ Passed	The description is directly related to the changeset and accurately describes the added metrics, logging, and config knobs.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/sqlite-worker-instrumentation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/database/standalone-sqlite.ts (1)

3391-3415: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

sqliteQueuedOps gauge can go negative — dec() fires for jobs never inc()'d.

takeWork() unconditionally decrements sqliteQueuedOps for every dispatched job. But stop() (further down) pushes a { method: 'terminate' } job directly onto workQueues[pool][role] without ever calling metrics.sqliteQueuedOps.inc() (only queueWork() does that, at the queueWork push site). When that terminate job is dispatched, the gauge is decremented without a matching increment, so it can dip negative during shutdown — exactly when accurate metrics are most useful for diagnosing a stuck drain.

🔧 Proposed fix: only decrement for tracked (enqueued-via-queueWork) jobs

           job = self.workQueues[pool][role].shift();

           // Split timing: record how long this op waited in the queue before
           // dispatch, and stamp the dispatch time so the reply handler can
           // measure service time.
           const dispatchedAt = performance.now();
           const method = job.message.method;
-          const queueWaitSeconds =
-            job.enqueuedAt !== undefined
-              ? (dispatchedAt - job.enqueuedAt) / 1000
-              : 0;
+          const wasTracked = job.enqueuedAt !== undefined;
+          const queueWaitSeconds = wasTracked
+            ? (dispatchedAt - job.enqueuedAt) / 1000
+            : 0;
           job.dispatchedAt = dispatchedAt;
           job.queueWaitSeconds = queueWaitSeconds;
-          metrics.sqliteQueuedOps.dec({ worker: pool, role });
-          metrics.sqliteMethodQueueWaitSeconds.observe(
-            { worker: pool, role, method },
-            queueWaitSeconds,
-          );
+          if (wasTracked) {
+            metrics.sqliteQueuedOps.dec({ worker: pool, role });
+            metrics.sqliteMethodQueueWaitSeconds.observe(
+              { worker: pool, role, method },
+              queueWaitSeconds,
+            );
+          }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/database/standalone-sqlite.ts` around lines 3391 - 3415, The `takeWork()`
dispatch path is decrementing `metrics.sqliteQueuedOps` for every job, but some
jobs such as the `terminate` job inserted by `stop()` are never counted by
`queueWork()` and therefore should not be decremented. Update `takeWork()` in
`standalone-sqlite.ts` to only call `sqliteQueuedOps.dec()` for jobs that were
actually tracked as queued, using the job metadata or a similar guard, while
keeping the existing queue-wait observation and dispatch logic for the tracked
jobs.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/database/standalone-sqlite.ts`:
- Around line 3391-3415: The `takeWork()` dispatch path is decrementing
`metrics.sqliteQueuedOps` for every job, but some jobs such as the `terminate`
job inserted by `stop()` are never counted by `queueWork()` and therefore should
not be decremented. Update `takeWork()` in `standalone-sqlite.ts` to only call
`sqliteQueuedOps.dec()` for jobs that were actually tracked as queued, using the
job metadata or a similar guard, while keeping the existing queue-wait
observation and dispatch logic for the tracked jobs.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 0425b299-581d-4583-964b-33e805546588

📥 Commits

Reviewing files that changed from the base of the PR and between fc77b92 and a5b4660.

📒 Files selected for processing (5)

• docker-compose.yaml
• docs/envs.md
• src/config.ts
• src/database/standalone-sqlite.ts
• src/metrics.ts

[arielmelendez]

@coderabbitai Good catch — addressed in ad19989.

You're right: stop() pushes the terminate job directly onto the queue, bypassing queueWork(), so it's never counted into sqliteQueuedOps and carries no enqueuedAt. takeWork() was decrementing unconditionally, which could drive the gauge negative on shutdown.

Fixed by gating the entire split-timing block on job.enqueuedAt !== undefined rather than only the queue-wait computation. That keeps the gauge balanced, and since dispatchedAt is now only stamped for tracked jobs, the reply handler's service-time instrumentation and slow-op log also skip these control jobs automatically — no spurious terminate latency series.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
cd /repo 2>/dev/null || true
git fetch origin ad199893 --depth=2 2>&1 | tail -5
git show ad199893 -- src/database/standalone-sqlite.ts | head -150

Length of output: 3424

---

@arielmelendez Confirmed — I reviewed commit ad19989 and the fix is correct. Gating the entire block (dispatch timestamp, queue-wait computation, gauge decrement, and histogram observation) on job.enqueuedAt !== undefined ensures:

• The terminate control job (and any other job bypassing queueWork()) never triggers sqliteQueuedOps.dec() without a matching inc(), so the gauge stays balanced across shutdown.
• Since dispatchedAt is left undefined for these jobs, the reply handler's service-time observation and slow-op warning log will naturally skip them too, avoiding a bogus terminate entry in the latency histograms.

Nice catch and clean fix — thanks for addressing it.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/data/gateways-data-source.ts`:
- Around line 175-183: The gateway socket connect histogram in
`gateways-data-source.ts` is measuring only the plain `connect` event, so
HTTPS/TLS sockets miss the handshake time. Update the `isNewSocket` timing logic
to use `secureConnect` for `TLSSocket` instances in the same block where
`metrics.gatewaySocketConnectSeconds.observe` is called, or otherwise adjust the
metric/help text to clearly indicate it only measures TCP connect time. Keep the
fix localized around the socket event hookup in `gateways-data-source.ts`.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 5891df17-8dc1-4cb1-bd07-5556a71f83e1

📥 Commits

Reviewing files that changed from the base of the PR and between ad19989 and 9906d80.

📒 Files selected for processing (5)

• docker-compose.yaml
• docs/envs.md
• src/config.ts
• src/data/gateways-data-source.ts
• src/metrics.ts

✅ Files skipped from review due to trivial changes (1)

• docs/envs.md

🚧 Files skipped from review as they are similar to previous changes (1)

• src/config.ts

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

src/config.ts (1)

359-371: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Dead default value in GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST branch.

The env.varOrUndefined('GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST') !== undefined check already guarantees raw is defined by the time parsePerHostNumber is called, so the defaultValue parameter (16) passed there is unreachable inside parsePerHostNumber. It's harmless, but the duplicated 16 magic number could drift from GATEWAY_MAX_SOCKETS_PER_HOST's actual default if that changes later.

♻️ Simplify by letting parsePerHostNumber return undefined when unset

-export const GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST: PerHostNumber =
-  env.varOrUndefined('GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST') !== undefined
-    ? parsePerHostNumber('GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST', 16)
-    : GATEWAY_MAX_SOCKETS_PER_HOST;
+export const GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST: PerHostNumber =
+  parsePerHostNumberOrUndefined('GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST') ??
+  GATEWAY_MAX_SOCKETS_PER_HOST;

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/config.ts` around lines 359 - 371, The
`GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST` initialization in `src/config.ts`
duplicates the `16` default even though the env presence check already
guarantees a value, so the fallback is dead and can drift from
`GATEWAY_MAX_SOCKETS_PER_HOST`. Simplify the branch by having
`parsePerHostNumber` handle the unset case directly or by reusing
`GATEWAY_MAX_SOCKETS_PER_HOST` as the fallback, and keep the behavior centered
around the `GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST` and `parsePerHostNumber`
symbols.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/config.ts`:
- Around line 284-351: The new exported symbols in config.ts are documented with
plain line comments instead of TSDoc. Convert the comments for PerHostNumber,
parsePerHostNumber, and resolvePerHostNumber to proper TSDoc blocks so they
satisfy the codebase guideline for touched exported APIs, while keeping the
existing descriptions accurate and concise.

---

Nitpick comments:
In `@src/config.ts`:
- Around line 359-371: The `GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST`
initialization in `src/config.ts` duplicates the `16` default even though the
env presence check already guarantees a value, so the fallback is dead and can
drift from `GATEWAY_MAX_SOCKETS_PER_HOST`. Simplify the branch by having
`parsePerHostNumber` handle the unset case directly or by reusing
`GATEWAY_MAX_SOCKETS_PER_HOST` as the fallback, and keep the behavior centered
around the `GATEWAY_UNTRUSTED_MAX_SOCKETS_PER_HOST` and `parsePerHostNumber`
symbols.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: cc89b5d6-4f9f-4ed9-830b-6a491611e354

📥 Commits

Reviewing files that changed from the base of the PR and between 24e8031 and 26dc7d1.

📒 Files selected for processing (5)

• docker-compose.yaml
• docs/envs.md
• src/config.test.ts
• src/config.ts
• src/data/gateways-data-source.ts

✅ Files skipped from review due to trivial changes (1)

• docs/envs.md

🚧 Files skipped from review as they are similar to previous changes (1)

• src/data/gateways-data-source.ts