ES-4034 botmonAlpha: workflow edge times, trace icons, rogue detection, chart checkpoint line, humanized durations

[Detroitiron]

Co-authored by Claude on 2026-08-27.

Summary

Six botmonAlpha defects found while dogfooding the new UI against legacy botmon. Legacy is the behavioral reference throughout; it is not changed here.

Three of the six looked cosmetic but were data-layer bugs:

• The workflow view's "0 / N/A" on every edge was a stats-merge crash that 500'd the stats API, plus a link-key direction mismatch that made the downstream half of every graph unmatchable.
• Rogue and blocked were structurally unreachable: the status code summed an errors field on read/write stats that the Leo stats table never writes, so window errors were always 0, the error-rate alarm never fired, and the catalog's Errors column read 0 for every bot.
• A rogue bot showed rogue in the tree but healthy on its own dashboard, because the header read only the shared catalog, which is fetched fire-and-forget (10s+ on a large bus) and keyed by bare ids while page ids can carry a bot: prefix.

The rest are presentation: trace icons built from the wrong asset prefix, a red chart line marking wall-clock "now" instead of the read checkpoint, and raw milliseconds on duration charts.

Tier

Tier: 2

Jira

• Brief: https://chb.atlassian.net/browse/ES-4033
• Codework: https://chb.atlassian.net/browse/ES-4034

Scope

One concern: botmonAlpha/legacy-botmon parity for the six defects reported on ES-4033. All six are display-correctness bugs in the same UI reported together during one dogfooding pass, and three share a root cause in how bot status reads stats — splitting them would separate a fix from its own regression test. No schema changes, no refactors bundled, no backend/pipeline changes.

What changed

1 · Workflow edge times (the N/A defect)

• mergeStatsResults threw Object.entries(undefined) on any record lacking read or write (write-only bots, queue records), which 500'd /api/workflow/stats and blanked every edge label. Guarded with ?? {}, matching legacy's || {}.
• New queue entries were assigned the same shared default object, aliasing every queue of a bot to one record; now cloned per queue.
• Link-stats lookups are direction-aware: the stats are keyed downstream-first, but the children tree looked them up upstream-first, so right-side edges could never match. Both lookup sites — the edge labels and the relationship-importance scoring — now build keys through one shared linkStatsKey() helper.
• The zero-event fallback indexed checkpoint maps with prefix-stripped ids; they are keyed by full refId (queue:foo). Quiet edges now show the checkpoint time like legacy instead of N/A.
• fetchBotStats checks res.ok and response shape instead of merging undefined and throwing inside an untracked async.

2 · Event-trace icons — trace-fanout-tree built PNG URLs from bare base. Static files are served from CloudFront via assets; base routes through API Gateway into the SvelteKit Lambda, which has no static routes → 404 → broken-image placeholders. Now uses the app-wide assets || base prefix that every other icon consumer already uses. Invisible in dev, where both prefixes are empty.

3 · Rogue / blocked detection

• Error and execution counts now read stats.execution — the only place the bus records errors.
• The persisted errorCount that drives rogue exists in two independent copies: the shared bot catalog, and the page's own settings record. Refresh timers now update both, so the count is no longer frozen at page-load state (legacy re-scanned the cron table every 10s). Refreshing only the catalog cannot reach the dashboard header, which prefers the page record.
• The relationship tree rebuilds after status evaluation instead of snapshotting status by value forever.

4 · Chart line — legacy draws a red read-cutoff line only on the one chart that can lag, at the read checkpoint. The new charts had inverted this: a wall-clock line on every chart, with the checkpoint line only where a prop happened to be passed. Removed the "now" line from the bucket, line, and sparkline components. Where a checkpoint exists — Events In Queue and the read-side sparklines — the checkpoint line is now the only line, and a || 0 call site that would otherwise pin it at epoch 1970 is guarded. GenericLineChart gets no line at all, deliberately: it backs execution count, error count, execution time, events written and write lag, none of which can lag behind a read checkpoint, and legacy draws nothing on them either.

5 · Duration formatting — added dataIsTimeBased to GenericBucketLineChart (mirroring the prop GenericLineChart already had) so Execution Time and Read Lag axis ticks and tooltips go through humanize(), matching their already-humanized summary totals.

6 · Rogue visibility — the dashboard header derives rogue from the page's own strongly-consistent settings record (the same source the Paused badge already used), falling back to the catalog, with prefix-tolerant id matching. Shown as a solid red badge with the error count and a ring on the bot avatar. In the tree, error-state nodes get a tinted fill and heavier stroke — a rogue node was previously a dark shape with a dark-red outline on a dark canvas, reading as quieter than the healthy green rings around it.

Supporting

• Badge logic extracted to resolveHeaderBotStatus / isSameBotId so it is unit-testable rather than living untestable in markup.
• Removed the phantom ReadWriteStats.errors field from the type and from mock-bots fixtures, which had been encoding the wrong record shape and validating the bug.
• AGENTS.md gains the six footguns this work surfaced (assets-vs-base, where errors live, absent read/write keys, refId-keyed checkpoints, rogue's count living in two places that both need re-fetching, and the downstream-first link-key convention).
• webapp bumped to 4.3.4.

Review round 2 (webapp 4.3.4)

A second review pass found three defects, two of which meant a fix in this PR never reached production. All three are fixed above; the details are in decision-log AD-008 through AD-011.

• The header rogue badge was still frozen at page load. The 4.3.1 fix refreshed the shared catalog, but the header prefers the page's own settings record, and a defined 0 from page load permanently shadowed the fresher catalog value. Worse, the unit test asserting "settings is fresher than the catalog" encoded a premise the runtime inverted — a green test over a condition the app never satisfied. Fixed by making the premise true: the refresh timer now re-reads the settings record.
• checkPointValue on GenericLineChart was unreachable. The wrapper never forwarded it and no caller passed one, so removing the wall-clock line left those charts with nothing rather than a checkpoint line. Per the decision above that is the intended behavior, so the dead prop is gone and the component states why instead of implying a line that could never render.
• The link-key direction fix was half-applied. calculateRelationshipImportance still built its key upstream-first in both directions, so every lookup missed once the edge-label fix landed: event counts fell to 0 and now - undefined made every importance score NaN, which emptied the relationship list whenever "include inactive" was off. The line predates this PR, but it is the same bug this PR claims to fix, so it is fixed here — through the shared helper, since two hand-built copies of the convention are what allowed the drift.

Two smaller items came along: the error-rate alarm compared a rate that is forced to 0 at zero executions, so a bot with errors and no successful runs never alarmed where legacy would — it now compares counts the way legacy does. And the lockfile's 108 peer: true flips are reverted; a plain npm install rewrites them, so they were version-dependent noise rather than a dependency change.

The review's remaining findings are deferred to follow-ups under ES-4142 rather than widening an open PR: the blocked status now being reachable with broader semantics than legacy's (a product call, not a code fix), the catalog Errors column mixing windowed and persisted counts, the cost of /api/workflow/relationships sitting on two timers, the tree timer's every-other-tick cadence, buildRelationShipTree() running where no tree is mounted, and mergeStatsResults' min_duration seed of 0.

Tests / Validation (Ran / Not Run)

Env / Path	Ran?	Evidence or testable rationale
Unit + component tests	✅	npm test — 192 passed (19 files), node + Storybook chromium projects
New regression tests	✅	Merge guards for absent read/write, per-queue aliasing, execution-sourced error counts, the zero-execution error alarm, and header rogue resolution (catalog-not-loaded, stale-catalog, threshold boundary, bot: prefix)
Link-key regression test	✅	link-stats-key.test.ts builds the map with the real writer and reads it back with the real key builder. Verified RED against the old key order: 7 of its 9 cases fail
Source-guard tests	✅	Fail if any chart reintroduces x={now}, any component builds a PNG URL from bare base, or the dashboard timer drops its settings re-read. These are literal source matches — deletion tripwires, not behavioral proofs; a rename or an equivalent rewrite walks past them
Type check	✅	svelte-check 38 errors vs 43 on origin/master (measured in a clean baseline worktree); net −5, unchanged by the 4.3.4 fixes, none in touched code
Prod data verification	✅	Read-only DynamoDB: rcs-service-prod-satori-attempt-emitter has LeoCron errorCount 29 vs threshold 10 — confirming the tree was right and the dashboard was the bug
Deployed to test-cup	✅	./scripts/deploy.sh test-cup — https://test-apps.dsco.io/botmonAlpha returns 200 on the deployed branch. Last deploy was webapp 4.3.3 on 2026-08-27; the 4.3.4 review fixes are not on the stage yet
Asset-prefix fix (defect 2)	✅	Verified live on test-cup: bot.png returns 200 via the CloudFront assets prefix the fix now uses, and 302 (auth redirect, not the image) via the bare-base path the old code built — confirming both the root cause and the fix on a real stage, which dev cannot show since both prefixes are empty there
Deployed-stage visual check	⏳	Stage is deployed and awaiting a human side-by-side pass against legacy botmon (edge times, trace icons, rogue badge, chart line, humanized durations)
Original rogue incident replay	❌	The bot from the original report had recovered (errorCount back to 0) — rogue is a transient, reset-on-success counter and history is not stored. Verified against a second, currently-rogue bot instead.

Rollout

• Feature flag: no
• Backwards compatible: yes — UI-only display fixes; no API contract, schema, or payload changes
• Data migration: no
• Rollback path: revert the PR and redeploy; nothing persists state

AI Usage

• Tool(s): Claude Code (Opus 5 / Fable 5), 2026-08-25 → 2026-08-28
• Skills / sessions: create-task-from-jira, task-toml, worktree-for-task, ambiguity-and-decision-log, knowledge-capture, pr-from-task; four parallel code-archaeology agents, one per reported defect
• The moment AI changed the outcome: the archaeology pass on the rogue defect found that calculateErrorCount summed errors off read/write stat entries — a field the Leo stats table never writes — by cross-reading legacy lib/stats.js, which records errors only under current.execution. That made blocked and the error-rate alarm structurally unreachable in production and pinned the catalog's Errors column to 0 for every bot, and it explained the reported symptom exactly: 35 errors visible on the bot's own Error Count chart (which reads execution.errors) while its status showed healthy. The same pass found the test fixtures encoded the same wrong shape, which is why the suite was green over a bug — so the fixtures were corrected alongside the code. Without that cross-read the likely "fix" was to lower the rogue threshold, which would have changed nothing.
• Where it needed a second pass: the same rogue work then shipped a fix that could not reach the surface it was written for, and a unit test that passed while asserting a freshness ordering the runtime inverted. An AI review pass caught it, and the follow-up traced the mechanism (settingsErrorCount ?? catalogEntry?.errorCount against a settings record nothing refreshed) and the two related defects in round 2 above. The lesson generalizes past this PR: a test asserting that one data source is fresher than another pins a premise that lives in the callers, not in the helper under test — so it can stay green over a live bug. That is now written up in the workspace knowledge base.

Reviewers

• L1 (right thing?): @mecham-lynn
• L2 (correct implementation?): @mecham-lynn

Checklist

• Brief + Codework JIRA tickets linked above
• Brief written before code started; plan reviewed; brief re-edited where plan revealed gaps (defects 5 and 6 added to brief, codework, and plan mid-flight)
• Tests present (or testable rationale for any "not run" row)
• AI usage section is substantive — names the moment AI changed the outcome
• Per-repo CLAUDE.md / AGENTS.md updated if a gap surfaced
• One concern per PR (no bundled schema + behavior + refactor)
• No ~/code/... paths in the body
• feature/<CODEWORK-KEY>-<slug> branch; <CODEWORK-KEY> <description> title
• Code follows project style; L1 then L2 self-review complete

---

Note

Medium Risk
Touches core bot status, stats merge, and workflow visualization paths used in production monitoring; changes are UI/display-focused with broad regression test coverage but high user visibility if edge cases remain.

Overview
Aligns botmonAlpha with legacy botmon for six dogfooding defects (ES-4034), mostly by fixing how stats are merged, keyed, and interpreted—not just UI polish.

Workflow graph — mergeStatsResults no longer throws on records missing read/write and clones per-queue defaults so edges don’t 404 the stats API or show 0 / N/A everywhere. Link stats are keyed downstream-first via shared linkStatsKey(), with correct handling of queue vs bot LeoStats shapes, checkpoint catch-up, and getLinkLabel() read lag (event age vs run time, window compareTimestamp). Refresh paths also re-fetch bot settings and rebuild the relationship tree when status changes.

Rogue / blocked / alarms — Window errors and executions read current.execution only; error-rate alarms use legacy’s count comparison. Dashboard header rogue uses the page settings errorCount (with catalog fallback and bot: id matching) and timers refresh both settings and catalog. Tree nodes get clearer error styling.

Charts & trace — Red annotation marks read checkpoint only where lag applies (not wall-clock “now” on every chart). Duration bucket charts gain dataIsTimeBased humanization. Trace PNGs use assets || base.

Docs & tests — AGENTS.md footguns, fixtures/types drop phantom read/write errors, plus unit, merge, link-key, link-record, and source-guard tests. Version 4.3.5.

^{Reviewed by Cursor Bugbot for commit 917b303. Bugbot is set up for automated code reviews on this repo. Configure here.}

[ch-snyk-sa]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1365891. Configure here.}