Skip to content

FE-1505: Time each interviewer turn by purpose - #9347

Open
lunelson wants to merge 10 commits into
ln/fe-1506-legible-panel-statusfrom
ln/fe-1505-timing
Open

FE-1505: Time each interviewer turn by purpose#9347
lunelson wants to merge 10 commits into
ln/fe-1506-legible-panel-statusfrom
ln/fe-1505-timing

Conversation

@lunelson

Copy link
Copy Markdown
Contributor

🌟 What is the purpose of this PR?

Condition 5 measured about 145 seconds per interviewer turn without saying where the time goes. This branch attributes each model call, compaction, and repair to a harness purpose, emits privacy-safe server telemetry spans, and makes baseline timing runs kill-safe and bounded so the per-purpose split can be read from a short run or from the Proof 0 panel run.

Draft: opened so the stack is visible end to end. The description is completed when the branch is tied off.

🔗 Related links

🚫 Blocked by

  • Stacks on ln/fe-1506-legible-panel-status; lands after it.

🔍 What does this change?

Per-purpose turn timing in the Flue binding and the baseline runner; OpenTelemetry spans in the server; BRUNCH_BASELINE_HARD_STOP and incremental condition-5.timings.jsonl writes; run-specific timing evidence preserved.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

@vercel

vercel Bot commented Aug 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview Aug 27, 2026 5:56pm
petrinaut Ready Ready Preview Aug 27, 2026 5:56pm
petrinaut-docs Ready Ready Preview Aug 27, 2026 5:56pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
hashdotdesign-tokens Ignored Ignored Preview Aug 27, 2026 5:56pm

Request Review

lunelson commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

@cursor

cursor Bot commented Aug 26, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Changes are confined to evaluation harness output, dev-server telemetry (content suppressed), and tests; no auth, persistence, or production interview logic paths are altered.

Overview
Condition 5 baseline runs can now break down wall-clock per Flue model call into interview, sweep, and repair instead of only reporting coarse per-turn cost.

A new createTurnTimingRecorder listens to Flue observe() events (prompt nesting, brunch_sweep refused/applied, <settlement-check / <sweep-repair signals) and records TurnTimingRecord rows. The harness runner attaches timings to each turn and condition-5.raw.json, appends condition-5.timings.jsonl after every interviewer turn (so short runs still leave evidence), and prints per-purpose ms in transcript turn headers. BRUNCH_BASELINE_HARD_STOP caps interviewer turns (default 24); BRUNCH_BASELINE_OUTPUT_DIR can isolate production evidence.

The brunch-agent dev app registers OpenTelemetry via @flue/opentelemetry with content: false; the walking-skeleton integration asserts expected agent/chat/tool spans and that sensitive content attributes stay off spans.

Tests cover the purpose splitter, incremental timing persistence under hard stop, and a baseline fixture path that refuses the first sweep then repairs so repair timings are exercised.

Reviewed by Cursor Bugbot for commit 45ab0cd. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions github-actions Bot added area/deps Relates to third-party dependencies (area) area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team area/tests New or updated tests area/apps labels Aug 26, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 35549b9. Configure here.

Comment thread apps/brunch-agent/test/baseline-harness.test.ts
lunelson and others added 10 commits August 27, 2026 19:41
Record Flue wall-clock timings in run artifacts so latency evidence no longer relies on token-volume inference.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Allow short cross-check runs and persist each completed turn's timing records immediately so interrupted runs retain evidence.

Co-authored-by: Cursor <cursoragent@cursor.com>
Select the applied sweep explicitly now that timing coverage exercises a refused repair path, and allow the incremental-write probe enough time under the full suite.

Co-authored-by: Cursor <cursoragent@cursor.com>
The purpose-splitter unit test imports Flue observation types, so the
hermetic entry-point inventory has to name it or the architecture gate
fails.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps area/deps Relates to third-party dependencies (area) area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

1 participant