Run View execution and structured Results

[5aumit]

Run View execution and structured Results

Status: draft for approval

Problem

SetScout currently makes an ML researcher wait for a mostly synchronous Markdown response. The
user cannot clearly see what meaningful work is happening, which parts completed with reduced
coverage, or why one dataset ranked above another.

The production UI also lacks a stable way to present the real pipeline as it runs. The existing
uncommitted Ledger presentation is based on an older visual direction and must not become the
production source by accident.

Selected solution

Build the Run View inside the existing Gradio application. Use the selected B: Split
workspace direction from prototype_run_view.py as the visual reference, but rewrite it as
production code.

During an active Run:

• Collapse the editable form into a read-only Search Brief.
• Keep the Search Brief and four persistent Stages in a left rail.
• Use the right workspace for curated Run Activity.
• Show only typed, user-safe updates and measured counts.
• Never show prompts, internal node names, retries, stack traces, percentages, or invented ETAs.

After a terminal outcome:

• Collapse the Search Brief and Stage history so Results receive most of the width.
• Let the user expand the completed Run details when needed.
• Show a compact Run Summary above the terminal content.
• For a successful Run, show the top three ranked Results first and a Show all Results action.
• Let each Result expand inside the ranked list, pushing lower cards down.
• Optimize the first version for choosing the best dataset, not side-by-side comparison.

How it works

Editable form
  -> submitted request
  -> Search Brief
  -> live or replay Run source
  -> typed Run Events
  -> source-neutral Run View state
  -> active progress or terminal Results

• setscout/graph/ continues to own the four-stage LangGraph pipeline.
• A live Run source exposes graph updates incrementally.
• setscout/runs.py translates pipeline facts into the stable Run Event contract established by
  Ledger: make Runs report what really happened #5.
• An offline replay source yields the same public contract from reviewed fixtures.
• A presentation module turns the current Run state into the selected B layout.
• app.py owns Gradio controls and switches the form, active Run View, and terminal controls.

The presentation must not know whether its events came from the real pipeline or replay.

Stable user-facing Stages

1. Prepare Search Brief
2. Search dataset sources
3. Gather documentation evidence
4. Evaluate and rank candidates

Each Stage moves from Waiting to Running, then to Completed, Completed with warnings, or Failed.
A Run ends exactly once as Completed, Completed with warnings, Empty Results, Failed, or
Cancelled.

Results contract

Results are structured data, not parsed Markdown. Every displayed dataset assessment needs:

• candidate identity
• dataset name
• source
• original dataset URL
• rank
• fit summary
• Requirement Checks
• Evidence Citations where evidence exists
• known issues
• dataset-specific Result Limitations

The current completed RunResults retains evaluations but not candidate name, source, or URL.
Production implementation must join or retain those candidate fields before presentation.

Every candidate selected by the search graph must receive exactly one rank before Results can be
shown. The default cap is eight, but the UI must handle a configured higher cap without changing
its basic structure.

Result interaction

• Show the top three Results initially when more than three exist.
• Show all Results reveals the complete ranked set and updates the visible count truthfully.
• A collapsed card shows rank, name, source, fit summary, and a compact Requirement Check summary.
• Activating a card expands its assessment in place.
• Expanded detail shows Requirement Checks, evidence, known issues, limitations, and an
  Open dataset link.
• Multiple cards may be expanded. No comparison mode is included in the first version.
• Dataset links open the original Hugging Face or Kaggle page in a new tab.

Outcomes

• Completed: Show Run Summary and ranked Results.
• Completed with warnings: Show Results plus persistent run-wide and dataset-specific
  limitations.
• Empty Results: Explain that sources completed successfully and suggest safe refinements.
• Failed: Preserve the Search Brief and completed Stage history, explain the failure in plain
  language, and show no incomplete Results.
• Cancelled: Preserve the Search Brief and completed Stage history and show no incomplete
  Results.

Entering edit mode preserves the previous completed Results for reference. Submitting a revised
Search Brief clears those Results before the new Run becomes Queued.

Replay and test mode

• Production presentation development must have a no-cost offline replay path.
• Replay fixtures are checked in, versioned, reviewed, and sanitized.
• Fixtures cover Completed, Completed with warnings, Failed, Cancelled, and Empty Results.
• The developer-only Test run control is enabled only when
  SETSCOUT_ENABLE_TEST_RUN=1 is loaded from the repository-root .env.
• Replay and live execution use the same Run View rendering path.

Presentation boundary

• Keep Gradio Blocks as the application shell.
• Prefer native Gradio controls for buttons, selectors, and form behavior.
• Use custom HTML and CSS where the Run View needs structured layout and expandable Result cards.
• Avoid custom JavaScript unless native HTML and Gradio events cannot provide the required
  behavior.
• Support light and dark themes through Gradio theme variables.
• Reflow into one usable column on narrow screens.
• Preserve keyboard operation, visible focus, and reduced-motion behavior.

Security

• Gemini credentials remain session-only.
• Credentials never enter the Search Brief, Run Events, Results, replay fixtures, URLs, logs, or
  persisted presentation state.
• Replay data contains no prompts, traces, stack traces, credentials, or private user content.

Acceptance

• A real API-backed Run visibly advances through the four Stages and reveals actual structured
  Results without waiting for one final Markdown response.
• The same presentation can replay reviewed offline fixtures without credentials or network.
• The active B split view shows persistent Stage state and curated Run Activity.
• Terminal Run details collapse while remaining expandable.
• Successful Results show the top three first, reveal all ranked candidates on request, and
  expand each assessment in place.
• Result cards contain real candidate identity, source, URL, Requirement Checks, evidence,
  known issues, and limitations supported by the result contract.
• Failed, Cancelled, and Empty Results never expose incomplete rankings.
• Completed with warnings retains visible limitations.
• The UI works in light, dark, narrow, keyboard, and reduced-motion conditions.
• Credentials and internal pipeline details never appear in public Run state.

Delivery

Work blockers-first through the existing tickets:

1. Run View: build the offline replay #6 builds the source-neutral production Run View with offline replay.
2. Run View: stream a real Run #7 connects a real API-backed Run to that same presentation path.
3. Run View: explain Results with evidence and warnings #8 completes expandable evidence and warning behavior.
4. Run View: handle Stop, failures, and changed searches #9 adds Stop, remaining terminal outcomes, and search revision behavior.
5. Run View: finish accessibility and credential safety #10 finishes responsive, accessible, theme, and credential-safety behavior.

#14 remains a separate deferred follow-up for downloading completed Runs.

Testing

• Keep contract tests around the Run Event adapter and complete-ranking rule.
• Add deterministic live-source tests using mocked graph updates.
• Add replay tests for all five terminal outcomes.
• Add end-to-end browser coverage for:
  • form collapse and Search Brief;
  • visible Stage progression;
  • Run Activity;
  • terminal rail collapse and expansion;
  • top-three and Show-all Results;
  • Result expansion and dataset links;
  • warnings, failure, cancellation, and Empty Results;
  • returning to edit and submitting a revised Run;
  • narrow layout, keyboard focus, and theme coherence.
• Manually smoke-test at least one real Gemini-backed Run before calling the feature ready.

Out of scope

• Downloading completed Runs, tracked separately in Allow users to download a completed Run #14.
• Persistent cross-session Run history or user accounts.
• A separate frontend deployment.
• Side-by-side Result comparison.
• New dataset sources or new evidence kinds.
• Parsing the generated Markdown report to reconstruct structured Results.
• Shipping prototype_run_view.py as production code.

Existing work

• Ledger: make Runs report what really happened #5 and the committed typed Run Event contract remain the foundation.
• prototype_run_view.py is the visual primary source only.
• The current uncommitted app.py, setscout/presentation.py, setscout/replay.py, fixture, and
  focused tests target the older Ledger direction. Review and salvage useful contracts or fixture
  work, but rewrite presentation decisions against this spec.
• Issues Run View: build the offline replay #6-Run View: finish accessibility and credential safety #10 should be revised rather than duplicated.

[5aumit]

Planning update

The immediate goal is now an honest public demo before the evaluation harness: usable layout, grounded evidence, truthful terminal states, and safe source failures.

Work order:

1. Finish Run View: explain Results with evidence and warnings #8 with the clarified evidence guarantees below.
2. Complete the focused foundation tickets: Make source search failures safe #17 source failures, Make dataset requirements honest #18 honest requirements, Keep Results usable on normal screen sizes #19 Result layout, and Add a minimal quality gate #20 quality checks.
3. Work Run View: handle Stop, failures, and changed searches #9 after Run View: explain Results with evidence and warnings #8.
4. Finish the remaining Run View: finish accessibility and credential safety #10 accessibility and credential-safety work after Run View: handle Stop, failures, and changed searches #9.
5. Return to the evaluation harness only after this demo slice is stable.

#14 remains intentionally deferred.

[5aumit]

Current-code references

These links show where the current Run View flow is implemented:

• The Gradio generator starts the Run, consumes streamed pipeline patches, and decides when Results are shown: app.py:129-191.
• The public streaming boundary yields raw graph-node patches: setscout/pipeline.py:57-68.
• The Run adapter translates those patches into public Run state: setscout/runs.py:201-275.
• The custom Run View renders the resulting state: setscout/presentation.py:183-215.

These are implementation references, not a replacement for this issue’s user-facing acceptance criteria.