Skip to content

fix(verdict): render per-case stdout in Run modal (reverses BRAT #2) - #28

Merged
LikeSundayLikeRain merged 4 commits into
mainfrom
fix/run-verdict-per-case-stdout
Jun 11, 2026
Merged

fix(verdict): render per-case stdout in Run modal (reverses BRAT #2)#28
LikeSundayLikeRain merged 4 commits into
mainfrom
fix/run-verdict-per-case-stdout

Conversation

@LikeSundayLikeRain

Copy link
Copy Markdown
Owner

Summary

The Run verdict modal showed the same combined stdout under every test-case tab. Each tab should show only its own case's print() output.

Root cause

The BRAT #2 fix (2026-06-05) at src/solve/verdictModalRenderer.ts treated code_output as a flat run-global blob. The reasoning was: when LC returns code_output as string[], each element looks like one line of combined stdout — verified against node_modules/@leetnotion/leetcode-api/lib/index.js:1887, where formatTestOutput joins the array on \n.

The leetnotion library does join on \n, but the wire shape is element-per-CASE, not per-line. The library normalization is what's flat; the underlying response isn't.

Verification

Live probe against leetcode.com/problems/two-sum/interpret_solution/ on 2026-06-11 with a 3-case Python3 solution that prints f"CASE-MARKER nums={nums} target={target}" per case. Response:

total_testcases: 3
code_output: [
  "CASE-MARKER nums=[2, 7, 11, 15] target=9",
  "CASE-MARKER nums=[3, 2, 4] target=6",
  "CASE-MARKER nums=[3, 3] target=6"
]
std_output_list: [
  "CASE-MARKER nums=[2, 7, 11, 15] target=9\n",
  "CASE-MARKER nums=[3, 2, 4] target=6\n",
  "CASE-MARKER nums=[3, 3] target=6\n",
  ""
]

Both fields carry per-case stdout. std_output_list is the modern canonical name; code_output is a deprecated alias (the xuhuafeifei/leetcode-runner Java model marks code_output @Deprecated with the comment "this field is not used anymore, please use std_output_list").

Changes

  • src/solve/types.ts — type std_output_list?: string[] and expected_std_output_list?: string[] on RunCheckResponse (previously fell through the [k: string]: unknown index signature).
  • src/solve/verdictModalRenderer.tsrenderActiveCase reads per-case stdout via splitOutput(res.std_output_list ?? res.code_output, arity)[activeIdx] with the trailing \n stripped. The code_output fallback covers older or partial response shapes (e.g., the existing run-sample.json fixture has empty code_output: [] despite carrying std_output_list).
  • tests/solve/verdictModalRenderer.test.ts — replaces the BRAT feat: v1.1 — Contest, AI Coach, and Preview #2 line-per-element regression tests with per-case wire-shape tests using the live probe payload. New assertions:
    • Each tab shows only its own case's stdout (clicking through Case 1/2/3 swaps the marker).
    • Trailing \n from std_output_list is stripped for display.
    • Falls back to code_output when std_output_list is absent.
    • Empty padded slots (e.g., LC pads to 4 entries for 3 cases) render no Stdout section.

Test plan

  • npx vitest run tests/solve/verdictModalRenderer.test.ts — 32/32 pass (4 new + 28 existing).
  • npm run build (tsc --noEmit + esbuild prod) — green.
  • npm run lint — green (1 unrelated unused-import warning).
  • Live verified the wire shape with a real LC API probe (Two Sum, Python3, 3 cases).
  • Deployed to interview-prep BRAT vault for manual verification.

The full vitest run shows 5 timeouts in unrelated AI/preview test files (tests/ai/clearKey.test.ts, tests/ai/probe-debounce.test.ts, tests/ai/reset-disclosures-command.test.ts, tests/preview/router.test.ts). These are pre-existing flakes under load — running them in isolation off main (without my changes) reproduces the same flake-vs-pass behavior. Confirmed by stashing my diff, re-running just those files: all 38 tests pass in 3.45s.

Previously the Run verdict modal showed the same combined stdout under
every test-case tab. The BRAT #2 fix (2026-06-05) treated `code_output`
as a flat run-global blob because the leetnotion library's
`formatTestOutput` joins the array on '\n' before exposing it
(node_modules/@leetnotion/leetcode-api/lib/index.js:1887). That join
made the field look line-per-element on the wire — but the wire shape
is element-per-case.

Verified live 2026-06-11 against leetcode.com with a 3-case Two Sum
probe printing distinct CASE-MARKER strings. Response carried:

  total_testcases: 3
  code_output:     [3 elements, exact-length, no trailing pad]
  std_output_list: [4 elements, padded with trailing '', each ending '\n']

Both fields are per-case. `std_output_list` is the modern canonical
name; `code_output` is a deprecated alias (per the leetcode-runner
Java model's @deprecated annotation pointing to std_output_list).

Changes:
- Type std_output_list / expected_std_output_list on RunCheckResponse
- Replace global asString(code_output) with splitOutput-driven per-case
  read inside renderActiveCase; prefer std_output_list, fall back to
  code_output for older response shapes
- Strip trailing newline added by std_output_list elements
- Replace BRAT #2 line-per-element regression tests with per-case
  wire-shape tests using the live probe payload as the fixture; pin
  that switching tabs swaps stdout and that empty padded slots render
  no Stdout section
When a Run hits a runtime error mid-execution (case N throws after
case N-1 passed), LC's interpret_solution returns:

  status_code: 15
  code_answer: ["[0,1]", ""]      (truncated at the throwing case)
  compare_result: "100"           (full arity, '0' for throw + skipped)
  total_testcases: 3
  full_runtime_error: <stack trace>

Pre-fix the renderer routed this to the per-case tabs path and showed
Case 1 PASS / Case 2 FAIL / Case 3 FAIL with empty Output boxes —
no stack trace anywhere. Verified live 2026-06-11 with a 3-case Two
Sum probe that solved case 1 and raised on call #2.

renderRunResult now detects mid-run RE (status_code 15 + error text +
partial code_answer) and:
  - tags throwIdx = code_answer.length - 1 as 'threw'
  - tags i > throwIdx as 'skipped' (LC stops sequentially)
  - default-activates the throwing tab so the user lands on the trace
  - throwing tab replaces Output with an Error pre containing the
    full stack trace; keeps Input + Expected for context
  - skipped tab shows a "Not executed — an earlier case raised an
    exception." placeholder; keeps Expected (LC pre-computes it)
  - PASS tabs render normally (Input + Stdout + Output + Expected)

Tab chips:
  PASS / FAIL / THREW (red, like fail) / SKIP (muted gray)

CSS additions in styles.css mirror existing chip variants.

7 new vitest cases pin: default-active throwing tab, chip variants,
stack-trace placement on the throwing tab, normal Output preserved
on PASS tabs, "Not executed" placeholder on SKIP tabs, AI Debug
button wiring, and a regression guard that the all-fail path still
routes through hasRunErrorPayload's single-error-block layout.
User-facing text only — clearer and more conventional than 'THREW'
for non-developer audiences. The internal state name (`'threw'`) and
CSS class (`leetcode-verdict-case-chip--threw`) stay unchanged so the
existing styles + tests keep working unchanged.
@LikeSundayLikeRain
LikeSundayLikeRain merged commit 2e20c6c into main Jun 11, 2026
1 check passed
@LikeSundayLikeRain
LikeSundayLikeRain deleted the fix/run-verdict-per-case-stdout branch June 11, 2026 20:05
LikeSundayLikeRain added a commit that referenced this pull request Jun 12, 2026
Promote 1.3.1 from beta (1.3.1-beta.3) to stable. Code is unchanged
from 1.3.1-beta.3 — only docs/chore commits differ.

- manifest.json / package.json / package-lock.json -> 1.3.1
- versions.json: add 1.3.1 -> minAppVersion 1.12.7 (CI does not patch this)
- CHANGELOG.md: backfill [1.3.0] (inline-widget milestone) and
  [1.3.1] (#24-#28) sections

All 5 BRAT beta items resolved. CI green on the promoted commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant