[TRUNK-17757] Document Test History tab on Flaky Tests test detail page#481
[TRUNK-17757] Document Test History tab on Flaky Tests test detail page#481samgutentag wants to merge 1 commit intomainfrom
Conversation
|
Merging to
|
|
Claude finished @samgutentag's task in 1m 21s —— View job PR Review
The documentation for the Test History tab is well-structured and clearly written. The reorganization of the "Test case details" section into a tabbed layout, and the renaming of the old "Test history" section to "Status history" to avoid naming conflict, are both good improvements. Two minor issues found (inline comments posted):
Everything else looks good — headings hierarchy is correct, tables are well-formatted, filter descriptions are accurate, and cross-references between docs are updated appropriately. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
|
||
| <table><thead><tr><th width="188">Column</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>When the test ran, displayed in UTC.</td></tr><tr><td>Duration</td><td>How long the test took to execute.</td></tr><tr><td>Branch</td><td>The branch context. Shows "Mergequeue testing #NNN" for merge queue runs, "#NNN" for pull request runs, or the branch name for other runs.</td></tr><tr><td>Commit</td><td>The first 7 characters of the commit SHA.</td></tr></tbody></table> | ||
|
|
||
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error. |
There was a problem hiding this comment.
The chart color legend above (lines 98–101) lists Gray = Skipped, but this row-border description mentions orange for error without including gray/skipped. The two color references are inconsistent — readers may wonder why the row borders and chart use different color sets.
Consider aligning them, e.g.:
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error. | |
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, gray for skipped, and orange for error. |
| <figure><img src="../.gitbook/assets/qurantine-individual-tests-revert.png" alt=""><figcaption></figcaption></figure> | ||
|
|
||
| To review a history of all quarantine changes on a test, use the **Quarantine Events** filter within the **Test History** section. This will show every override, setting change, and comment, along with the author and timestamp for each entry. | ||
| To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry. |
There was a problem hiding this comment.
The last sentence "These views show every override, setting change, and comment..." conflates two tabs that show different things: the Events tab shows overrides/setting changes/comments, while the Test History tab (filtered to Quarantined) shows runs. Consider splitting for clarity:
| To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry. | |
| To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. The Events tab shows every override, setting change, and comment, along with the author and timestamp for each entry. |
samgutentag
force-pushed
the
sam-gutentag/test-run-history-tab
branch
from
389c2e8 to
427dd03
Compare
|
Claude finished @samgutentag's task in 1m 9s —— View job PR Review
The new Test History tab documentation is well-structured and the reorganization into a tabbed layout is an improvement. The same two issues from the previous review remain unaddressed (inline comments re-posted):
|
|
|
||
| <table><thead><tr><th width="188">Column</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>When the test ran, displayed in UTC.</td></tr><tr><td>Duration</td><td>How long the test took to execute.</td></tr><tr><td>Branch</td><td>The branch context. Shows "Mergequeue testing #NNN" for merge queue runs, "#NNN" for pull request runs, or the branch name for other runs.</td></tr><tr><td>Commit</td><td>The first 7 characters of the commit SHA.</td></tr></tbody></table> | ||
|
|
||
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error. |
There was a problem hiding this comment.
The daily runs chart legend above (lines ~93–101) includes Gray = Skipped, but gray/skipped is missing from this row border color list. Consider adding it for consistency:
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error. | |
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, gray for skipped, and orange for error. |
| <figure><img src="../.gitbook/assets/qurantine-individual-tests-revert.png" alt=""><figcaption></figcaption></figure> | ||
|
|
||
| To review a history of all quarantine changes on a test, use the **Quarantine Events** filter within the **Test History** section. This will show every override, setting change, and comment, along with the author and timestamp for each entry. | ||
| To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry. |
There was a problem hiding this comment.
The closing sentence "These views show every override, setting change, and comment..." still conflates the two tabs: the Events tab shows overrides/setting changes/comments, while the Test History tab only shows runs. Applying the cleaner split from the previous suggestion:
| To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry. | |
| To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. The Events tab shows every override, setting change, and comment, along with the author and timestamp for each entry. |
|
Triage status (2026-05-05): merge conflicts with main in |
|
Verification status (2026-05-07): Verified: customers can use this. Ready to publish.
Merge readiness:
Suggested next action: push the rebased branch, then merge. |
Update the dashboard page to describe the new tabbed layout (Summary, Test History, Monitors, Events) and add comprehensive documentation for the Test History tab including the daily runs chart, filter bar, runs table, and run detail panel. Re-applied against current main and integrated with the branch glob filter content from #576. Addresses prior PR review comments: row border colors now include gray for skipped, and the quarantining wording is scoped to the Events tab. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
samgutentag
force-pushed
the
sam-gutentag/test-run-history-tab
branch
from
427dd03 to
d41713b
Compare
|
Claude finished @samgutentag's task in 2m 47s —— View job PR Review
The two issues from prior reviews are resolved in the rebased branch — the row-border color list now includes Found four new items on this pass (inline comments posted):
Branch is currently flagged |
|
|
||
| The Summary tab shows an overview of the test’s recent run results and groups past failures by unique failure type. | ||
|
|
||
| #### **Failure types** |
There was a problem hiding this comment.
Formatting inconsistency. This H4 heading uses bold (#### **Failure types**), but its siblings under "Test History tab" — #### Daily runs chart, #### Filters, #### Runs table, #### Run detail panel — and the adjacent #### Failure details are plain. Either drop the bold here for consistency, or apply it to the others.
| #### **Failure types** | |
| #### Failure types |
| ### Status history | ||
|
|
||
| <figure><picture><source srcset="../.gitbook/assets/test-history-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/test-history-light.png" alt=""></picture><figcaption></figcaption></figure> | ||
|
|
||
| Tests may transition between Healthy, Flaky, and Broken states multiple times over their lifetime. You can see previous status changes in the Status History view, including an explanation for why each transition occurred and which monitor triggered it. |
There was a problem hiding this comment.
Section placement is unclear. "Status history" is rendered at H3 — the same level as ### Summary tab and ### Test History tab — which makes it look like another tab. But it isn't listed in the tabs bullet list at lines 70–73 (Summary / Test History / Monitors / Events). Readers will reasonably ask: where does Status history actually live in the UI? Is it part of the Events tab? A separate panel on the page?
Consider either (a) adding it to the tabs bullet list at the top, or (b) prefacing this section with one sentence that locates it (e.g., "The Events tab also surfaces a Status History view, which..."). Right now the heading hierarchy implies one thing and the content implies another.
|
|
||
| The runs table displays a paginated list of individual test runs (25 per page) with the following columns: | ||
|
|
||
| <table><thead><tr><th width="188">Column</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>When the test ran, displayed in UTC.</td></tr><tr><td>Duration</td><td>How long the test took to execute.</td></tr><tr><td>Branch</td><td>The branch context. Shows "Mergequeue testing #NNN" for merge queue runs, "#NNN" for pull request runs, or the branch name for other runs.</td></tr><tr><td>Commit</td><td>The first 7 characters of the commit SHA.</td></tr></tbody></table> |
There was a problem hiding this comment.
Verify product naming. "Mergequeue testing" (one word) doesn't appear anywhere else in this docs repo — every other reference uses "merge queue" (two words). If the UI literally renders "Mergequeue testing #NNN" as a single token (e.g., it's an internal label not yet rebranded), then this is correct documentation. Otherwise it should match the product's user-facing form, e.g., "Merge queue testing #NNN".
| <table><thead><tr><th width="188">Filter</th><th>Description</th></tr></thead><tbody><tr><td>Result</td><td>Toggle between <strong>All</strong>, <strong>Pass</strong>, <strong>Fail</strong>, and <strong>Quarantined</strong> to show only runs with a specific outcome.</td></tr><tr><td>SHA</td><td>Filter by commit hash. Matches runs whose SHA starts with the entered text.</td></tr><tr><td>Branch</td><td>Filter by branch name. Accepts exact names or glob patterns. Use <code>*</code> to match any sequence of characters and <code>?</code> to match a single character.</td></tr></tbody></table> | ||
|
|
||
| Branch filter examples: | ||
|
|
||
| | Pattern | Matches | | ||
| |---|---| | ||
| | `main` | The branch named `main` exactly | | ||
| | `release/*` | All release branches, e.g. `release/1.0`, `release/2.3` | | ||
| | `feature-??` | Feature branches with a two-character suffix, e.g. `feature-v2` | | ||
| | `trunk-merge/*` | All merge queue branches | | ||
|
|
||
| All filters combine using AND logic, so you can use them together to find specific runs. For example, select **Fail** and enter a branch name to see all failures on that branch. Click the **Reset** button to clear all active filters. | ||
|
|
||
| Filter state is saved in the URL, so you can share or bookmark a filtered view. | ||
|
|
||
| #### Runs table | ||
|
|
||
| The runs table displays a paginated list of individual test runs (25 per page) with the following columns: | ||
|
|
||
| <table><thead><tr><th width="188">Column</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>When the test ran, displayed in UTC.</td></tr><tr><td>Duration</td><td>How long the test took to execute.</td></tr><tr><td>Branch</td><td>The branch context. Shows "Mergequeue testing #NNN" for merge queue runs, "#NNN" for pull request runs, or the branch name for other runs.</td></tr><tr><td>Commit</td><td>The first 7 characters of the commit SHA.</td></tr></tbody></table> | ||
|
|
||
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, gray for skipped, and orange for error. |
There was a problem hiding this comment.
Mismatched result categories across three places in this section.
- Daily runs chart legend (lines 121–124): Pass / Fail / Quarantined / Skipped
- Result filter (line 132): All / Pass / Fail / Quarantined (no Skipped)
- Row border colors (line 153): pass / fail / quarantined / skipped / error (adds Error)
If the product genuinely doesn't let users filter by Skipped and the chart genuinely doesn't bucket Error, this is fine — but a reader will hit it and wonder why. Worth either reconciling the lists or adding one sentence acknowledging the asymmetry (e.g., "Errored runs are included in the Fail category in the chart" or "Skipped runs aren't filterable, since they don't reflect a test outcome").
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
1 participant
See PR body below