Skip to content

mark qrep failure in db and filter out of result - #4775

Open
jgao54 wants to merge 2 commits into
mainfrom
mark-qrep-failure
Open

mark qrep failure in db and filter out of result#4775
jgao54 wants to merge 2 commits into
mainfrom
mark-qrep-failure

Conversation

@jgao54

@jgao54 jgao54 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

When a QRep workflow fails (for example a snapshot clone during a CDC initial load or a table addition), its peerdb_stats.qrep_runs row keeps consolidate_complete = false. InitialLoadSummary reports it as in progress forever, and a retried attempt for the same table adds a second row, so the UI shows a table with more than one in-progress run.

Changes:

  • Add a failed boolean column to peerdb_stats.qrep_runs
  • QRepFlowWorkflow now runs a MarkQRepRunFailed activity when it returns an error
  • InitialLoadSummary filters out failed runs, so only completed and in-progress runs reach the UI.
  • Failed rows and their partitions stay in the catalog, so the history of failed attempts is preserved for debugging.

Testing:

  • add integration test
  • verified peerdb UI and db locally

Fixes: DBI-1105

@jgao54 jgao54 changed the title mark qrep failure in db and filter out of result mark qrep failure in db and filter out of stats Sep 4, 2026
@jgao54 jgao54 changed the title mark qrep failure in db and filter out of stats mark qrep failure in qrep_runs and filter out of stats Sep 4, 2026
@jgao54 jgao54 changed the title mark qrep failure in qrep_runs and filter out of stats clear stale qrep_runs on retry Sep 5, 2026
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: Not flaky: the PR's new test Test_QRep_Retry_Sweeps_Unfinished_Run fails identically and instantly on all three matrix legs with a "duplicate key value violates unique constraint flows_name_unique" error, indicating the test/code creates the same QRep flow name twice; a separate pg18-only snapshot timeout is likely a genuine flake but is not the run-blocking failure.
Confidence: 0.92

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@jgao54 jgao54 changed the title clear stale qrep_runs on retry mark qrep failure in db and filter out of result Sep 5, 2026
@jgao54
jgao54 marked this pull request as ready for review September 5, 2026 01:20
@jgao54
jgao54 requested review from a team as code owners September 5, 2026 01:20
@claude

claude Bot commented Sep 5, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Comment thread flow/workflows/qrep_flow.go Outdated
) (_ *protos.QRepFlowState, err error) {
defer func() {
if err != nil && !workflow.IsContinueAsNewError(err) && !temporal.IsCanceledError(err) && ctx.Err() == nil {
markQRepRunFailed(ctx, config, workflow.GetInfo(ctx).OriginalRunID)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach is generic but makes it a bit harder to match the misbehaving area with the error handling for it.
Wdyt of this approach:

  • separate the first monitoring.InitializeQRepRun into its own activity right before GetQRepPartitions
  • put this defer after it

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, also separated out monitoring.InitializeQRepRun into RecordQRepRun and RecordQRepPartitions because the txn there for both tables was deceiving anyways as the second insert into qrep_runs is a no-op and that was always bothering me. Now it is more explicit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants