fix(submit): batch daily_breakdowns writes on the merge path - #96
Closed
yoominho91 wants to merge 1 commit into
Closed
fix(submit): batch daily_breakdowns writes on the merge path#96yoominho91 wants to merge 1 commit into
yoominho91 wants to merge 1 commit into
Conversation
…otfun#93) The merge path writes one row per incoming day with a separate PostgREST call, so a submission with long history costs one round-trip per day — ~270 for the report in sculptdotfun#93. The route races that against a 25s timer, and since the timer doesn't cancel the in-flight work, the write can land after the race already rejected. The user sees a generic 500 telling them to check cc.json while the data is on the leaderboard, which invites a retry of a write that succeeded. `daily_breakdowns` has UNIQUE(submission_id, date), so a single upsert with `onConflict: "submission_id,date"` covers both the update and insert branches. Collect the rows in the fold loop and write them in chunks of 500 afterwards, turning N round-trips into ceil(N/500). This removes the cause for the reported case rather than widening the timeout. The race itself (rejecting without cancelling) is still there and worth handling separately — it just stops being reachable at these sizes.
|
@yoo-minho is attempting to deploy a commit to the sculpt Team on Vercel. A member of the Team first needs to authorize it. |
Author
|
Closing — reopened from my personal account as #98 (same commit, same diff). This one was opened from a work account by mistake. Sorry for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the cause reported in #93.
What happens
updateExistingSubmissionwrites onedaily_breakdownsrow per incoming day, each as its own PostgREST call:A submission with long history therefore costs one round-trip per day — about 270 in the report on #93. The route races that against a 25s timer, and the timer does not cancel the in-flight work, so the writes can land after the race has already rejected. The user gets a generic 500 telling them to check
cc.jsonwhile their data is visible on the leaderboard, which invites retrying a write that already succeeded.Change
daily_breakdownshasUNIQUE(submission_id, date)(001_initial_schema.sql:83), so one upsert withonConflict: "submission_id,date"covers both branches. The fold loop now collects rows and writes them in chunks of 500 afterwards, turning N round-trips intoceil(N/500)— 270 days becomes a single request.Behaviour is otherwise unchanged: the per-machine merge in
mergeMachineContributionstill decides each row, so#43semantics (new machine adds a slice, same machine replaces its own) are untouched.dailyMapis still populated the same way, so the totals recalculated below see the same values.Scope
This removes the cause for the reported size rather than widening the timeout. The race itself — rejecting without cancelling, so a late success looks like a failure — is still there and worth handling separately; it just stops being reachable at these sizes. Happy to take that on in a follow-up if you want it.
Checks
pnpm exec tsc --noEmit: 14 pre-existing errors onmain, still 14 after — none in the touched file.pnpm build: compiles successfully. Page-data collection fails on bothmainand this branch without Supabase env vars, so unrelated.