fix: serialize fellowship document-phase status recomputation - #89
Merged
Conversation
syncFellowshipStatus did an unlocked read-compute-write inside a READ COMMITTED transaction: save the document, read its sibling, then write the fellowship only if the computed status differed. Two concurrent document mutations each read the other's pre-commit row, both concluded "nothing changed", and neither wrote. Parallel uploads left both documents PENDING_REVIEW with the fellowship stuck at AWAITING_DOCUMENTS, which also drops it out of the ?status=DOCUMENTS_IN_REVIEW admin queue. Parallel approvals left both documents APPROVED with the fellowship stuck at DOCUMENTS_IN_REVIEW — that one is terminal, since reviewDocument rejects any document not in PENDING_REVIEW, so no admin action could re-trigger a sync and start-contract failed permanently. Take a pessimistic write lock on the fellowship row as the first statement of the transaction, before the document write. fellowship.applicationId is UNIQUE, so the fellowship row is a valid mutex for its application's documents. The loser blocks and then succeeds; nothing is aborted. Also: - syncFellowshipStatus takes the locked entity rather than re-reading it, so the precondition is structural. - Narrow the status write from save() to update(), which cannot clobber startDate/endDate/amountUsd. - startContract was an unlocked, non-transactional read-modify-write; wrap it in the same lock and re-check the status after acquiring it. - Log both branches that previously returned silently. - Add a migration recomputing status for fellowships in the document phase, repairing existing stuck rows in both directions. PENDING, ACTIVE and COMPLETED rows are never touched. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A fellowship was found with both fellow documents (
SIGNED_CONTRACT,W8BEN) atPENDING_REVIEWwhile the fellowship itself was stillAWAITING_DOCUMENTS. The fellow had submitted everything; the status never advanced.syncFellowshipStatusis the only writer of document-phase fellowship status, and it did an unlocked read-compute-write inside aREAD COMMITTEDtransaction: save the document → read its sibling → write the fellowship only if the computed status differed.When the fellow's UI uploaded both PDFs in parallel:
PENDING_REVIEWPENDING_REVIEWAWAITING_UPLOAD(B uncommitted)AWAITING_UPLOAD(A uncommitted)AWAITING_DOCUMENTS= current → no writeBoth documents land at
PENDING_REVIEW; the fellowship staysAWAITING_DOCUMENTS. It also drops out of the?status=DOCUMENTS_IN_REVIEWadmin queue, so nobody sees it needs review, and both no-op branches logged nothing.The mirror case is worse. Two parallel approvals leave both documents
APPROVEDbut the fellowship atDOCUMENTS_IN_REVIEW. That state is terminal —reviewDocumentrejects any document not inPENDING_REVIEW, so no admin action can re-trigger the sync andstart-contractfails forever with "Both fellowship documents must be approved…". Only a manual DB write clears it.Fix
Take a pessimistic write lock on the fellowship row as the first statement of the transaction, before the document write, in both
uploadDocumentandreviewDocument.fellowship.applicationIdisUNIQUE, so the fellowship row is a valid mutex for its application's documents.This blocks rather than aborts — the second request waits for the first to commit, then proceeds against committed data. Both succeed; nothing is rolled back. Measured contention is single-digit milliseconds, since the Google Drive upload happens before the transaction opens and never holds the lock.
Rejected alternatives:
SERIALIZABLEwould abort the loser with40001, andDbTransactionServicehas no retry loop, so it would surface as a 500.@VersionColumnwouldn't fire at all here — neither transaction writes the fellowship in the racing case, so there's no version bump to conflict on.Also in this PR:
syncFellowshipStatustakes the locked entity instead of re-reading it, making the precondition structural.save()(whole entity) toupdate(…, { status }), so it cannot clobberstartDate/endDate/amountUsd.startContractwas an unlocked, non-transactional read-modify-write; now wrapped in the same lock with a post-lock status re-check.Migration
1785456000000-migrations.tsrecomputes status for fellowships in the document phase, repairing existing stuck rows in both directions. The status filter mirrorsDOCUMENT_PHASE_STATUSES, soPENDING/ACTIVE/COMPLETEDrows are never touched. It is idempotent and produces no schema delta.Verified against 9 seeded scenarios — it repaired the 3 broken ones (the incident, the terminal case, and a rejected-document pullback), left healthy rows alone, and left all out-of-phase rows untouched even with complete documents.
Notes for the reviewer
jest-e2e.config.tspoints at ane2e/directory that doesn't exist, and a real Postgres is required — mocks can't reproduce MVCC snapshot visibility, so a mock-based "concurrency test" would pass on the broken code and prove nothing. I verified with throwaway harnesses driving the real service against Postgres, running each scenario with and without the fix: both races reproduce onmainand converge correctly with this change. Standing up the e2e harness is worth a follow-up.resolveDocumentreads the document outside the transaction, so two concurrent reviews of the same document both pass thePENDING_REVIEWguard and the loser silently overwrites the winner (e.g. a reject clobbering an approve). It doesn't cause stuck statuses, so it's out of scope; the remedy is re-reading and re-checking the document under the lock.lock_timeoutis configured anywhere in the repo, so a lock wait is unbounded. Fine given millisecond hold times, but a hung transaction would stall subsequent requests for the same fellowship rather than failing fast.🤖 Generated with Claude Code