Skip to content

fix: serialize fellowship document-phase status recomputation - #89

Merged
theanmolsharma merged 1 commit into
mainfrom
fellowship-stuck-under-review
Jul 31, 2026
Merged

fix: serialize fellowship document-phase status recomputation#89
theanmolsharma merged 1 commit into
mainfrom
fellowship-stuck-under-review

Conversation

@theanmolsharma

Copy link
Copy Markdown
Collaborator

Problem

A fellowship was found with both fellow documents (SIGNED_CONTRACT, W8BEN) at PENDING_REVIEW while the fellowship itself was still AWAITING_DOCUMENTS. The fellow had submitted everything; the status never advanced.

syncFellowshipStatus is the only writer of document-phase fellowship status, and it did an unlocked read-compute-write inside a READ COMMITTED transaction: save the document → read its sibling → write the fellowship only if the computed status differed.

When the fellow's UI uploaded both PDFs in parallel:

Tx A (signed contract) Tx B (W8-BEN)
1 save signed = PENDING_REVIEW save w8ben = PENDING_REVIEW
2 reads sibling → AWAITING_UPLOAD (B uncommitted) reads sibling → AWAITING_UPLOAD (A uncommitted)
3 computes AWAITING_DOCUMENTS = current → no write same → no write

Both documents land at PENDING_REVIEW; the fellowship stays AWAITING_DOCUMENTS. It also drops out of the ?status=DOCUMENTS_IN_REVIEW admin queue, so nobody sees it needs review, and both no-op branches logged nothing.

The mirror case is worse. Two parallel approvals leave both documents APPROVED but the fellowship at DOCUMENTS_IN_REVIEW. That state is terminalreviewDocument rejects any document not in PENDING_REVIEW, so no admin action can re-trigger the sync and start-contract fails forever with "Both fellowship documents must be approved…". Only a manual DB write clears it.

Fix

Take a pessimistic write lock on the fellowship row as the first statement of the transaction, before the document write, in both uploadDocument and reviewDocument. fellowship.applicationId is UNIQUE, so the fellowship row is a valid mutex for its application's documents.

This blocks rather than aborts — the second request waits for the first to commit, then proceeds against committed data. Both succeed; nothing is rolled back. Measured contention is single-digit milliseconds, since the Google Drive upload happens before the transaction opens and never holds the lock.

Rejected alternatives: SERIALIZABLE would abort the loser with 40001, and DbTransactionService has no retry loop, so it would surface as a 500. @VersionColumn wouldn't fire at all here — neither transaction writes the fellowship in the racing case, so there's no version bump to conflict on.

Also in this PR:

  • syncFellowshipStatus takes the locked entity instead of re-reading it, making the precondition structural.
  • Status write narrowed from save() (whole entity) to update(…, { status }), so it cannot clobber startDate / endDate / amountUsd.
  • startContract was an unlocked, non-transactional read-modify-write; now wrapped in the same lock with a post-lock status re-check.
  • Both previously-silent branches now log, including the document status vector, so a recurrence is greppable.

Migration

1785456000000-migrations.ts recomputes status for fellowships in the document phase, repairing existing stuck rows in both directions. The status filter mirrors DOCUMENT_PHASE_STATUSES, so PENDING / ACTIVE / COMPLETED rows are never touched. It is idempotent and produces no schema delta.

Verified against 9 seeded scenarios — it repaired the 3 broken ones (the incident, the terminal case, and a rejected-document pullback), left healthy rows alone, and left all out-of-phase rows untouched even with complete documents.

Notes for the reviewer

  • No tests are included. jest-e2e.config.ts points at an e2e/ directory that doesn't exist, and a real Postgres is required — mocks can't reproduce MVCC snapshot visibility, so a mock-based "concurrency test" would pass on the broken code and prove nothing. I verified with throwaway harnesses driving the real service against Postgres, running each scenario with and without the fix: both races reproduce on main and converge correctly with this change. Standing up the e2e harness is worth a follow-up.
  • A separate race is not fixed here. resolveDocument reads the document outside the transaction, so two concurrent reviews of the same document both pass the PENDING_REVIEW guard and the loser silently overwrites the winner (e.g. a reject clobbering an approve). It doesn't cause stuck statuses, so it's out of scope; the remedy is re-reading and re-checking the document under the lock.
  • No lock_timeout is configured anywhere in the repo, so a lock wait is unbounded. Fine given millisecond hold times, but a hung transaction would stall subsequent requests for the same fellowship rather than failing fast.

🤖 Generated with Claude Code

syncFellowshipStatus did an unlocked read-compute-write inside a READ
COMMITTED transaction: save the document, read its sibling, then write the
fellowship only if the computed status differed. Two concurrent document
mutations each read the other's pre-commit row, both concluded "nothing
changed", and neither wrote.

Parallel uploads left both documents PENDING_REVIEW with the fellowship stuck
at AWAITING_DOCUMENTS, which also drops it out of the
?status=DOCUMENTS_IN_REVIEW admin queue. Parallel approvals left both
documents APPROVED with the fellowship stuck at DOCUMENTS_IN_REVIEW — that
one is terminal, since reviewDocument rejects any document not in
PENDING_REVIEW, so no admin action could re-trigger a sync and start-contract
failed permanently.

Take a pessimistic write lock on the fellowship row as the first statement of
the transaction, before the document write. fellowship.applicationId is
UNIQUE, so the fellowship row is a valid mutex for its application's
documents. The loser blocks and then succeeds; nothing is aborted.

Also:
- syncFellowshipStatus takes the locked entity rather than re-reading it, so
  the precondition is structural.
- Narrow the status write from save() to update(), which cannot clobber
  startDate/endDate/amountUsd.
- startContract was an unlocked, non-transactional read-modify-write; wrap it
  in the same lock and re-check the status after acquiring it.
- Log both branches that previously returned silently.
- Add a migration recomputing status for fellowships in the document phase,
  repairing existing stuck rows in both directions. PENDING, ACTIVE and
  COMPLETED rows are never touched.

Co-Authored-By: Claude <noreply@anthropic.com>
@theanmolsharma
theanmolsharma merged commit 892330d into main Jul 31, 2026
2 checks passed
@theanmolsharma
theanmolsharma deleted the fellowship-stuck-under-review branch July 31, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant