Fix concurrent map access on the shared drsm tables - #289
Open
midwell wants to merge 1 commit into
Open
Conversation
An AMF or SMF pod aborts with an unrecoverable "fatal error: concurrent map
writes" when a peer pod goes down during a rolling update. The stack is entirely
inside drsm:
runtime.mapassign_fast32
drsm.(*chunk).scanChunk drsm/scan.go:23
drsm.(*chunk).claimChunk.gowrap1
podDownDetected starts one claimChunk goroutine per reclaimed chunk, and each
successful claim starts a scanChunk goroutine that writes d.scanChunks. A pod
that owned two or more chunks therefore races itself. Drsm declares a single
mutex, globalChunkTblMutex, and it guards only globalChunkTbl.
scanChunks is not the only unguarded table, so this covers all four:
- scanChunks written in scan.go, read in ReleaseInt32ID
- localChunkTbl written in scan.go, ranged in AllocateInt32ID
- podMap written by addPod from both the change stream and the
checkAllChunks ticker, read by podDownDetected
- podChunks written by addChunk from those same two goroutines, ranged by
podDownDetected
localChunkTbl matters because scan.go writes it on the line after the reported
crash, so guarding only scanChunks moves the abort to "concurrent map iteration
and map write". podMap needs no pod-down event at all: addChunk reaches addPod
from the change-stream goroutine and from the 3-second checkAllChunks ticker, so
two goroutines insert into it concurrently in steady state.
The two scan tables already have a lock - the package-level mutex held by
AllocateInt32ID and ReleaseInt32ID - and the scan goroutines simply did not take
it, so startScan and completeScan now do. The critical sections cover only the
map operations; resourceValidCb is still invoked without the lock held, so a
callback that re-enters drsm cannot deadlock.
podMap and the podChunks reachable through it get a new podMapMutex, and every
access moves behind a helper that holds it. No path holds two locks at once:
addChunk releases podMapMutex before taking globalChunkTblMutex, and the delete
handler signals podDown after releasing it, since podDownDetected acquires the
same lock.
Two behaviour changes fall out of the refactor:
- podDownDetected dereferenced d.podMap[p] without checking it was present,
which would panic for an unknown pod. podChunkIds returns no ids instead.
- addChunk logs podChunks before writing globalChunkTbl rather than after,
because that log has to read the map under the lock.
drsm had no tests. drsm_test.go adds three that drive the real call paths with
no MongoDB connection; the two concurrency tests report 10 to 20 data races per
run with the new locks removed, and pass with them in place.
A separate write/write race on chunk.Owner is left alone here: claimChunk sets
it on a successful claim while the change-stream handler sets the same fields
from the update that claim triggered. It is a different defect from the tables
and gets its own change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edvin Lindqvist <edvin.lindqvist@forsway.com>
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The crash
An AMF or SMF pod aborts with an unrecoverable
fatal error: concurrent map writeswhen apeer pod goes down during a rolling update. The stack is entirely inside
drsm:podDownDetectedstarts oneclaimChunkgoroutine per reclaimed chunk, and each successfulclaim starts a
scanChunkgoroutine that writesd.scanChunks. A departing pod that ownedtwo or more chunks therefore races itself.
Drsmdeclares a single mutex,globalChunkTblMutex, and it guards onlyglobalChunkTbl.Observed on a single-node cluster during a routine
kubectl set image; the replacement podstarted, detected the outgoing one, and aborted a second later.
Root cause
scanChunksis not the only unguarded table onDrsm:scanChunksscan.go:23,scan.go:52ReleaseInt32ID(api.go:104)localChunkTblscan.go:51,GetNewChunkAllocateInt32ID(ranged),ReleaseInt32IDscan.go:51podMapaddPodpodDownDetected, change-stream handlerpodChunksaddChunk, change-stream handlerpodDownDetected(ranged)globalChunkTblTwo of these matter more than the reported one:
localChunkTblis written atscan.go:51, the line after the reported crash, and isranged by
AllocateInt32ID. Guarding onlyscanChunksmoves the abort toconcurrent map iteration and map write.podMapneeds no pod-down event at all.addChunkreachesaddPodfrom thechange-stream goroutine and from the 3-second
checkAllChunksticker, so two goroutinesinsert into it concurrently in steady state. This is a second, independent crash path on a
timer rather than on a rollout.
The fix
The two scan tables already have a lock — the package-level
mutexheld byAllocateInt32IDandReleaseInt32ID— and the scan goroutines simply never took it, sostartScanandcompleteScannow do.podMapand thepodChunksreachable through it geta new
podMapMutex, with every access moved behind a helper that holds it.Deliberate properties, since this is a locking change:
addChunkreleasespodMapMutexbefore takingglobalChunkTblMutex, and the delete handler signalspodDownonly after releasing it,because
podDownDetectedacquires the same lock. So there is no lock order to get wrong.resourceValidCbis still invoked with no lock held. The critical sections cover onlythe map operations, so an NF callback that re-enters drsm cannot deadlock.
AllocateInt32IDalready holds the package-levelmutexacross a MongoDB insert insideGetNewChunk's retry loop, sostartScan/completeScancan now wait on a chunkallocation. That is a bounded delay to a background scan goroutine, not a deadlock —
MongoDB does not re-enter drsm. Measured cost below.
Two behaviour changes fall out of the refactor:
podDownDetecteddereferencedd.podMap[p]without checking presence, which would panicfor an unknown pod.
podChunkIdsreturns no ids instead.addChunklogspodChunksbefore writingglobalChunkTblrather than after, because thatlog has to read the map under the lock.
Testing
drsmhad no tests.drsm/drsm_test.goadds three that drive the real call paths with noMongoDB connection. They are mutation-verified rather than merely green: with the new locks
removed they report 10–20 data races per run across three runs, and they pass with the locks
in place.
Separately, a throwaway harness (not included here — it needs a MongoDB replica set, which
CI has no service for) ran
InitDRSMagainst mongo 7.0 with the full goroutine set, a peerpod owning 8 chunks, a real keepalive deletion to trigger the pod-down path, and 8 goroutines
hammering allocate/release. Before and after, under
-race:scan.go:23 ↔ scan.go:23(the line in the stack above),updates.go:204 ↔ updates.go:289(podChunks), and thechunk.Ownerpair belowchunk.Ownerpair onlyLiveness held: no stall, all 8 peer chunks reclaimed, and 117M allocate/release round-trips
in 45s without
-race, so the added acquisitions are not a throughput concern.Worth stating plainly: the
fatal erroritself did not reproduce in three pre-fix runswithout
-race. The window is nanoseconds wide and the claim goroutines arrive staggered byMongoDB round-trips, which matches the field observation that not every rollout crashes. The
race detector, not a clean run, is what demonstrates this.
Not in this change
A write/write race on
chunk.Ownerremains, and predates this change:claimChunksetsc.Owner.PodName/PodIpon a successful claim (claim.go:46-47) while the change-streamhandler sets the same fields (
updates.go:180-181) from the very update that claimtriggered.
c.Owneris also read byFindOwnerInt32IDviaGetOwner(), which returns&c.OwnerunderglobalChunkTblMutex— a lock neither writer takes. The visible effect is atorn
PodId, soFindOwnerInt32IDcan pair aPodNamewith another pod'sPodIp. It is adifferent defect from the tables and I would rather send it as its own change than widen this
one; happy to fold it in if you prefer.
Compatibility
No
go.mod/go.sumchange, and the exported surface is unchanged (diffed againstv1.8.4,17 symbols), so
DrsmInterfaceand everything consumers mock against are untouched.