Skip to content

Fix concurrent map access on the shared drsm tables - #289

Open
midwell wants to merge 1 commit into
omec-project:mainfrom
midwell:fix-drsm-map-races
Open

Fix concurrent map access on the shared drsm tables#289
midwell wants to merge 1 commit into
omec-project:mainfrom
midwell:fix-drsm-map-races

Conversation

@midwell

@midwell midwell commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

The crash

An AMF or SMF pod aborts with an unrecoverable fatal error: concurrent map writes when a
peer pod goes down during a rolling update. The stack is entirely inside drsm:

fatal error: concurrent map writes

goroutine 1266 [running]:
runtime.mapassign_fast32(...)
github.com/omec-project/util/drsm.(*chunk).scanChunk(...)
	util@v1.8.4/drsm/scan.go:23 +0x98
github.com/omec-project/util/drsm.(*chunk).claimChunk.gowrap1()
	util@v1.8.4/drsm/claim.go:48 +0x25
created by github.com/omec-project/util/drsm.(*chunk).claimChunk in goroutine 894

podDownDetected starts one claimChunk goroutine per reclaimed chunk, and each successful
claim starts a scanChunk goroutine that writes d.scanChunks. A departing pod that owned
two or more chunks therefore races itself. Drsm declares a single mutex,
globalChunkTblMutex, and it guards only globalChunkTbl.

Observed on a single-node cluster during a routine kubectl set image; the replacement pod
started, detected the outgoing one, and aborted a second later.

Root cause

scanChunks is not the only unguarded table on Drsm:

Table Written by Read by Guarded before
scanChunks scan.go:23, scan.go:52 ReleaseInt32ID (api.go:104) no
localChunkTbl scan.go:51, GetNewChunk AllocateInt32ID (ranged), ReleaseInt32ID partly — every site except scan.go:51
podMap addPod podDownDetected, change-stream handler no
podChunks addChunk, change-stream handler podDownDetected (ranged) no
globalChunkTbl yes, correctly

Two of these matter more than the reported one:

  • localChunkTbl is written at scan.go:51, the line after the reported crash, and is
    ranged by AllocateInt32ID. Guarding only scanChunks moves the abort to
    concurrent map iteration and map write.
  • podMap needs no pod-down event at all. addChunk reaches addPod from the
    change-stream goroutine and from the 3-second checkAllChunks ticker, so two goroutines
    insert into it concurrently in steady state. This is a second, independent crash path on a
    timer rather than on a rollout.

The fix

The two scan tables already have a lock — the package-level mutex held by
AllocateInt32ID and ReleaseInt32ID — and the scan goroutines simply never took it, so
startScan and completeScan now do. podMap and the podChunks reachable through it get
a new podMapMutex, with every access moved behind a helper that holds it.

Deliberate properties, since this is a locking change:

  • No path holds two locks at once. addChunk releases podMapMutex before taking
    globalChunkTblMutex, and the delete handler signals podDown only after releasing it,
    because podDownDetected acquires the same lock. So there is no lock order to get wrong.
  • resourceValidCb is still invoked with no lock held. The critical sections cover only
    the map operations, so an NF callback that re-enters drsm cannot deadlock.
  • AllocateInt32ID already holds the package-level mutex across a MongoDB insert inside
    GetNewChunk's retry loop, so startScan/completeScan can now wait on a chunk
    allocation. That is a bounded delay to a background scan goroutine, not a deadlock —
    MongoDB does not re-enter drsm. Measured cost below.

Two behaviour changes fall out of the refactor:

  • podDownDetected dereferenced d.podMap[p] without checking presence, which would panic
    for an unknown pod. podChunkIds returns no ids instead.
  • addChunk logs podChunks before writing globalChunkTbl rather than after, because that
    log has to read the map under the lock.

Testing

drsm had no tests. drsm/drsm_test.go adds three that drive the real call paths with no
MongoDB connection. They are mutation-verified rather than merely green: with the new locks
removed they report 10–20 data races per run across three runs, and they pass with the locks
in place.

Separately, a throwaway harness (not included here — it needs a MongoDB replica set, which
CI has no service for) ran InitDRSM against mongo 7.0 with the full goroutine set, a peer
pod owning 8 chunks, a real keepalive deletion to trigger the pod-down path, and 8 goroutines
hammering allocate/release. Before and after, under -race:

race pairs reported
before scan.go:23 ↔ scan.go:23 (the line in the stack above), updates.go:204 ↔ updates.go:289 (podChunks), and the chunk.Owner pair below
after the chunk.Owner pair only

Liveness held: no stall, all 8 peer chunks reclaimed, and 117M allocate/release round-trips
in 45s without -race, so the added acquisitions are not a throughput concern.

Worth stating plainly: the fatal error itself did not reproduce in three pre-fix runs
without -race. The window is nanoseconds wide and the claim goroutines arrive staggered by
MongoDB round-trips, which matches the field observation that not every rollout crashes. The
race detector, not a clean run, is what demonstrates this.

Not in this change

A write/write race on chunk.Owner remains, and predates this change: claimChunk sets
c.Owner.PodName/PodIp on a successful claim (claim.go:46-47) while the change-stream
handler sets the same fields (updates.go:180-181) from the very update that claim
triggered. c.Owner is also read by FindOwnerInt32ID via GetOwner(), which returns
&c.Owner under globalChunkTblMutex — a lock neither writer takes. The visible effect is a
torn PodId, so FindOwnerInt32ID can pair a PodName with another pod's PodIp. It is a
different defect from the tables and I would rather send it as its own change than widen this
one; happy to fold it in if you prefer.

Compatibility

No go.mod/go.sum change, and the exported surface is unchanged (diffed against v1.8.4,
17 symbols), so DrsmInterface and everything consumers mock against are untouched.

An AMF or SMF pod aborts with an unrecoverable "fatal error: concurrent map
writes" when a peer pod goes down during a rolling update. The stack is entirely
inside drsm:

    runtime.mapassign_fast32
    drsm.(*chunk).scanChunk         drsm/scan.go:23
    drsm.(*chunk).claimChunk.gowrap1

podDownDetected starts one claimChunk goroutine per reclaimed chunk, and each
successful claim starts a scanChunk goroutine that writes d.scanChunks. A pod
that owned two or more chunks therefore races itself. Drsm declares a single
mutex, globalChunkTblMutex, and it guards only globalChunkTbl.

scanChunks is not the only unguarded table, so this covers all four:

  - scanChunks    written in scan.go, read in ReleaseInt32ID
  - localChunkTbl written in scan.go, ranged in AllocateInt32ID
  - podMap        written by addPod from both the change stream and the
                  checkAllChunks ticker, read by podDownDetected
  - podChunks     written by addChunk from those same two goroutines, ranged by
                  podDownDetected

localChunkTbl matters because scan.go writes it on the line after the reported
crash, so guarding only scanChunks moves the abort to "concurrent map iteration
and map write". podMap needs no pod-down event at all: addChunk reaches addPod
from the change-stream goroutine and from the 3-second checkAllChunks ticker, so
two goroutines insert into it concurrently in steady state.

The two scan tables already have a lock - the package-level mutex held by
AllocateInt32ID and ReleaseInt32ID - and the scan goroutines simply did not take
it, so startScan and completeScan now do. The critical sections cover only the
map operations; resourceValidCb is still invoked without the lock held, so a
callback that re-enters drsm cannot deadlock.

podMap and the podChunks reachable through it get a new podMapMutex, and every
access moves behind a helper that holds it. No path holds two locks at once:
addChunk releases podMapMutex before taking globalChunkTblMutex, and the delete
handler signals podDown after releasing it, since podDownDetected acquires the
same lock.

Two behaviour changes fall out of the refactor:

  - podDownDetected dereferenced d.podMap[p] without checking it was present,
    which would panic for an unknown pod. podChunkIds returns no ids instead.
  - addChunk logs podChunks before writing globalChunkTbl rather than after,
    because that log has to read the map under the lock.

drsm had no tests. drsm_test.go adds three that drive the real call paths with
no MongoDB connection; the two concurrency tests report 10 to 20 data races per
run with the new locks removed, and pass with them in place.

A separate write/write race on chunk.Owner is left alone here: claimChunk sets
it on a successful claim while the change-stream handler sets the same fields
from the update that claim triggered. It is a different defect from the tables
and gets its own change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edvin Lindqvist <edvin.lindqvist@forsway.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant