Fix the races on chunk.FreeIds and chunk.ScanIds - #291
Open
midwell wants to merge 1 commit into
Open
Conversation
The scan goroutine started by claimChunk walks and mutates a chunk's id slices
with no lock, while the two API entry points mutate the same slices under the
package-level mutex:
- scan.go seeds ScanIds, pops from it, appends to FreeIds and writes AllocIds;
- AllocateIntID pops FreeIds and ReleaseIntID appends to it, and when the chunk
is still Scanning ReleaseIntID also walks ScanIds to remove the id
(chunk.go:83-94), both called with mutex held.
This is reachable by design rather than by accident. ReleaseInt32ID looks the
chunk up in scanChunks specifically so that an id belonging to a chunk that is
still being scanned can be released (api.go:104), which is exactly when the two
sides touch the same slices.
A torn slice header here does not abort the process the way a map write does. It
silently loses or duplicates an entry in FreeIds, so the pool can hand out an
identifier that is still in use - for the AMF a 5G-TMSI, for the SMF an FSEID.
The fields already have a lock; the scan goroutine just never took it. The three
accesses move behind helpers next to the existing FreeIds logic in chunk.go, and
the loop is restructured so that resourceValidCb is still called outside the
lock: pop under it, consult the callback, then record the result under it. The
callback is supplied by the NF and may re-enter drsm, which would deadlock on a
non-reentrant mutex.
appendScanIds builds the slice before taking the lock, so the critical section
covers the append only, and it appends rather than assigns to keep the previous
behaviour for a chunk that is rescanned.
Note on the test: unlike the chunk.Owner assertion, this one relies on the race
detector - the invariants it can check without it (an id is never handed out
twice, ScanIds always drains) still hold under the unguarded version. With the
locks removed it reports eight or more races per run. This repo's CI runs
go test without -race, so in CI the test only exercises the paths; the
pre-commit hook's `go test -race ./... -count=1` is what makes it a guard. Happy
to add `test_flags: -race` to the unit-tests job in a separate change if you
want that closed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edvin Lindqvist <edvin.lindqvist@forsway.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The race
The scan goroutine that
claimChunkstarts walks and mutates a chunk's id slices with nolock, while the two API entry points mutate the same slices under the package-level
mutex:scan.goseedsScanIds, pops from it, appends toFreeIdsand writesAllocIds;AllocateIntIDpopsFreeIds, andReleaseIntIDappends to it — and while the chunk isstill
Scanning,ReleaseIntIDalso walksScanIdsto remove the id(
chunk.go:83-94). Both are called withmutexheld.This one is reachable by design, not by accident.
ReleaseInt32IDlooks the chunk up inscanChunksspecifically so that an id belonging to a chunk still being scanned can bereleased (
api.go:104) — which is precisely when the two sides touch the same slices.Why it matters
A torn slice header here does not abort the process the way a map write does. It silently
loses or duplicates an entry in
FreeIds, so the pool can hand out an identifier that isstill in use — a 5G-TMSI for the AMF, an FSEID for the SMF. That is the quiet failure mode,
and the one that will not announce itself.
The fix
The fields already have a lock; the scan goroutine simply never took it. The three accesses
move behind helpers next to the existing
FreeIdslogic inchunk.go, and the tick body isrestructured so
resourceValidCbis still called outside the lock — pop under it,consult the callback, record the result under it. The callback is supplied by the NF and may
re-enter drsm, which would deadlock on a non-reentrant mutex.
appendScanIdsbuilds the slice before taking the lock, so the critical section covers onlythe append, and it appends rather than assigns so a rescanned chunk behaves as before.
Testing
drsm/scan_test.godrives the scan goroutine's handling of the three fields againstAllocateIntIDandReleaseIntID, called withmutexheld exactly asapi.gocalls them,which is what makes the scan side the unsynchronised one.
Being straight about its limits: unlike the
chunk.Ownerassertion in #290, this testrelies on the race detector. The invariants it can check without it — an id is never handed
out twice,
ScanIdsalways drains — still hold under the unguarded version. With the locksremoved it reports eight or more races per run; without
-raceit passes either way. Sincethis repo's CI runs
go testwith notest_flags, in CI the test only exercises the paths,and it is the pre-commit hook's
go test -race ./... -count=1that makes it a guard. Happyto add
test_flags: -raceto the unit-tests job as a separate change if you want that gapclosed.
A throwaway harness (not included — it needs a MongoDB replica set) also confirmed it live
against mongo 7.0 with the full goroutine set and a real pod-down, releasing ids belonging to
chunks still being scanned:
One methodological note, in case it is useful to anyone reproducing this: a first attempt
reported zero races because the releases ran in a tight loop that finished in
microseconds, while
scanChunk's ticker only fires every 5s — there was almost no overlap.Spreading the same releases across the run surfaced them immediately. A clean run here means
little unless you can show the probe actually overlapped the window.
Relationship to #289 and #290
All three are branched from
mainand independently reviewable. They touch neighbouringlines, so there are exactly two one-hunk conflicts across the set, both mechanical:
updates.gocp.setOwner(...)from #290,if !d.recordChunkOwner(...)from #289scan.god.startScan(c)from #289,c.appendScanIds(1000)from hereI have merged all three locally: it builds, vets clean, all 7 tests across the three test
files pass together under
-race(the test files are named and scoped so they share noidentifiers), and the live harness reports zero races on the union. Happy to rebase in
whatever order suits you, or to fold all three into a single PR if three is more than you
want to carry — just say which.
Not in this change
Between the pop and the record, an id belongs to no list, so a release arriving in that
window is appended to
FreeIdsbyReleaseIntIDand then appended again by the scan step.That window predates this change — the existing code already popped, called the callback and
then appended — and the locking does not widen it, so it is left exactly as it was rather
than fixed here.