Skip to content

Fix the races on chunk.FreeIds and chunk.ScanIds - #291

Open
midwell wants to merge 1 commit into
omec-project:mainfrom
midwell:fix-drsm-chunk-slice-races
Open

Fix the races on chunk.FreeIds and chunk.ScanIds#291
midwell wants to merge 1 commit into
omec-project:mainfrom
midwell:fix-drsm-chunk-slice-races

Conversation

@midwell

@midwell midwell commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

The race

The scan goroutine that claimChunk starts walks and mutates a chunk's id slices with no
lock, while the two API entry points mutate the same slices under the package-level mutex:

  • scan.go seeds ScanIds, pops from it, appends to FreeIds and writes AllocIds;
  • AllocateIntID pops FreeIds, and ReleaseIntID appends to it — and while the chunk is
    still Scanning, ReleaseIntID also walks ScanIds to remove the id
    (chunk.go:83-94). Both are called with mutex held.

This one is reachable by design, not by accident. ReleaseInt32ID looks the chunk up in
scanChunks specifically so that an id belonging to a chunk still being scanned can be
released (api.go:104) — which is precisely when the two sides touch the same slices.

Why it matters

A torn slice header here does not abort the process the way a map write does. It silently
loses or duplicates an entry in FreeIds, so the pool can hand out an identifier that is
still in use — a 5G-TMSI for the AMF, an FSEID for the SMF. That is the quiet failure mode,
and the one that will not announce itself.

The fix

The fields already have a lock; the scan goroutine simply never took it. The three accesses
move behind helpers next to the existing FreeIds logic in chunk.go, and the tick body is
restructured so resourceValidCb is still called outside the lock — pop under it,
consult the callback, record the result under it. The callback is supplied by the NF and may
re-enter drsm, which would deadlock on a non-reentrant mutex.

appendScanIds builds the slice before taking the lock, so the critical section covers only
the append, and it appends rather than assigns so a rescanned chunk behaves as before.

Testing

drsm/scan_test.go drives the scan goroutine's handling of the three fields against
AllocateIntID and ReleaseIntID, called with mutex held exactly as api.go calls them,
which is what makes the scan side the unsynchronised one.

Being straight about its limits: unlike the chunk.Owner assertion in #290, this test
relies on the race detector.
The invariants it can check without it — an id is never handed
out twice, ScanIds always drains — still hold under the unguarded version. With the locks
removed it reports eight or more races per run; without -race it passes either way. Since
this repo's CI runs go test with no test_flags, in CI the test only exercises the paths,
and it is the pre-commit hook's go test -race ./... -count=1 that makes it a guard. Happy
to add test_flags: -race to the unit-tests job as a separate change if you want that gap
closed.

A throwaway harness (not included — it needs a MongoDB replica set) also confirmed it live
against mongo 7.0 with the full goroutine set and a real pod-down, releasing ids belonging to
chunks still being scanned:

before:  Read chunk.go:112 <-> prev write scan.go:43     (FreeIds)
         Read chunk.go:120 <-> prev write scan.go:39     (ScanIds)
after:   no races, 3200 releases landed, 0 unknown ids, all 8 peer chunks reclaimed

One methodological note, in case it is useful to anyone reproducing this: a first attempt
reported zero races because the releases ran in a tight loop that finished in
microseconds, while scanChunk's ticker only fires every 5s — there was almost no overlap.
Spreading the same releases across the run surfaced them immediately. A clean run here means
little unless you can show the probe actually overlapped the window.

Relationship to #289 and #290

All three are branched from main and independently reviewable. They touch neighbouring
lines, so there are exactly two one-hunk conflicts across the set, both mechanical:

merge conflict resolution
#289 + #290 updates.go take cp.setOwner(...) from #290, if !d.recordChunkOwner(...) from #289
that + this scan.go take d.startScan(c) from #289, c.appendScanIds(1000) from here

I have merged all three locally: it builds, vets clean, all 7 tests across the three test
files pass together under -race (the test files are named and scoped so they share no
identifiers), and the live harness reports zero races on the union. Happy to rebase in
whatever order suits you, or to fold all three into a single PR if three is more than you
want to carry — just say which.

Not in this change

Between the pop and the record, an id belongs to no list, so a release arriving in that
window is appended to FreeIds by ReleaseIntID and then appended again by the scan step.
That window predates this change — the existing code already popped, called the callback and
then appended — and the locking does not widen it, so it is left exactly as it was rather
than fixed here.

The scan goroutine started by claimChunk walks and mutates a chunk's id slices
with no lock, while the two API entry points mutate the same slices under the
package-level mutex:

  - scan.go seeds ScanIds, pops from it, appends to FreeIds and writes AllocIds;
  - AllocateIntID pops FreeIds and ReleaseIntID appends to it, and when the chunk
    is still Scanning ReleaseIntID also walks ScanIds to remove the id
    (chunk.go:83-94), both called with mutex held.

This is reachable by design rather than by accident. ReleaseInt32ID looks the
chunk up in scanChunks specifically so that an id belonging to a chunk that is
still being scanned can be released (api.go:104), which is exactly when the two
sides touch the same slices.

A torn slice header here does not abort the process the way a map write does. It
silently loses or duplicates an entry in FreeIds, so the pool can hand out an
identifier that is still in use - for the AMF a 5G-TMSI, for the SMF an FSEID.

The fields already have a lock; the scan goroutine just never took it. The three
accesses move behind helpers next to the existing FreeIds logic in chunk.go, and
the loop is restructured so that resourceValidCb is still called outside the
lock: pop under it, consult the callback, then record the result under it. The
callback is supplied by the NF and may re-enter drsm, which would deadlock on a
non-reentrant mutex.

appendScanIds builds the slice before taking the lock, so the critical section
covers the append only, and it appends rather than assigns to keep the previous
behaviour for a chunk that is rescanned.

Note on the test: unlike the chunk.Owner assertion, this one relies on the race
detector - the invariants it can check without it (an id is never handed out
twice, ScanIds always drains) still hold under the unguarded version. With the
locks removed it reports eight or more races per run. This repo's CI runs
go test without -race, so in CI the test only exercises the paths; the
pre-commit hook's `go test -race ./... -count=1` is what makes it a guard. Happy
to add `test_flags: -race` to the unit-tests job in a separate change if you
want that closed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edvin Lindqvist <edvin.lindqvist@forsway.com>
@midwell
midwell requested a review from a team August 20, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant