Drop SKIP LOCKED and make dispatch ids unique in index queue#438
Open
mcop1 wants to merge 1 commit into
Open
Conversation
"FOR UPDATE SKIP LOCKED" is not supported on all databases Pimcore runs on (e.g. MariaDB < 10.6), causing the index queue to fail. The locking is not required: concurrent claiming is handled by stamping rows with a dispatch id in dispatchItems(), and the non-dispatching read is a pure peek. Both queue queries are therefore plain SELECTs now. Additionally, dispatch ids were derived from a millisecond timestamp, so two workers dispatching within the same millisecond received the same id and re-fetched each other's rows. The id now keeps the timestamp in the high-order digits and adds a random suffix in the low-order digits, guaranteeing uniqueness while preserving the stale-item re-dispatch. Resolves #357 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Changes in this pull request
Resolves #357
generic-data-index-bundleusedSELECT ... FOR UPDATE SKIP LOCKEDin the index queue, butSKIP LOCKEDis not available on every database Pimcore supports (notably MariaDB < 10.6, where it raises a syntax error). Raising the minimum DB version is a BC break that does not belong in a minor/bugfix release, so this PR removes the dependency instead.1. Drop
SKIP LOCKED(both queue queries are now plainSELECTs)dispatchItems(), which atomically stamps rows with a dispatch id viaUPDATE ... SET dispatched = :id. The follow-up read only ever fetches the worker's own, exclusively-owned rows.dispatched = 0rows without claiming), so it neither needs nor should hold row locks.2. Make dispatch ids unique (
dispatchItems())SELECT WHERE dispatched = :idreturned the union of both workers' rows → double processing.timestamp * 1_000_000 + random_int(0, 999_999)), guaranteeing uniqueness within a millisecond while preserving the chronological ordering the 24h stale-item re-dispatch relies on. The value stays well within thebigint unsignedrange of thedispatchedcolumn.Additional info
dispatchedcolumn.getUnhandledIndexQueueEntries()anddispatchItems()are unchanged.usleep(1ms)workaround (it existed only to dodge the same-millisecond id collision this PR fixes), so its removal now actively asserts the fix.