Skip to content

Drop SKIP LOCKED and make dispatch ids unique in index queue#438

Open
mcop1 wants to merge 1 commit into
2.5from
bugfix/357-skip-locked-mariadb-compat
Open

Drop SKIP LOCKED and make dispatch ids unique in index queue#438
mcop1 wants to merge 1 commit into
2.5from
bugfix/357-skip-locked-mariadb-compat

Conversation

@mcop1

@mcop1 mcop1 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Changes in this pull request

Resolves #357

generic-data-index-bundle used SELECT ... FOR UPDATE SKIP LOCKED in the index queue, but SKIP LOCKED is not available on every database Pimcore supports (notably MariaDB < 10.6, where it raises a syntax error). Raising the minimum DB version is a BC break that does not belong in a minor/bugfix release, so this PR removes the dependency instead.

1. Drop SKIP LOCKED (both queue queries are now plain SELECTs)

  • The locking was not actually needed. Concurrent claiming is already handled by dispatchItems(), which atomically stamps rows with a dispatch id via UPDATE ... SET dispatched = :id. The follow-up read only ever fetches the worker's own, exclusively-owned rows.
  • The non-dispatching path is a pure "peek" (read of dispatched = 0 rows without claiming), so it neither needs nor should hold row locks.
  • Result: portable to all supported databases, no version bump, no DDL.

2. Make dispatch ids unique (dispatchItems())

  • Previously the dispatch id was a plain millisecond timestamp. Two workers dispatching within the same millisecond received the same id, so each SELECT WHERE dispatched = :id returned the union of both workers' rows → double processing.
  • The id now keeps the millisecond timestamp in the high-order digits and adds a random suffix in the low-order digits (timestamp * 1_000_000 + random_int(0, 999_999)), guaranteeing uniqueness within a millisecond while preserving the chronological ordering the 24h stale-item re-dispatch relies on. The value stays well within the bigint unsigned range of the dispatched column.

Additional info

  • No schema change / no migration — the fix reuses the existing dispatched column.
  • Backwards compatible — public signatures and behaviors of getUnhandledIndexQueueEntries() and dispatchItems() are unchanged.
  • The functional test no longer needs its usleep(1ms) workaround (it existed only to dodge the same-millisecond id collision this PR fixes), so its removal now actively asserts the fix.
  • ⚠️ Worth validating on a large-scale, highly concurrent dataset (millions of rows / many workers): confirm no row is processed twice and none is stranded. The random suffix gives 1,000,000 slots per millisecond, so collisions are astronomically unlikely but not mathematically zero.

"FOR UPDATE SKIP LOCKED" is not supported on all databases Pimcore runs
on (e.g. MariaDB < 10.6), causing the index queue to fail. The locking
is not required: concurrent claiming is handled by stamping rows with a
dispatch id in dispatchItems(), and the non-dispatching read is a pure
peek. Both queue queries are therefore plain SELECTs now.

Additionally, dispatch ids were derived from a millisecond timestamp,
so two workers dispatching within the same millisecond received the same
id and re-fetched each other's rows. The id now keeps the timestamp in
the high-order digits and adds a random suffix in the low-order digits,
guaranteeing uniqueness while preserving the stale-item re-dispatch.

Resolves #357

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@mcop1 mcop1 self-assigned this Jun 19, 2026
@mcop1 mcop1 added the Bug label Jun 19, 2026
@mcop1 mcop1 added this to the 2.5.4 milestone Jun 19, 2026
@pimcore-deployments pimcore-deployments modified the milestones: 2.5.4, 2.5.5 Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants