Reduce server hitching: dead dispatcher config, per-packet allocation, and unindexed queries

[GeekOfWires]

What this is

A pass over the sources of lag spikes and hitching that connected clients experience. It
came out of a subsystem-by-subsystem read of the network layer, zone/event fan-out, actor
scheduling, GUID allocation and the persistence layer.

Three root causes account for most of it:

1. The dispatcher configuration never applied to anything. akka.conf assigns
   dispatchers to 30+ actor paths and dispatchers.conf defines them all carefully, but no
   actor occupies those paths, and config-based deployment does not apply to typed actors
   in any case. Socket I/O, session logic and every zone therefore shared one unconfigured
   default pool running at Akka's stock throughput = 5.
2. Hot paths copied whole collections. The blockmap copied every sector in the zone per
   spatial query and then indexed the copy as a linked list; sector groups eagerly built ten
   entity lists when callers wanted one; the GUID pools walked Lists of up to 16,000
   entries per allocation and release.
3. Nothing was indexed. CREATE INDEX appeared in exactly one of seventeen migrations.

Each commit is self-contained and explains its own reasoning; they can be reviewed or
reverted individually.

Changes by area

Threading and configuration

flowchart LR
  subgraph before["Before - deployment config matched no actor"]
    direction TB
    SA1["SocketActor"] --> DD["akka.actor.default-dispatcher<br/>unconfigured, throughput 5"]
    MW1["MiddlewareActor"] --> DD
    SS1["SessionActor"] --> DD
    LA1["LoginActor"] --> DD
    ZA1["ZoneActor, 32 of them"] --> DD
  end
  subgraph after["After - selected at each spawn site"]
    direction TB
    SA2["SocketActor"] --> NL["network-listener<br/>pinned, one thread per port"]
    MW2["MiddlewareActor"] --> WS["world-session"]
    SS2["SessionActor"] --> WS
    LA2["LoginActor"] --> LS["login-session"]
    ZA2["ZoneActor, 32 of them"] --> ZD["one pool per zone<br/>z1-z10, c1-c6, i1-i4, home, tz"]
  end

Loading

• Select dispatchers explicitly at each spawn site — socket actors onto the pinned
  network-listener, middleware and sessions onto world-session, login onto
  login-session, and each zone onto its own pool. The zone lookup is guarded with
  hasPath and falls back to the default pool with a warning rather than refusing to boot.
• That guard immediately found a bug: the TR sanctuary dropship zone is tzdrtr but its
  dispatcher was spelled tzsdrtr, so it had no pool. Every other zone matched exactly.
• Configure akka.actor.default-dispatcher (which everything had silently been using) and
  enlarge the scheduler wheel, which holds a per-session packet timer plus per-player and
  per-entity ticks and collides heavily at the default 512 buckets.
• Raise maxActiveConnections from 5. It caps concurrent SQL statements for the whole
  server, not player connections, and the driver queues rather than rejecting, so five slots
  showed up purely as latency.
• Give the packaged server an explicit heap and G1 with a pause target; it previously passed
  no memory or collector options at all. Align the compose JDK with the Dockerfile — it ran
  JDK 8, whose default ParallelGC does multi-second stop-the-world collections.

Per-packet cost

flowchart LR
  Q["outQueueBundled<br/>each bundle capped at the 440 byte MTU"] --> TICK{"bundler run<br/>every 50 ms"}
  TICK -->|"before: dequeue exactly 1"| B["about 20 bundles/s<br/>under 9 KB/s per client"]
  TICK -->|"after: drain up to 8"| A["about 160 bundles/s<br/>tunable per deployment"]

Loading

• Rewrite Md5Mac over primitive arrays. It ran on every packet in both directions while
  holding state in ArrayBuffer[Byte] (boxing every byte), reading four bytes at a known
  offset by allocating and walking an iterator ~128 times per block, and deep-copying its
  entire state on each reset() — twice per packet.
• Drain up to a configurable number of bundles per run of the packet bundler. It previously
  sent exactly one bundle per timer tick, capping every client near 9 KB/s regardless of
  backlog, which is why zone loads and base captures visibly trickled in.
• Hoist codecs and error strings out of the encode/decode path: MultiPacketEx.sizeCodec
  rebuilt its codec tree per sub-packet, CommonFieldData's two codecs were reconstructed
  per object in every ObjectCreateMessage, and the unimplemented-opcode path rendered the
  whole payload to hex eagerly because Err takes its message by value.

Zone and event fan-out

flowchart TB
  UP["PlayerStateMessageUpstream<br/>several per second, per player"] --> UB["SessionData.updateBlockMap"]
  UB --> SEC["blockMap.sector, 550 m draw range, about 144 sectors"]
  SEC --> OLD["Before<br/>copy every sector in the zone into a List<br/>then resolve each index, O index per lookup"]
  SEC --> NEW["After<br/>index an IndexedSeq directly"]
  OLD --> SG1["SectorGroup<br/>eagerly materialise all 10 entity lists"]
  NEW --> SG2["SectorGroup<br/>condense only the categories actually read"]

Loading

• Index blockmap sectors directly instead of copying the zone per query.
• Condense sector-group entity lists on demand rather than materialising all ten.
• Reuse the sector query in the SOI recalculation, which ran it twice per facility every
  five seconds.
• Make BundledEnvelope.unwind linear; it was O(n²) and ran twice because a non-elided
  assert repeated it.

GUID allocation

• Replace list walks with indexed access in GenericPool and ExclusivePool. The generic
  pool is the fallback for every exhausted pool and receives the whole hub, which made its
  scan quadratic at exactly the moment the server is under pressure.

Correctness bugs that present as performance problems

• TaskWorkflow synchronised on a boxed Boolean, i.e. the cached TRUE/FALSE
  singletons, so every task bundle server-wide contended one of two JVM-global monitors. It
  was also not a valid mutex: the monitor identity changed when the flag flipped, so two
  threads could complete the promise.
• The container move path sent two Resumes for one Wait on ask timeout, releasing an
  unrelated concurrent move's guard.
• A per-session thread pool was never shut down, and a fixed-rate task never cancelled.

Database

Measured on the real loadCampaignKdaData query against 500 avatars and a 200k-row kill log:

flowchart TB
  subgraph noidx["Without V018 - 10.29 ms, 3917 buffers"]
    direction TB
    N1["Gather, 2 parallel workers"] --> N2["Parallel Seq Scan on killactivity"]
    N2 --> N3["filters every row in the kill log<br/>cost grows with the table forever"]
  end
  subgraph idx["With V018 - 2.47 ms, 636 buffers"]
    direction TB
    I1["Hash Join"] --> I2["Bitmap Heap Scan on killactivity"]
    I2 --> I3["BitmapOr"]
    I3 --> I4["Bitmap Index Scan<br/>killactivity_killer_id_idx"]
    I3 --> I5["Bitmap Index Scan<br/>killactivity_victim_id_idx"]
  end

Loading

• V018__PerformanceIndexes.sql indexes the foreign keys read during login and character
  select. killactivity matters most: it is an append-only kill log read on every login, so
  login cost grew with every kill the server had ever recorded.

Unrelated but required

• LocalActionTest was missing a Default import, which broke Test/compile for the whole
  project on master. No test could run until this was fixed. Worth pulling out separately if
  you'd rather land it on its own.

Measured impact

Each old implementation was taken from psf/master and benchmarked against the new one in
the same JVM with trials interleaved. Temurin 17, G1, 16 logical CPUs.

Read these numbers with their limits in mind. This ran on a working desktop, not a quiet
bench host, so absolute figures drifted 10–25% between rounds; every ratio reproduced
within a few percent across three rounds including a cold JVM, so the ratios are the
trustworthy output. There is no JMH in this project, so these are careful manual benchmarks
— warmup, auto-calibrated iteration counts, median of nine trials, checksum accumulation to
defeat dead-code elimination — not JMH-grade measurements. They also measure components in
isolation; none of them establish end-to-end server behaviour under real load.

All seven changes measured faster. No regressions, no ambiguous results.

Packet path — Md5Mac

Runs on every inbound and outbound packet, so this is the per-packet floor.

xychart-beta
    title "Md5Mac reset + updateFinal, microseconds per MAC"
    x-axis ["440 B before", "440 B after", "64 B before", "64 B after"]
    y-axis "microseconds" 0 --> 45
    bar [39.5, 1.7, 18.9, 0.62]

Loading

payload	before	after	ratio
440 B (MTU-sized)	39.5 µs	1.70 µs	~23x
64 B	18.9 µs	0.62 µs	~30x

Old and new produce an identical MAC, verified in the harness as well as by the 6,000-case
differential run described under Verification.

Zone and movement path

The blockmap helper is called twice per movement packet per player; the sector group is
built once per movement packet.

xychart-beta
    title "Blockmap sector resolution, microseconds per lookup"
    x-axis ["before", "after"]
    y-axis "microseconds" 0 --> 700
    bar [642, 16.3]

Loading

xychart-beta
    title "SectorGroup construct then read livePlayerList.size, microseconds"
    x-axis ["before", "after"]
    y-axis "microseconds" 0 --> 30
    bar [26.0, 4.74]

Loading

case	before	after	ratio
blockmap resolution, 6,724-sector zone, 144 indices	642 µs	16.3 µs	~39x
sector group over 144 sectors	26.0 µs	4.74 µs	at least 5.7x

The sector-group figure is a lower bound, and deliberately so. The harness could only
populate the live-player category; the other nine were empty, yet the old constructor still
walked all 144 sectors for each of them. A real zone has buildings, amenities and
environment entries in nearly every sector, so the production gap is larger than 5.7x. Do
not quote it as an upper bound.

Event bundling

BundledEnvelope.unwind was O(n²) and the non-elided assert ran it a second time. The
interesting result is that the gap widens with bundle size, which is what an O(n²) to O(n)
change should look like — and bundles are built per recipient over the zone population.

xychart-beta
    title "BundledEnvelope speedup by bundle size (times faster)"
    x-axis ["10 envelopes", "50 envelopes", "200 envelopes"]
    y-axis "times faster" 0 --> 55
    bar [6.1, 16.5, 47.6]

Loading

bundle size	before	after	ratio
10	955 ns	156 ns	~6x
50	14.5 µs	883 ns	~16x
200	193 µs	4.06 µs	~48x

GUID allocation

xychart-beta
    title "ExclusivePool Get + Return, microseconds per operation"
    x-axis ["before", "after"]
    y-axis "microseconds" 0 --> 30
    bar [28.2, 0.042]

Loading

case	before	after	ratio
ExclusivePool Get+Return, 16,000-number pool	28.2 µs	41.7 ns	~680x
GenericPool.first, 59,152 numbers allocated	1.80 s	1.53 ms	~1100-1300x

GenericPool.first is not a typo. With the configured pools nearly full it took roughly two
seconds per call, on the pool actor's thread, blocking that mailbox — and this is the
path every allocation falls back to once any fixed pool is exhausted, which is precisely
when the server is already under load. It is charted only as a table because a bar for the
new value would not be visible next to the old one.

Codec acquisition

CommonFieldData's codecs are obtained once per object in every ObjectCreateMessage.

case	before	after
codec(extra = false)	478-589 ns	at the harness noise floor
codec2(extra = false)	484-591 ns	at the harness noise floor

Reported as roughly 480-590 ns saved per acquisition rather than as a ratio. The new path
is a lazy-val field read, which is close enough to the harness's resolution floor that the
apparent ~165x is partly an artifact of how cheap it now is. The saving itself is real.

Database

Measured on the real loadCampaignKdaData query, 500 avatars and a 200k-row kill log. This
one is a genuine end-to-end query measurement rather than a microbenchmark.

xychart-beta
    title "Campaign KDA query on login, milliseconds"
    x-axis ["without V018", "with V018"]
    y-axis "milliseconds" 0 --> 12
    bar [10.29, 2.47]

Loading

	time	buffers	plan
without V018	10.29 ms	3,917	parallel seq scan, 2 workers
with V018	2.47 ms	636	bitmap index scan

The ratio matters less than the shape: without the indexes this is a sequential scan whose
cost grows with every kill the server ever records, so the figure above is what it looks like
on a young database.

Not measured

• Dispatcher assignment. Its effect is thread placement and contention under concurrent
  load, which a microbenchmark cannot capture. Needs a profiled server with real sessions.
• The bundler drain limit. The 20 to 160 bundles/s figure quoted earlier is arithmetic
  from bundle size and tick interval, not an observation.

Verification

• Full suite: 41 failures against a measured baseline band of 40–42 — the same code varies
  run to run, since the ActorTests are timing-sensitive, as build.sbt notes. No suite fails
  that does not already fail on master. Baseline was established on master plus only the
  import fix, because master's test sources do not compile without it.
• Md5Mac had to stay bit-identical, as this crypto is verified against the live server.
  CryptoTest's known MAC vector passes unmodified, and the rewrite was additionally checked
  differentially against the previous implementation over 6,000 randomised cases — varying
  keys, update counts, lengths straddling the 64-byte block boundary, output lengths and
  interleaved resets — with no mismatches.
• All 35 dispatcher selectors (three fixed pools plus one per ZoneInfo) resolve against the
  real merged configuration.
• The new middleware config key resolves through pureconfig; a mismatch would prevent boot.
• V018 was applied over the full V001→V018 chain on both PostgreSQL 14 (what compose
  declares) and 16 (what is actually deployed), and re-applying it is a clean no-op. Against
  500 avatars and a 200k-row kill log, the campaign KDA query plans as a BitmapOr across
  the two new indexes at 2.5 ms / 636 buffers, versus a parallel sequential scan at 10.3 ms /
  3,917 buffers without them — and the sequential scan's cost grows with the table forever.

Relationship to other open pull requests

Recommendation: merge #1384 before or together with this one.

#1384 (Fix client instability / crashes when zoning frequently) carries the commit that
makes smpHistoryLength configurable and raises it above 100, because 100 entries was
already too small during zone loads. This PR's bundler drain limit shortens the wall-clock
span that same history covers, so the two are coupled: merged together they are
complementary, and this one merged alone puts more pressure on a history that is already
undersized. If only this PR lands, lower packet-bundling-drain-limit from 8 until #1384
follows.

The mechanics are fine — this was checked rather than assumed. Merging fix/zoning-players
into this branch auto-merges with zero conflicts, and the combined result compiles.
That is despite four files being touched by both:

file	this PR	#1384
Config.scala	adds packetBundlingDrainLimit	adds smpHistoryLength
application.conf	adds the drain limit key	adds the history length key
MiddlewareActor.scala	drain loop in processQueue	history sizing and the RelatedA path
InterstellarClusterService.scala	zone dispatcher selection	GetInstantActionSpawnPoint token

Two further overlaps worth flagging to reviewers:

• RelatedB Slot #1343 (RelatedB Slot) also modifies MiddlewareActor.scala. Not checked for conflicts
  here, and it touches the same reliable-delivery machinery the drain limit interacts with,
  so it is worth sequencing deliberately rather than merging blind alongside this.
• GUID Number Pool Shuffling #1380 (GUID Number Pool Shuffling) changes only the guid-pools/*.json configuration,
  so there is no code conflict with the GenericPool/ExclusivePool work here. The two are
  complementary: that PR reshapes how pools are laid out, while this one removes the
  quadratic scan that makes pool exhaustion catastrophic rather than merely slow.

Deliberately not included

• Sector-scoped movement fan-out. PlayerState is still published to the whole zone and
  filtered per recipient, which is O(N²) in player count and is the largest remaining win.
  The receive side is genuinely per-recipient stateful logic (visibility history, adaptive
  delay), so the change is entirely in subscription lifecycle — subscribe each session to its
  neighbouring sectors and re-subscribe on crossing. Getting that subtly wrong makes players
  invisible to each other at sector boundaries, there is no test coverage for the path, and
  it needs a live server and two clients walking a boundary to validate. It should land on
  its own.
• RandomSelector.Return still reports failure when no number is outstanding, leaking
  that GUID. Making it succeed unconditionally would let a double return insert the same
  number twice and eventually issue one GUID to two entities, which is worse. Fixing it
  properly means tracking allocation state explicitly instead of inferring it from a cursor.
• Anything touching the SMP retransmit history. An earlier revision of this branch also
  null-guarded the partially populated history ring, which throws and lets supervision drop
  the client. That fix already exists, together with making the history configurable, in
  Fix client instability / crashes when zoning frequently #1384, so it was removed from here rather than land the same change twice and conflict.
  See the note on the drain limit below for why the two interact.
• Off-mailbox scheduled closures, DB write coalescing and login query pipelining,
  selectDataCodec's remaining per-call construction, and the event bus's use of actor paths
  as channel names (which grows the subscriber classifier for the life of the process).

For reviewers — where the risk is

• Dispatcher assignment is the highest-risk change and wants a thread dump on a running
  server to confirm actors land where intended. Config resolution is verified; actual thread
  placement is not.
• The bundler drain limit changes outbound flow control, and it has a known interaction
  with the SMP retransmit history that a reviewer should weigh before this is deployed.
  MiddlewareActor retains sent SlottedMetaPackets in a fixed 100-entry ring and answers
  the client's RelatedA retransmit requests from it. Draining up to eight bundles per run
  instead of one means that ring wraps up to eight times faster in wall-clock terms, so the
  window in which a retransmit can still be served shrinks accordingly. Per the protocol
  notes, a reliable packet that can never be retransmitted blocks that channel's delivery
  rather than degrading gracefully.
  Fix client instability / crashes when zoning frequently #1384 makes smpHistoryLength configurable and raises it, because 100 entries was
  already too small during zone loads. These two want to land together, or this default wants
  lowering until that one is in — see Relationship to other open pull requests above. The
  drain limit is configurable precisely so it can be tuned down in the meantime.
  It has also not been exercised against a real client on a constrained connection.
• SectorGroup laziness moves when a category is snapshotted, from construction to first
  read. Those reads were already unsynchronised, so this shifts an existing window rather
  than creating one, and removes it entirely for categories never consulted — but it is a
  behavioural change worth a second opinion.
• Components are measured; the server is not. The figures under Measured impact are
  component microbenchmarks plus one real query measurement. They establish that each change
  is faster in isolation, which is not the same as establishing what dominates a live server
  under load, or that the aggregate effect on client-visible hitching is what we expect. A
  profile against a populated server is still the thing that would settle that, and none of
  these numbers substitute for it.

[ScrawnyRonnie]

If this is all a fancy way of saying zones will load much faster (among other things), then it worked.

[GeekOfWires]

The expectation is that PR #1384 should improve zoning reliability and this should improve performance for zoning as well.

Other areas include checking player kill performance via queries.

[Dethdeath]

Can confirm zoning speed increase.

[GeekOfWires]

Addressed the review note that several comments explain how the code changed rather than what it now does — fair point, and this PR was the worst offender for it. Since a reader arriving at these files has no view of the previous implementation, that framing carried no meaning for them; the before-and-after reasoning belongs in the commit messages, where it already was.

Pushed as Improve comment etiquette for our previous work. It reworks every comment this branch added, across 19 files / 92 lines, and is comments only — no behavioural change, and the build was verified before committing.

The sweep covered configuration and SQL as well as Scala, since the same habit had crept into akka.conf, application.conf, build.sbt, docker-compose.yml and the V018 migration.

A few representative rewrites:

• TaskWorkflow previously led with "the monitor must be a dedicated object … so every task bundle contended the same two JVM-wide monitors. It was also not a valid mutex …". It now states what the lock is — a per-bundle object guarding the completion flag so exactly one subtask resolves the promise — and then why the flag itself cannot serve as the monitor. That second part is a live property of java.lang.Boolean caching, not a fact about deleted code, so it stayed.
• ContainableBehavior dropped "The previous form recovered an AskTimeoutException …" entirely. It now simply says one Resume answers the Wait, that onComplete covers both success and failure, and that a second Resume would release an unrelated concurrent move's guard.
• akka.conf was a paragraph about deployment entries that "do not correspond to any actor". It now describes what the default dispatcher is sized for and points to the spawn sites where dispatchers are actually selected.
• application.conf lost "the process previously sent exactly one bundle per run" and the note about the old connection-pool value.

What I deliberately kept: rationale that is still a live fact a reader needs — configured pool sizes, why Err builds its message eagerly, why scodec codecs are safe to share, and a handful of "rather than" phrasings that describe current Akka/driver/Postgres behaviour rather than our previous code.

One comment also gained detail rather than losing it. The InterstellarClusterService dispatcher note now names Zone.Nowhere as the case that actually takes the fallback path, which a local deployment confirmed, instead of the vaguer "not every zone is guaranteed to have an entry".