feat(engine): migrate Agrona buffer types to UnsafeBufferEx extension library#1733
Open
jfallows wants to merge 130 commits into
Open
feat(engine): migrate Agrona buffer types to UnsafeBufferEx extension library#1733jfallows wants to merge 130 commits into
jfallows wants to merge 130 commits into
Conversation
jfallows
changed the title
- Add Arena constructor and asReadOnly() to SafeBuffer - Add arena() method to EngineContext interface - Create EngineContextScoped abstract class with Arena field - Implement Arena.ofConfined() lifecycle in EngineWorker - Replace UnsafeBuffer with SafeBuffer in all layout classes (Streams, Budgets, Events, Scalars, Histograms) - Replace UnsafeBuffer with SafeBuffer in EngineWorker WIP: DefaultBufferPool, EngineEventContext, EchoWorker, TlsWorker, flyweight generator, and binding modules still pending. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
…ontextScoped Arena management is unnecessary — MemorySegment.ofArray() and MemorySegment.ofBuffer() already handle scope implicitly. The migration is a pure UnsafeBuffer → SafeBuffer swap with no Arena lifecycle needed. - Remove Arena constructor from SafeBuffer - Remove arena() from EngineContext - Remove EngineContextScoped abstract class - Keep asReadOnly() on SafeBuffer for immutable views https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
…ontext, and all flyweight generators - Replace UnsafeBuffer with SafeBuffer in DefaultBufferPool and EngineEventContext (completing engine core migration) - Add SAFE_BUFFER_TYPE to flyweight TypeNames - Replace UNSAFE_BUFFER_TYPE with SAFE_BUFFER_TYPE across all 20 flyweight generator files so generated code uses SafeBuffer - Revert unnecessary Arena/EngineContextScoped from EngineContext https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Bulk replacement of UnsafeBuffer → SafeBuffer in 295 files across all runtime bindings, specs, models, guards, vaults, catalogs, and test code. This eliminates all direct usage of Agrona's UnsafeBuffer (backed by sun.misc.Unsafe) in favor of SafeBuffer (backed by MemorySegment + VarHandle). https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Move all mmap-backed layout creation (Streams, BufferPool, Budgets, Counters, Gauges, Histograms, Events) and plugin wiring (bindings, guards, vaults, catalogs, stores, models, exporters, metrics) from the EngineWorker constructor to onStart(). This ensures all initialization runs on the confined worker thread. Add Arena.ofConfined() created in onStart() and closed in onClose(), giving the JVM thread-confinement guarantees for all MemorySegments allocated on the worker. Add Arena constructor to SafeBuffer for arena-backed native memory allocation. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
All UnsafeBuffer usage has been replaced with SafeBuffer backed by MemorySegment + VarHandle, eliminating the dependency on sun.misc.Unsafe and the jdk.unsupported module. Note: Agrona may still require jdk.unsupported transitively — verify after build. If so, this removal needs to wait until Agrona itself migrates away from Unsafe. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Move SafeBuffer import from the org.* group to the io.* group with proper blank line separators between import groups, as required by checkstyle ImportOrder rules. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Move SafeBuffer from engine.internal.concurent to engine.concurrent so generated flyweight code can reference it without internal package access. Add engine as test dependency of flyweight-maven-plugin. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Revert engine dependency from flyweight-maven-plugin pom. Instead, copy SafeBuffer into the plugin's test sources so generated flyweight test code can compile without depending on the engine module. The test copy implements AtomicBuffer directly (not AtomicBufferEx) since the DirectBufferEx/MutableDirectBufferEx interfaces are not available in the flyweight plugin's classpath. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Create runtime/common-agrona module containing Agrona-enhancing classes: SafeBuffer, DirectBufferEx, MutableDirectBufferEx, AtomicBufferEx, ManyToOneRingBuffer. These are pure Agrona extensions with no engine dependencies, breaking the cyclic dependency between the flyweight maven plugin and the engine module. Rename runtime/common to runtime/common-feature for consistency. New package: io.aklivity.zilla.runtime.common.agrona.buffer New package: io.aklivity.zilla.runtime.common.agrona.concurrent https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Flyweight plugin tests create anonymous SafeBuffer subclasses for buffer mutation tracking. UnsafeBuffer was not final, so these tests worked before the migration. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
SafeBufferTest uses JUnit 4 annotations (org.junit.Test). https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
…ation Add common-agrona to specs and incubator dependencyManagement. Replace direct agrona dependency with common-agrona in engine.spec (which doesn't depend on engine transitively). Other spec modules get common-agrona transitively via engine. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
…mon-agrona Fix checkstyle ImportOrder violations by ensuring common.agrona.buffer imports come before common.agrona.concurrent and runtime.engine imports. Move ManyToOneRingBufferTest to common-agrona alongside its source. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
ManyToOneRingBufferTest uses mockito for mocking AtomicBuffer. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
The import-ordering script accidentally removed these imports. Also fix blank line separator between org.* and io.* import groups. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Use JAVA_LONG_UNALIGNED and JAVA_INT_UNALIGNED for VarHandle atomic operations (volatile, ordered, CAS) to match UnsafeBuffer behavior. Aligned layouts reject offsets not divisible by 8/4, which breaks DefaultBufferPool slot metadata access patterns. Alignment optimization can be revisited separately after migration. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
Unaligned ValueLayout VarHandles don't support volatile/ordered/CAS access modes at all (UnsupportedOperationException). Revert to aligned JAVA_LONG and JAVA_INT VarHandles for atomic operations. All Zilla data structures that use atomic access (ring buffers, budget layouts, buffer pool metadata) already guarantee 8-byte aligned offsets by construction. The DefaultBufferPool test failure from the earlier commit was caused by the unaligned VarHandle not supporting ordered writes, not by actual misalignment. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
SafeBuffer atomic operations require 8-byte-aligned offsets via MemorySegment VarHandles. Heap byte arrays may not start at aligned addresses (e.g., Java 25 compact headers with ARRAY_BYTE_BASE_OFFSET=12). Change the test-only DefaultBufferPool constructor to use allocateDirect() instead of allocate(). Direct ByteBuffers are always 8-byte aligned. Production code already uses MappedByteBuffer (direct) via BufferPoolLayout. https://claude.ai/code/session_01Ets6Ubs1ai4FyTZWvGWv4f
TlsWorker uses Agrona's OneToOneRingBuffer.read() which delivers MutableDirectBuffer, not MutableDirectBufferEx. https://claude.ai/code/session_015ADyA3u6WuwfBNWVJJmZqZ
…BufferEx Bulk migration across model-json, model-core, model-avro, model-protobuf, binding-kafka-grpc, binding-grpc-kafka, and tests. Also cast DirectBufferInputStream.buffer() to DirectBufferEx in JsonModelHandler. https://claude.ai/code/session_015ADyA3u6WuwfBNWVJJmZqZ
jfallows
force-pushed
the
feature/1723-unsafebufferex-migration
branch
from
0d471d7 to
ade3ac1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DirectBufferEx/MutableDirectBufferEx/AtomicBufferExextendingAgrona's buffer interfaces with
segment()accessor andwrapAdjustment()-basedoffset semantics — zero allocation on the hot path
UnsafeBufferExextendingUnsafeBufferwithMemorySegmentoverlay,ExpandableArrayBufferExandExpandableDirectByteBufferExfor expandable bufferswrap/tryWrap/buffer()signatures fromDirectBuffer/MutableDirectBufferto their Ex counterpartsMessageHandlerExandRingBufferExwithreadEx()deliveringMutableDirectBufferEx;MessageConsumerextendsMessageHandlerExEngineWorkertoreadExforMemorySegment-aware frame dispatchcommon-agronamodule housing all Ex buffer types and ring buffer extensionsResolves #1723
Test plan
binding-httprfc7540 MessageFormatIT passes (exercises hot-path buffer pool)binding-kafkaCacheFetchIT passes (exercises expandable buffer paths)https://claude.ai/code/session_015ADyA3u6WuwfBNWVJJmZqZ