EIP-8297 partitioned binary tree commitment engine - #22942
Draft
awskii wants to merge 58 commits into
Draft
Conversation
EIP-8297 binary state trie as a new commitment engine alongside HexPatriciaHashed. M0 scope: in-memory over MockState, account and storage zones, validated against a transcription of the spec's own reference algorithm. No external API changes.
First step of the EIP-8297 binary commitment engine. pbinBitpath holds an up-to-528-bit tree key as big-endian words so divergence is XOR plus LeadingZeros64, clamped by the shorter path so bits past bitLen never leak into the result. pbinEncodeBitPath/pbinDecodeBitPath spell a path as packed MSB-first bits plus a trailing bitLen-mod-8 byte; the count trails so a subtree's records stay contiguous in the keyspace, and every non-canonical spelling is rejected on read so one path maps to exactly one DB key. Nothing outside the new files changes.
Add pbinAddr32, pbinTreeKeyAccount and pbinTreeKeyStorage deriving the 34-byte account-zone and 66-byte storage-zone keys, with slots below 64 routed into the account header at HEADER_STORAGE_OFFSET+slot. A two-level digest cache memoizes key_hash(addr32) per address and key_hash(addr32||tree_index) per 256-slot group; pbinKeyHasher exposes the primary leaf's key as a keyHasher, leaving Updates untouched. Vectors are pinned against x/crypto keccak computed in the test body, not against the helper under test.
BASIC_DATA packs version/code_size/nonce/balance big-endian into 32 bytes; a balance the 16-byte field cannot hold errors instead of truncating, which would commit a wrong root. CODE_HASH maps an unset hash to the empty-code hash. Storage values are left-padded to the fixed leaf width. Expectations are hand-written hex rather than the encoder's own output: the Task 4 oracle consumes this same encoder, so a differential root test is structurally blind to value-encoding bugs.
Naive transcription of the spec's BinaryTree/_insert/merkelize (eip:112-222) in the test package: one bit per byte, no memoised hashes, x/crypto Keccak rather than the engine's fastkeccak so a hasher bug cannot cancel out on both sides. This is the ground truth root equality is measured against. Corpus builders cover empty, single key (root is a leaf), divergence at bit 0 and bit 527, a split-inside-prefix triple that forces prefix[matched+1:], one account's full stem, and a mined deep-shared-prefix cluster. Self-consistency tests pin encode_bit_prefix and the leaf/branch hashes against hand-written bytes, and assert permutation independence and prefix-freedom over every corpus.
pbinCell carries a single tree-key-space prefix instead of HPH's hashedExtension/extension pair: that split exists to hold the hashed and plain key spaces separately, while PBin derives the tree key from the plain key on demand. There is no memoized leaf hash either — H(0x00||key||value) commits the complete key, so it never depends on where the leaf sits and nothing is worth caching. The record codec is PBin-local. A 66-byte storage prefix does not fit the shared cellEncodeData.extension field, and PatriciaContext moves branch payloads as opaque bytes, so nothing shared changes. Every record encodes both child cells (bitmap = afterMap), which removes the merge-with-previous path: at arity 2 the untouched sibling is the whole other half of the subtree, and merging is what loses it. The prefix bit count is an explicit uvarint and the authority on length; decode derives the byte count from it, rejects a short buffer, rejects non-zero pad bits, and rejects trailing bytes, so one node has one record spelling.
Add the binary tree's two node hashes and the single cell hasher they back. - pbinLeafHash = H(0x00 || key || value) over the complete 34/66-byte tree key, so a leaf's hash is independent of where it sits. - pbinBranchHash = H(0x01 || encode_bit_prefix(prefix) || left || right), with encode_bit_prefix's explicit two-byte bit count keeping a 7-bit prefix distinct from an 8-bit one agreeing on the pad bit. - One 133-byte scratch buffer (tag + count + 66-byte prefix + two children) holds every preimage either shape produces, so a node costs one hash call and no allocation. - Exactly one cell hasher: cellHash dispatches on node kind and derives a leaf's complete key from the descent path plus the cell prefix. That key is also self-describing enough to pick the leaf value, so no extra cell field is needed to tell BASIC_DATA, CODE_HASH and a header-resident slot apart. - The empty subtree is 32 zero bytes per eip:208, deliberately not empty.RootHash. Tests diff every shape against the Task 4 reference oracle: single leaf, a branch at eight prefix lengths including empty and the deepest a 528-bit key admits, a nested branch, an absent child, and two-key corpus roots folded by hand through the primitives. M0 gap: the shared Update carries no code size, and extending it is an external API change, so BASIC_DATA encodes code_size as 0 until code chunking lands. Engine and oracle share the encoder, so the M0 root-equality gate is unaffected.
Add PBinPatriciaHashed with the [528][2] grid, a bit-path currentKey, the context and the Keccak state, plus the two navigation primitives. needUnfolding answers with an action rather than the hex engine's cpl+1 nibble count. That count folds three different situations into one number and only works because the trailing +1 discounts a hex terminator, which the binary tree does not have. Descend (the probe key agrees with the whole cell prefix) and Split (it leaves the prefix partway) are kept apart because only Split shortens a stored node's prefix, and EIP-8297 puts the prefix inside the branch hash — so a split invalidates a cached child hash, which the hex engine never has to deal with. unfold ports the hex engine's two-step descent: consuming a cell's prefix materializes a row whose cell has no prefix left, and only then is the record read, keyed by the parent path plus that stored prefix. The parent's copy of the prefix is the only place those bits exist, so it is what makes the descent key reconstructible. Cell kind is an explicit field rather than "prefix length is zero", since EIP-8297 admits a branch with an empty prefix and a missing record below one is an inconsistency, not an empty subtree. Dropped from the hex port: terminator arithmetic and clampToAccountBoundary (one key space, no account/storage split), and the depth>64 accountAddrLen reset in fillFromUpperCell. Split rehash is structural only here; materialize-on-split lands with fold.
fold dispatches a row by how many cells survive it: branch (two), propagate (one), delete (none). A branch fold hashes both cells, stores the row as one self-contained record keyed by the encoded bit path, and hands the parent a branch cell. A propagate collapses the row into the cell above, prepending the bits the row consumed and writing no record. EIP-8297 puts a node's prefix inside its hash, so any prefix change invalidates it. The plan covered the split direction; the ordinary descent has the same problem in reverse, since unfold moves a prefix out of a cell and the propagate that follows moves it back. Cells this run built now carry their two child hashes in memory, so a prefix change re-derives; cells that arrived from a record still materialize from their own record, which is unchanged by a split. Counters measure how often that fallback fires. A record carries plain keys, not values, so an untouched sibling loaded from one is read back from state before it is hashed.
Name the EIP, the Keccak-256 choice for both node hashing and tree-key derivation, and the M0 scope boundaries at the top of the engine file. Kept off the package clause: the comment describes this engine, not the whole commitment package.
execution/commitment: persist the root cell so a stored tree is reloadable. Records are keyed by the path that reaches the node, which leaves the root itself unnamed: its prefix is empty only when the tree spans both zones, so a single-zone tree (and a bare-leaf root, which writes no record at all) was rebuilt from the touched keys alone after Reset — a wrong root with no error. The root cell now goes to a reserved key no bit path encodes. execution/commitment: reject a delete that lands on a live leaf. ModeDirect carries no Update, so the stream-delete guard was unreachable and every absent state read returned nil, leaving the stale leaf in the tree. execution/commitment: reject a record leaf with no plain key — it hashed a zero-valued state instead of failing. Also: one divergence primitive instead of two, with the word scan on the descent path; drop pbinMaxDepths, pbinGrid.reset and the pbinBranchData alias; move the test-only leafHash into the test file; cover the context error paths. .github, docs: register the two pbin fuzzers, cap fuzz minimization so a late crash cannot minimize past the coordinator deadline. cmd/integration: commitment visualize decodes hex records only — reject --trie=bin instead of reporting hex-shaped stats for it.
pbinSplitSlot took a pointer and returned a 32-byte array by value, so the copy it avoided on the way in was paid on the way out. The array it built existed only to be copied again into the digest buffer. The tree index is slot>>8, i.e. a zero byte followed by the slot's top 31 bytes, so groupDigest writes it straight into the buffer it hashes. Drops one stack array and two copies per storage-key derivation, and narrows the cache key to the 31 bytes that actually discriminate.
…rpus Drives both engines over the same updates and reports branch-record count, record bytes and path depth in key bits, plus the binary engine's split counters. Roots differ by construction, so the test asserts they disagree. Adds a stem co-location test: slots sharing a tree_index differ only in the sub-index byte, a path random 32-byte slots never reach. The hex engine needs accountKeyLen=20 here — the existing corpora use one-byte stand-in addresses, which the binary engine cannot consume at all since it derives its zone from the plain-key length.
Vectors exported from the reference implementation in ethereum/execution-specs branch projects/binary-trie. That reference hashes with BLAKE3 and this engine with Keccak-256, so roots and tree-key bodies cannot be compared yet. Two things survive the hash difference and are checked: BASIC_DATA packing, which involves no hash, and the zone/length/sub-index routing, which is positional. The routing vectors cover the header/storage-zone boundary at slots 63 and 64 and a slot of 2**255.
…n oracle H was applied at four sites — two node preimages and the two key digests — so the previous claim that the hash lived in pbinHasher alone was wrong. All four now route through pbinHashFn, defaulting to Keccak-256, which leaves production behaviour unchanged and lets a suite be substituted whole. The reference implementation in ethereum/execution-specs hashes with BLAKE3. Replaying its vectors under BLAKE3 reproduces all 7 fixed roots and all 600 sequence roots. That reference builds the tree by canonical rebuild while the oracle inserts incrementally as the EIP's pseudocode does, so agreement across that difference is what rules out a shared misreading of the spec. Deletes in the sequences are applied by rebuilding from the surviving key set: the EIP's insert has no removal and the reference's removal semantics are still open, so nothing here depends on a delete algorithm.
The oracle already replays the reference's roots; this drives the engine itself, so the comparison no longer goes through our own transcription. Vectors carry raw tree keys and raw values while the engine rebuilds a leaf value from an Update, so each value is mapped onto the field the engine reads for that key position. Tree keys are digests and cannot be inverted, so the plain key is synthetic: only its length is read, to pick the cell field. Six of seven vectors reproduce. full_header_stem spans code-chunk sub-indices, which no Update field carries, and is excluded by an asserted name list so gaining code support breaks this test rather than quietly widening it.
Wires the M0 engine into the domain path so a dev-chain container can boot on it with --experimental.bin-commitment. Keccak stays the production hash; BLAKE3 is a test-only override for replaying the reference vectors. The header state-root check becomes an independent toggle defaulting to on, so a self-produced chain keeps it as an oracle and only a foreign-header chain turns it off. Grounded in a survey of the five integration surfaces. Carries 13 hazards as acceptance criteria and 5 open questions, two of which need an upstream answer before the paths that depend on them can be built.
Domain iteration treats a zero-length key as end-of-stream, and the empty key sorts first in TblCommitmentVals — so a root record stored under it truncated the whole iteration and the datadir read back as fresh (H2). Replace the empty root key with a single-byte 0x08 sentinel, which no pbinAppendBitPath encoding can produce: every bit-path key ends in a trailing bit-count byte <= 7.
…es with real prevData The domain layer refuses a nil value in DomainPut, so the empty-root storeRoot path and foldDelete would fail the first time the engine runs over a real datadir. Zero-length is the deletion encoding at that boundary (TemporalMemBatch routes any len==0 write to DeleteWithPrev), so both sites now hand over a non-nil empty slice. Every PutBranch also carries the record it replaces: the grid retains each row's record bytes at unfold and the engine retains the root record across load/store, sparing the domain a GetLatest per branch write and keeping history rows accurate.
The pbin state blob is the root cell plus the three root flags. Proven restorable-as-zero: every row-indexed grid array is initialized at unfold before activeRows exposes the row, so nothing per row is serialized and no depth ever meets a one-byte encoding (H6). The commitmentdb save/restore gates (LatestCommitmentState, encodeCommitmentState, restorePatriciaState) now assert the optional StatefulTrie interface instead of enumerating hex variants, the bin-variant ctor panic is gone, and the context carries the real variant tag. The 16-byte txNum-blockNum header stays byte-identical across variants.
…ent flag, trie_variant persistence, togglable header root check The bin trie variant becomes a whole-datadir property: a new trie_variant key in erigondb.toml (written only when bin, absent = hex) is reconciled at every settings resolve — a persisted bin adopts bin process-wide, while a hex datadir under the bin flag, a legacy datadir, references-in-branches, and the streaming/parallel experiments are refused rather than degraded (guards H4, H10). PickTrieVariant gains the bin case and replaces the duplicated inline switch in squeeze.go. The header state-root check is now independently togglable via dbg.CheckHeaderStateRoot (CHECK_HEADER_STATE_ROOT, default ON), honoured through headerRootMismatch at all five comparison sites in exec3.go, exec3_serial.go and committer.go, with a loud startup warning when disabled (guards H13).
…ep the bin variant at genesis, refuse it on hex-only paths
# Conflicts: # execution/commitment/parallel_patricia_hashed.go
Working documents for the M0/M1 milestones; the engine doc and the M1b gate record carry what outlives them.
Cut milestone and plan narration, hazard-list tags pointing at the deleted plan docs, rationale repeated across sites, and docstrings restating the code. Comment-only: 632 insertions, 890 deletions, no code line changed.
The reference names its fork BinaryTree — Amsterdam rules, state committed through the binary tree — so the fixtures were unrunnable: the fork registry had no such network. Selecting it needs the statecfg hash field, not just SetPBinHashSuite. The settings resolver reads that field to persist trie_hash, and re-applies what it persisted; a suite set before the datadir is resolved is otherwise overwritten by the keccak default. 67 of 70 blockchain fixtures pass. The three failures are an account deletion, which the engine refuses, and two multi-block state-root divergences.
awskii
requested review from
AskAlexSharov,
bloxster,
lupin012,
lystopad,
mh0lt,
mriccobene,
sudeepdino008,
taratorio and
yperbasis
as code owners
August 1, 2026 15:57
The EEST fixtures on execution-specs projects/binary-trie are the external conformance check now, so the in-repo vectors only need enough sequence coverage to keep the oracle honest: 20 ops per seed rather than 120, stored minified. pbin_vs_hex_compare_test.go asserted only that two engines disagree on a root, which is true by construction; the rest was measurement that has served its purpose. The M1b smoke record was a working document.
Collaborator
|
Reviewed the full diff (engine + integration). Builds clean, The engine core looks solid: prefix-free zone keys, canonical branch/bitpath codecs that reject non-canonical spellings, a reference oracle plus fuzzing, and the hex paths genuinely untouched. Six things worth fixing before merge; the first is a live defect in shared test infrastructure. 1. Test-global leak in
|
awskii
marked this pull request as draft
August 3, 2026 06:45
Member
Author
|
yes most of the changes reported are intended - PBT is not compatible to hex trie |
Member
|
@awskii plz ethpandaops/eth-client-docker-image-builder#398 with the branch |
Both conflicts were the same collision: this branch swapped the root predicate for headerRootMismatch, while main independently changed the other half of the same line. Kept both — headerRootMismatch with main's cc.fail(target) and se.logWrongTrieRoot.
execution-specs 7b6aed29c revised the Amsterdam state-access schedule.
These constants still held the pre-revision values, so every Amsterdam
fixture diverged on gas used and, through it, on the receipt root.
COLD_STORAGE_ACCESS 3000 -> 2100
ACCOUNT_WRITE 8000 -> 9000
CREATE_ACCESS now pairs ACCOUNT_WRITE with COLD_ACCOUNT_ACCESS
rather than COLD_STORAGE_ACCESS (11000 -> 12000)
REFUND_STORAGE_CLEAR 12480 -> 11616, recomputed from the new
COLD_STORAGE_ACCESS
CALL_VALUE and the access-list storage cost are already expressed in
terms of the two changed constants and follow on their own.
EIP-8038 is gated on IsAmsterdam and no chainspec schedules Amsterdam,
so no live network reaches this schedule.
`debug_executionWitness` and `eth_getProof` had only a hex-trie path, so on a binary-tree chain they had nothing to serve. ## Changes - Witness capture for `PBinPatriciaHashed`: a tracer on the hasher records the nodes the fold emits, and `PBinWitnessNodesForKeys` prunes that superset down to what a verifier needs for the proved keys. - `RecordingState` wiring so the RPC re-executes the block and collects the touched keys. - Spec catch-up carried alongside: all code chunks moved into the content-addressed code zone (execution-specs#3310), and the EIP-7702 delegation indicator given its own header leaf at sub-index 2 (execution-specs#3324). Changes to the trie engine itself are limited to the tracer field, `Release` hygiene, and splitting `followAndUpdate` into `seek` + `updateCell`. Against the EIP-8297 fixtures filled from execution-specs `e4d7865b6` (227 cases), the blockchain corpus passes 78/82 with this branch merged into the base. Four remain: three receipt-root mismatches (`contract_creating_transaction`, `selfdestruct_same_transaction_leaves_no_account`, `extcodecopy_from_initcode_clones_chunked_code`) and one gas gap in `initcode_size_limit_boundary[at_max]` that does not track the EIP-8038 constants. The 63 state-test fixtures are not a signal either way: `selectCommitmentVariant` is wired only into the blocktest path, so they execute on the hex trie and compare against binary-tree roots.
…pts in The revised state-access schedule was applied globally, so every Amsterdam fixture in the pinned spec-test corpora diverged on gas used and, through it, on the state and BAL roots: 13073 of 24041 zkevm sub-tests, all six devnet spec shards, and the execution/vm gas tests. Keep the pre-revision constants as the default and put the revised set behind Rules.EIP8038Revised, which only Forks[BinaryTree] sets: its fixtures track head-of-spec, the pinned corpora do not. No scheduled network reaches it. CreateAccessEIP2780 now derives from CreateAccessEIP8038 instead of repeating the number; the two had already drifted apart. tests-zkevm@v0.6.2: 24041 ran, 0 failed.
…d deferInLoop Six err re-assignments become declarations, and the storage-layout test releases its trie per iteration rather than deferring inside the loop.
The bin trie is sequential-only and settings resolution refuses it combined with parallel or streaming commitment, so these tests force both off and restore the process-wide values on cleanup.
…bin witness code Removes doc comments that restate the signature, the "ok is false when the witness proves it absent" tail repeated at four sites, and a task reference. The canonical statements are kept where they belong: the absence cases at leaf(), the leafCellHash cross-reference, the root-first ordering contract, and the engine rationale in pbin_witness_context.go's file comment. Comments only.
…g transactions CreateAccessEIP8038Revised reached only the CREATE and CREATE2 opcodes. A contract-creating transaction prices through CreateAccessEIP2780, which derives from the unrevised constant, so a config opting into the revised schedule charged 12000 for the opcode and 11000 for the transaction.
loadCellState substituted 32 zero bytes for a storage leaf whose plain key the domain no longer held, and committed it. EIP-8297 collapsed zero into absent (spec 7852514), so no entry set produces that root — the value is unrepresentable rather than merely stale. The account arm already refused; the storage arm was the outlier, as was the witness context, which refuses the same read with errPBinWitnessNoState. The condition means the caller left a removal out of its update set. Removal lives on the update path and the grid only walks forward, so the fold cannot repair it and fails instead.
Member
Author
|
Witness sizes under the bin engine, from
State-only: bin 6,835 B vs hex 2,729 B, 2.50x. Headers identical at 3,527 B. The test also prints a corpus total of hex 22,648 B vs bin 10,362 B (0.46x), excluding bin's What the corpus does support: bin proves only the chunks a block executes, where hex ships the whole contract. The magnitude needs a dense-bytecode corpus before it means anything. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EIP-8297 partitioned binary tree as a commitment engine, behind
--experimental.bin-commitment. Off by default; the hex trie is untouched.67 of 70 EIP-8297 blockchain fixtures pass (execution-specs
projects/binary-trie).The three failures are fixtures the reference marks as pinning current provider
behaviour rather than conformance:
state_pbt.pydeletes a slot on a zero write, whileEIP-8297 keeps a zero-valued leaf as distinct from an absent key. This engine
follows the EIP.
storage; the reference calls the semantics an open consensus question.
Deletion caveats:
the reference's two providers disagree on it.
tree is a function of history, not of current state. A rebuild from the state
domains cannot know they exist, which makes recompute-from-domains invalid as
an oracle for a code-bearing account. Reachable via an EIP-7702 delegation clear.
Docker image branch: ethpandaops/eth-client-docker-image-builder#400 (tracking issue #398).