feat: Block hash calculation v2, continuity validation and refactoring

.github/workflows/e2e-leader-follower.yml

@@ -87,7 +87,17 @@ jobs:
       fail-fast: false
       matrix:
         test:
-          [brlc, importer, miner, change, deploy, kafka, health, tx-types]
+          [
+            brlc,
+            importer,
+            miner,
+            change,
+            deploy,
+            kafka,
+            health,
+            tx-types,
+            blockhash,
+          ]
         use_block_changes_replication: [false, true]
     name: "E2E Leader & Follower on ${{ matrix.test }} (Replication: ${{ matrix.use_block_changes_replication }})"
     needs: setup_cache

docs/continuity.md

@@ -0,0 +1,150 @@
+# Block continuity
+
+## Progress model
+
+Stratus tracks two process-local progress tips:
+
+- `latest_sealed` is the execution tip. Temporary storage owns its latest in-process state overlay and final hash. Its hash is the parent used to build the next header; after startup, the durable underlying state still comes from RocksDB.
+- `last_saved` is the durable tip. `StratusStorage` stores only its block number and hash because the complete block and state already live in RocksDB.
+
+On a populated database, both are initialized from the latest permanent block. On an empty database, temporary storage keeps the canonical genesis reference, but its pending block remains block 0 and `last_saved` remains empty until genesis is imported or persisted.
+
+Normal leader and follower flows seal and save sequentially, so the tips usually match. The offline importer deliberately pipelines execution and persistence, allowing `latest_sealed` to run ahead.
+
+```mermaid
+flowchart LR
+    subgraph durable [Permanent progress]
+        P0["Block N-1"] --> P1["Block N: last_saved"]
+    end
+
+    subgraph backlog [Offline FIFO backlog]
+        Q1["Block N+1"] --> Q2["Block N+2"]
+    end
+
+    T["Block N+3: latest_sealed"]
+    P1 --> Q1
+    Q2 --> T
+```
+
+These are not independent chains. Permanent progress must always be an ordered prefix of sealed progress.
+
+## Hash schemes
+
+Genesis retains the legacy V1 hash:
+
+```text
+V1 = keccak256(number as 8-byte big-endian)
+```
+
+Locally sealed non-genesis blocks use V2:
+
+```text
+V2 = keccak256(
+    number as 8-byte big-endian
+    || timestamp as 8-byte big-endian
+    || transactions_root
+    || parent_hash
+)
+```
+
+Reexecution validates that the imported hash is V2 or the temporary V1 compatibility hash. Replication receives a prebuilt block and relies on the universal saved-chain continuity checks.
+
+## Guarded sealing
+
+`PendingSession` owns a `PendingBlockGuard`, which wraps temporary storage's `pending_session` mutex colocated with `InMemoryChainState`. Callers use session methods instead of manually pairing lock-acquiring methods with variants that accept an existing guard.
+
+One session spans the complete pending-block lifecycle:
+
+```text
+set pending header when importing
+→ execute and save transactions
+→ snapshot pending state
+→ build the block and determine its final hash
+→ finish pending state
+```
+
+Local sealing and reexecution use the snapshot to build a block without destroying pending state. Local sealing calculates V2; reexecution validates the external V2 or V1 hash. If external validation fails, pending remains unchanged. Replication skips this snapshot-and-build step because it receives a complete block.
+
+Once the block is accepted, `finish_pending_block` uses `std::mem::replace` to move the original pending state into `latest_sealed`, attaches the final hash, and creates the next pending state.
+
+Pending and latest sealed state share one `RwLock<InMemoryChainState>`, so the move, hash update, and next-pending creation are one atomic write. Another pending session cannot start until `PendingSession` is dropped or consumed by sealing.
+
+Local synchronous modes also retain the existing `mine_and_commit` mutex from sealing through persistence. It guarantees that concurrent local triggers cannot seal blocks in one order and race to save them in another order. Offline importer does not use this mutex; its single executor, FIFO channel, and single saver provide ordering.
+
+## Saving and continuity
+
+`save_block` is the authoritative continuity boundary for leader mining, follower reexecution, follower replication, fake leader, and offline import.
+
+Before saving block `N`, it validates:
+
+```text
+N == last_saved.number + 1
+N.parent_hash == last_saved.hash
+```
+
+RocksDB is written only after these checks pass. `last_saved` advances only after the write succeeds, and no permanent read is required during each save.
+
+Online follower modes perform the same check as a read-only preflight before emitting Kafka events. `save_block` repeats it authoritatively before persistence.
+
+External reexecution calculates the block locally and accepts the external V2 hash or the temporary V1 compatibility hash. Replication receives a prebuilt block, but both modes still pass through the universal saved-chain continuity check.
+
+If the process restarts, sealed-but-unsaved work is discarded. The durable tip is loaded from RocksDB, temporary state resumes from it, and the unsaved range is executed again.
+
+## Block-hash lookup
+
+The block-hash cache does not determine chain progress or parent continuity. Its normal role is accelerating the EVM `BLOCKHASH` opcode.
+
+```mermaid
+flowchart LR
+    EVM["EVM BLOCKHASH"] --> TempCheck{"Is it the latest sealed block?"}
+    TempCheck -->|"yes"| Result["Block hash"]
+    TempCheck -->|"no"| Cache["Block-hash cache"]
+    Cache -->|"hit"| Result
+    Cache -->|"miss"| Permanent["Permanent storage"]
+    Permanent --> Result
+
+    Queue["Offline sealed backlog"] -. "temporary workaround" .-> Cache
+```
+
+Lookup order is:
+
+1. The latest sealed block, whether or not it was already saved.
+2. The block-hash cache.
+3. Permanent storage.
+
+The first step does not consult `last_saved`. A sealed block keeps its hash when it is persisted, so the sealed tip is authoritative for its own number either way. Reading it would otherwise put the `BLOCKHASH` opcode behind the whole block saving critical section, which holds `last_saved` across the RocksDB write.
+
+The cache is a fixed ring indexed by `number % capacity`, not a general purpose cache. Block numbers are dense and published in order, so the ring retains exactly the most recent `capacity` blocks, and two numbers share a slot only when they are `capacity` apart. Nothing inside the window can be evicted, including a hash that is published and never read. That matters because a general purpose cache does the opposite: it spreads entries over independently sized shards and evicts the never read ones first, which is exactly what a freshly sealed hash is.
+
+The default is 256 entries, matching the `BLOCKHASH` window. Administrators may shrink it, and a configured zero is raised to a single slot: that slot can only ever hold the sealed tip, which is answered from temporary storage before the ring is consulted, so it is a cache in name only. Importer-offline always adds capacity for its bounded sealed-but-unsaved backlog:
+
+```text
+configured capacity + batch_size × (queue_size + 2)
+```
+
+The extra two batches cover one batch being built by the executor and one being processed by the saver. Sizing the ring this way is what makes the unsaved backlog unevictable rather than merely likely to survive.
+
+Older unsaved offline blocks are temporarily dependent on this cache because permanent storage cannot serve them yet. Remove this workaround when importer-offline is removed.
+
+## Lock roles
+
+- `PendingSession`: exposes pending-header setup, execution append, and sealing under one temporary-storage session lock.
+- `PendingBlockGuard`: private capability held by `PendingSession` and passed only to lower storage layers.
+- `TransactionGuard`: lets fake leader hold the executor transaction mutex before miner locks, matching RPC lock order and preventing deadlock.
+- `mine_and_commit`: preserves seal-to-save ordering for synchronous local modes.
+- `commit`: serializes permanent writes.
+- `last_saved`: serializes durable continuity validation and advancement.
+- `transient_state_lock`: preserves consistency between permanent writes and latest account/slot caches.
+
+## TODO: Temporary storage naming
+
+`InmemoryTransactionTemporaryStorage` and `transaction_storage` are misleading names. This component does not represent one Ethereum transaction or a database transaction. It owns:
+
+- The pending block header and all transaction executions.
+- Aggregated pending account and slot changes.
+- The latest sealed block state and hash.
+- The transition from pending to latest sealed.
+
+For example, an interval-mined pending block can contain many Ethereum transactions. When sealed, the complete pending state becomes `latest_sealed`; an individual transaction does not.
+
+A future focused refactor should rename it. Preferred naming is `InMemoryBlockStateStorage` with a `block_storage` field. `InMemoryExecutionStateStorage` with `execution_storage` is another reasonable option.

e2e/cloudwalk-contracts/integration/test/leader-follower-blockhash.test.ts

@@ -0,0 +1,159 @@
+import { expect } from "chai";
+import { concat, keccak256, toBeHex } from "ethers";
+
+import { ALICE, BOB, CHARLIE } from "./helpers/account";
+import {
+    sendAndGetFullResponse,
+    sendWithRetry,
+    toHex,
+    updateProviderUrl,
+    waitForFollowerToSyncWithLeader,
+} from "./helpers/rpc";
+
+const HASH_ZERO = "0x" + "0".repeat(64);
+
+// Root reported by blocks that mined no transaction.
+const EMPTY_TRANSACTIONS_ROOT = "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421";
+
+// How many of the most recent blocks are inspected, so that the suite does not walk a chain of arbitrary length.
+const INSPECTED_BLOCKS = 20;
+
+// Recomputes the hash of a mined block from the header fields it reports, mirroring the node.
+//
+// The preimage is the block number and the timestamp, both as 8 byte big endian integers, followed by the
+// 32 byte transactions root and the 32 byte parent hash.
+function calculateBlockHash(block: any): string {
+    const number = toBeHex(BigInt(block.number), 8);
+    const timestamp = toBeHex(BigInt(block.timestamp), 8);
+    return keccak256(concat([number, timestamp, block.transactionsRoot, block.parentHash]));
+}
+
+// The genesis keeps the legacy scheme, which hashes only the block number.
+function calculateGenesisHash(): string {
+    return keccak256(toBeHex(0n, 8));
+}
+
+async function getBlock(node: string, blockNumber: number): Promise<any> {
+    updateProviderUrl(node);
+    return await sendWithRetry("eth_getBlockByNumber", [toHex(blockNumber), false]);
+}
+
+describe("Leader & Follower block hash integration test", function () {
+    // The most recent blocks both nodes agree on, plus the first mined block, which is the oldest one
+    // hashed with the current scheme. Set once the follower catches up with the leader.
+    let inspectedBlocks: number[] = [];
+
+    it("Validate initial Leader and Follower health", async function () {
+        updateProviderUrl("stratus");
+        expect(await sendWithRetry("stratus_health", [])).to.equal(true);
+
+        updateProviderUrl("stratus-follower");
+        expect(await sendWithRetry("stratus_health", [])).to.equal(true);
+    });
+
+    it("Genesis block on the Leader is hashed with the legacy scheme", async function () {
+        const genesis = await getBlock("stratus", 0);
+
+        expect(genesis.number).to.equal("0x0");
+        expect(genesis.parentHash, "genesis has no parent").to.equal(HASH_ZERO);
+        expect(genesis.hash, "genesis hash is the keccak of its number").to.equal(calculateGenesisHash());
+    });
+
+    it("Genesis block on the Follower is identical to the one on the Leader", async function () {
+        const leaderGenesis = await getBlock("stratus", 0);
+        const followerGenesis = await getBlock("stratus-follower", 0);
+
+        expect(followerGenesis, "genesis blocks differ between leader and follower").to.deep.equal(leaderGenesis);
+    });
+
+    it("Send transactions to the Leader so that the inspected blocks carry a transactions root", async function () {
+        updateProviderUrl("stratus");
+
+        for (const [sender, receiver] of [
+            [ALICE, BOB],
+            [BOB, CHARLIE],
+            [CHARLIE, ALICE],
+        ]) {
+            const nonce = parseInt(await sendWithRetry("eth_getTransactionCount", [sender.address, "latest"]), 16);
+            const signedTx = await sender.signWeiTransfer(receiver.address, 0, nonce);
+            const txHash = keccak256(signedTx);
+
+            const response = await sendAndGetFullResponse("eth_sendRawTransaction", [signedTx]);
+            expect(response.data.result).to.equal(txHash);
+
+            await sendWithRetry("eth_getTransactionReceipt", [txHash]);
+        }
+    });
+
+    it("Wait for Follower to sync with Leader", async function () {
+        const { leaderBlock } = await waitForFollowerToSyncWithLeader();
+        const syncedBlock = parseInt(leaderBlock, 16);
+        expect(syncedBlock, "chain should have blocks past the genesis").to.be.greaterThan(0);
+
+        const oldest = Math.max(1, syncedBlock - INSPECTED_BLOCKS + 1);
+        const recent = Array.from({ length: syncedBlock - oldest + 1 }, (_, i) => oldest + i);
+        inspectedBlocks = recent[0] === 1 ? recent : [1, ...recent];
+    });
+
+    it("Leader hashes every block over its number, timestamp, transactions root and parent hash", async function () {
+        let blocksWithTransactions = 0;
+
+        for (const blockNumber of inspectedBlocks) {
+            const block = await getBlock("stratus", blockNumber);
+
+            expect(block.hash, `hash of block ${blockNumber}`).to.equal(calculateBlockHash(block));
+
+            if (block.transactionsRoot !== EMPTY_TRANSACTIONS_ROOT) {
+                blocksWithTransactions++;
+            }
+        }
+
+        // without this the transactions root would be constant and its contribution to the hash unverified
+        expect(blocksWithTransactions, "no inspected block mined a transaction").to.be.greaterThan(0);
+    });
+
+    it("Leader chains every block to the hash of its parent", async function () {
+        for (const blockNumber of inspectedBlocks) {
+            const block = await getBlock("stratus", blockNumber);
+            const parent = await getBlock("stratus", blockNumber - 1);
+
+            expect(block.parentHash, `parent hash of block ${blockNumber}`).to.equal(parent.hash);
+        }
+    });
+
+    it("Follower reports the same hashes as the Leader for every block", async function () {
+        for (const blockNumber of [0, ...inspectedBlocks]) {
+            const leaderBlock = await getBlock("stratus", blockNumber);
+            const followerBlock = await getBlock("stratus-follower", blockNumber);
+
+            expect(followerBlock.hash, `hash of block ${blockNumber}`).to.equal(leaderBlock.hash);
+            expect(followerBlock.parentHash, `parent hash of block ${blockNumber}`).to.equal(leaderBlock.parentHash);
+            expect(followerBlock.timestamp, `timestamp of block ${blockNumber}`).to.equal(leaderBlock.timestamp);
+            expect(followerBlock.transactionsRoot, `transactions root of block ${blockNumber}`).to.equal(
+                leaderBlock.transactionsRoot,
+            );
+        }
+    });
+
+    it("Follower stores hashes that match the fields of the blocks it imported", async function () {
+        for (const blockNumber of inspectedBlocks) {
+            const block = await getBlock("stratus-follower", blockNumber);
+
+            expect(block.hash, `hash of block ${blockNumber}`).to.equal(calculateBlockHash(block));
+        }
+    });
+
+    it("Both nodes resolve blocks by the hash the other one reports", async function () {
+        for (const blockNumber of [0, ...inspectedBlocks.slice(-1)]) {
+            const leaderBlock = await getBlock("stratus", blockNumber);
+
+            updateProviderUrl("stratus-follower");
+            const byHashOnFollower = await sendWithRetry("eth_getBlockByHash", [leaderBlock.hash, false]);
+            expect(byHashOnFollower.number, `block ${blockNumber} by hash on follower`).to.equal(leaderBlock.number);
+
+            updateProviderUrl("stratus");
+            const byHashOnLeader = await sendWithRetry("eth_getBlockByHash", [leaderBlock.hash, false]);
+            expect(byHashOnLeader.number, `block ${blockNumber} by hash on leader`).to.equal(leaderBlock.number);
+        }
+    });
+});

e2e/contracts/TestBlockHash.sol

@@ -0,0 +1,47 @@
+// SPDX-License-Identifier: MIT
+pragma solidity ^0.8.0;
+
+contract TestBlockHash {
+    event BlockHashRecorded(uint256 blockNumber, bytes32 blockHash);
+
+    mapping(uint256 => bytes32) public recordedBlockHashes;
+
+    /// @dev Reads the BLOCKHASH opcode for an arbitrary block.
+    /// @return The hash of the block, or zero when it is outside the 256 block window
+    function getBlockHash(uint256 blockNumber) external view returns (bytes32) {
+        return blockhash(blockNumber);
+    }
+
+    /// @dev Reads the BLOCKHASH opcode for the block being executed, which the EVM always answers with zero.
+    /// @return The number of the block being executed and its hash
+    function getCurrentBlockHash() external view returns (uint256, bytes32) {
+        return (block.number, blockhash(block.number));
+    }
+
+    /// @dev Reads the BLOCKHASH opcode for the parent of the block being executed. The parent number is returned
+    ///      along with the hash so that callers can resolve it without racing against newly mined blocks.
+    /// @return The number and the hash of the parent block
+    function getParentBlockHash() external view returns (uint256, bytes32) {
+        uint256 parentNumber = block.number - 1;
+        return (parentNumber, blockhash(parentNumber));
+    }
+
+    /// @dev Same as `getBlockHash`, but executed while mining a block instead of during a call.
+    /// @return The hash of the block
+    function recordBlockHash(uint256 blockNumber) external returns (bytes32) {
+        bytes32 blockHash = blockhash(blockNumber);
+        recordedBlockHashes[blockNumber] = blockHash;
+        emit BlockHashRecorded(blockNumber, blockHash);
+        return blockHash;
+    }
+
+    /// @dev Records the hash of the parent of the block that mines this transaction.
+    /// @return The number and the hash of the parent block
+    function recordParentBlockHash() external returns (uint256, bytes32) {
+        uint256 parentNumber = block.number - 1;
+        bytes32 blockHash = blockhash(parentNumber);
+        recordedBlockHashes[parentNumber] = blockHash;
+        emit BlockHashRecorded(parentNumber, blockHash);
+        return (parentNumber, blockHash);
+    }
+}