feat(chain-midnight): index Midnight signature requests for the MPC

[Pessina]

Why

We are adding Midnight as a source chain for the MPC. This PR is the read path; the publish path lands separately. Stacked on #1075.

What

• New mpc-chain-midnight crate: contract state decoded with Midnight's own ledger crates, the singleton's notification map diffed per finalized block, the carried ledger path walked to the caller's request map, declared-width record decode, request-id recompute, EIP-1559 assembly, epsilon conversion, catchup-then-live indexer under the node supervisor.
• Node wiring: CLI args, off unless supplied. MidnightPublisher bails so requests retry rather than settle falsely; the publish path lands separately.
• Capture-backed tests run real chain bytes through the pipeline, asserted against contract-pinned values; the goldens are regression pins.

What this protects

• A caller only gets a signature over a payload it filed: the request id is recomputed from the record and must match the key it was filed under.
• A caller only gets a signature under its own key: epsilon derives from the address the record was read from, and sender must equal it.
• One bad entry cannot stall the chain: caller-attributable failures are counted drops, caller reads are budget-bounded, and only our-own-schema drift halts indexing, loudly.

Known gaps

• Provenance is not enforced: caller_address is producer-supplied, so a third party can trigger a signature over a record another contract filed (content stays authenticated). TODO in process_entry, scoped as its own PR.
• Responses are not indexed, so requests never settle; deferred with the publisher.
• The notification map only grows, so per-block read cost rises with history.
• A fresh node anchors at the finalized head, so pre-boot requests are never seen (same as Solana and Ethereum, Lack of checkpoint should start indexers at signet contract genesis block #777).
• The path encoding (full 32 bytes, lowercase hex) is decided node-side; the TS derivation reference, the fakenet, and the epsilon goldens move to the same rule before first deploy.

[BRBussy]

Might be worth adding a comment about what could break this in future here, if anything else can end up looking like a 15 entry array (i.e. full chunk size) other than a compiler chunk then this decoding will break down. Some Comment like:

/// Resolve a flat ledger field index to its node in the state tree.
///
/// Chunk detection assumes that only compiler chunks look like FULL 15 (chunk size) entry arrays.
/// That holds today: no ledger type compiles to an array node except List,
/// whose node is a three-slot linked list [element value, rest of list, count],
/// so not 15 length. If a future ledger type compiles to an 15 length array and
/// lands as a contract's last field, the walk counts it as a chunk and every
/// field index shifts. Re-verify when Midnight adds ledger types.

[Pessina]

Superseded by 121416b: the heuristic is gone. The notification now carries the caller's resolved ledger path and the walk follows it via the ledger's own Array::get, so there is no arity inference left to document.

[BRBussy]

I think this degraded read should throw an error instead of just logging. The only way that execution reaches here is when the central state decoded fine but the field walk failed, i.e. schema drift or a wrong central address. So it is deterministic how to get here and what is going wrong, but it is not transient. So from this block onwards we log this error on every block, but we keep emitting Block events, the checkpoint keeps advancing, and the notifications filed at these heights are dropped for good.
If we eventually fix the binary or the address, catchup restarts from a checkpoint that has already moved past everything we never evaluated, so the normal recovery re-walk cannot get them back. The indexer will look alive but just be logging errors while it progresses.
There is already precedent for treating this class of error as fatal: if the central state itself is undecodable, that becomes an Err in central_tree that halts ingestion in the retry loop until we ship a fix. An error here is the same class of problem (our own trusted contract not matching this binary).
On the caller side failures can drop, but on our data side we should halt I think.

[Pessina]

Fixed in a98ecb4: central-field failures hard-error now, and the parent-diff degrade branch is deleted (same drift), so the checkpoint freezes loudly instead of advancing past dropped requests.

[BRBussy]

I think this drop should be an error instead. The map key here is built by our own central contract (SignetMapKey{count, requestId} in signBidirectional), the caller only supplies the 32 byte requestId value and has no way to change the shape of the key. So if a key fails to parse it means this binary no longer matches our own contract's schema. That is deterministic, not a bad entry, and it will hit every entry in the map. Meanwhile we keep emitting Block events, the checkpoint keeps advancing, and all of these requests are dropped for good, with no way for the recovery re-walk to get them back after we ship a fix. Same argument as my comment on the parent None read.

[Pessina]

Fixed in a98ecb4: a key that does not parse as SignetMapKey is a hard error. The singleton mints the key from its own counter, so callers cannot produce this.

[BRBussy]

Same argument as the map key above. The wire shape here is the typed struct {version: Uint<8>, payload: Bytes<128>}, enforced by the signBidirectional circuit's signature, so no caller can file an entry that breaks this decode. A failure here means the binary has drifted from our own contract, which is deterministic and will affect every entry while we keep progressing checkpoints.

[Pessina]

Fixed in a98ecb4, extended to the version byte: the circuit asserts version == 1, so a non-1 version is drift too. Only the caller-writable depth/path bytes remain a per-entry drop.

[BRBussy]

An extra test may be good here to pin the decoding here to real bytes from the chain snapshots. The test can take the real notification entry that is already in the 1366 fixture and run it through decode_notification and unpack_notification_v1, asserting the known caller address and field index. The current tests build their cells with the same widths the decoder expects, so they can't catch the case where both are wrong about the actual contract.

[Pessina]

Deferred to the fixture recapture: the committed 1366 capture predates the path payload (byte 32 is the old flat ordinal 0x04), so it cannot back this test. Queued first on the recapture checklist.

[Pessina]

Done in dde376d: fixtures recaptured on the field-path contract (blocks 63/64, caller state included), and the capture-backed tests now run the real notification, record, parent diff and epsilon derivation through the pipeline, asserted against contract-pinned values.

[BRBussy]

I think we do need to do this confirmation since not confirming that the notification was added as a result of a cross contract call opens up some fringe attack cases such as where a contract stores a SignatureRequest but for whatever reason is not ready yet to request the signature. In that fringe scenario not checking here would allow an attacker to spoof a request to get a signature for that request (which may not be ready yet). Also an attacker could set the index location of the requests map to some other variable location in the supposed caller contract, which would be dangerous if (as an edge case example) the contract authors choose to store "spent" requests, or "cancelled requests" or something in a map. Using events could allow us to dodge this whole thing.

[Pessina]

Agreed it is real, and the path payload widens it: a spoof can name any depth-4 node (e.g. a cancelled-requests archive), and the caller template's own remove circuits make file-then-cancel concrete. Scoping as its own PR since the state-diff path never sees transactions (the commitment join is new read surface); a98ecb4 removes the DoS edge meanwhile.

[BRBussy]

We could read tx_param_type (fixed atom index 7 in the cell) here to add a check to reject unsupported transaction parameter types before doing capacities count. Then if the compact types run ahead of the MPC the rejection error will be clear about the reason for failure (unsupported txn_param_type) instead of a capacity counting error. Then we also have the point set where we can switch on the txn_param_type for the other types we'll add.

[Pessina]

Fixed in 2f4388c: decode_record rejects a foreign tx_param_type by name (atom 7) before capacity recovery.

[BRBussy]

This function, its return type and the consts it uses could be renamed to clarify that they are EVM_TYPE2 specific:

• capacities() → evm_type2_capacities()
• Capacities → EvmType2Capacities
• REQUEST_FIXED_VALUE_ATOMS → EVM_TYPE2_FIXED_ATOMS
• RECORD_HEAD_ATOMS → EVM_TYPE2_HEAD_ATOMS

[Pessina]

Fixed in 2f4388c: EvmType2Capacities, evm_type2_capacities(), EVM_TYPE2_FIXED/HEAD/TAIL_ATOMS.

[BRBussy]

Understanding this function assumes a lot of prior knowledge of how the evm type2 SignBidirectionalEvent flattens. It may help future readers to say in this comment what the function is actually doing: recovering the capacity parameters the caller contract was compiled with (#maxCalldataWords, #maxAccessListEntries, #maxStorageKeysPerEntry) from the record's atom shape alone. Or include a diagram like this somewhere in the file, maybe above the three atom-count constants (), since they are its atom field numbers:

export struct SignBidirectionalEvent<TxParams, #LenOut, #LenResp> {

  FIXED HEAD -- 18 atoms, indices 0..17 (RECORD_HEAD_ATOMS = 18)
  +----+---------------------------------------------+-------------+
  |  0 | sender                                      | Bytes<32>   |
  |  1 | requestNonce                                | Uint<64>    |
  |  2 | keyVersion                                  | Uint<8>     |
  |  3 | path                                        | Bytes<32>   |
  |  4 | algo                                        | enum, 1B    |
  |  5 | dest                                        | enum, 1B    |
  |  6 | params                                      | Bytes<64>   |
  |  7 | txParamType                                 | enum, 1B    |
  |    | txParams: EvmType2TxParams<#W, #E, #K> {    |             |
  |  8 |   chainId                                   | Uint<64>    |
  |  9 |   nonce                                     | Uint<64>    |
  | 10 |   maxPriorityFeePerGas                      | Uint<128>   |
  | 11 |   maxFeePerGas                              | Uint<128>   |
  | 12 |   gasLimit                                  | Uint<64>    |
  | 13 |   to                                        | Bytes<20>   |
  | 14 |   value                                     | Uint<128>   |
  |    |   calldata: Maybe<EvmCalldata<#W>> {        |             |
  | 15 |     isSome                                  | Boolean, 1B |
  |    |     value: EvmCalldata<W> {                 |             |
  | 16 |       selector                              | Bytes<4>    |
  | 17 |       noWords                               | Uint<16>    |
  +----+---------------------------------------------+-------------+

  SCALED REGION -- W atoms, indices 18 .. 18+W-1, always at capacity
  +~~~~~~~~~~+-----------------------------------+----------------+
  | 18 ...   |   words: Vector<W, Bytes<32>>     | W x Bytes<32>  |
  +~~~~~~~~~~+-----------------------------------+----------------+
  |    |     } }                                 |             |

  FIXED ATOM, FLOATING POSITION
  +------+-------------------------------------------+-----------+
  | 18+W |   accessListEntryCount                    | Uint<8>   |
  +------+-------------------------------------------+-----------+

  SCALED REGION -- E x (2+K) atoms, each entry flattened in place
  +~~~~~~~~~~+---------------------------------------+-----------+
  | ...      |   accessList: Vector<E, Entry<K>>     |           |
  |          |     entry.address                     | Bytes<20> |
  |          |     entry.storageKeyCount             | Uint<8>   |
  |          |     entry.storageKeys                 | K x 32B   |
  +~~~~~~~~~~+---------------------------------------+-----------+
  |    | }                                           |             |

  FIXED TAIL -- 3 atoms, sliced off the END (RECORD_TAIL_ATOMS = 3)
  +-------+--------------------------------------+----------------+
  | end-3 | caip2Id                              | Bytes<32>      |
  | end-2 | outputDeserializationSchema          | Bytes<LenOut>  |
  | end-1 | respondSerializationSchema           | Bytes<LenResp> |
  +-------+--------------------------------------+----------------+
}

fixed atoms:  18 head + 1 count + 3 tail = 22   (REQUEST_FIXED_VALUE_ATOMS)
total atoms:  22 + W + E*(2+K)

why the tail anchors from the end: storage keys, calldata words and
caip2Id are ALL Bytes<32>, so no forward width-scan can find the
head/region/tail boundaries; only the tail's fixed count of 3 can.

[Pessina]

Adopted the prose half in 2f4388c: the doc states the function recovers #maxCalldataWords / #maxAccessListEntries / #maxStorageKeysPerEntry from the declared widths, plus the tail-anchor rationale. Skipped the full diagram for comment density; can add it if the prose still reads thin.

[BRBussy]

perhaps name these 3 fields the same as the dynamic sizing parameters from Signet.compact that they carry to help readers:

• maxCalldataWords
• maxAccessListEntries
• maxStorageKeysPerEntry

Named as they are now there could be confusion as well between them and the actual integer no. of used words and access list entries.

[Pessina]

Fixed in 2f4388c: max_calldata_words, max_access_list_entries, max_storage_keys_per_entry.

[BRBussy]

could be worth adding a warning comment here that the field ordering is critical here so that the decoding happens in the right order, and that the field order must exactly match the field ordering in Signet.compact. Something like:

/// Field order here is critical: the struct literal's decode calls run in
/// textual order and each consumes the next atom, so this list must mirror,
/// field for field, the declaration order of `SignBidirectionalEvent` in
/// signet-midnight's Signet.compact. `decode_tx_params` likewise mirrors
/// `EvmType2TxParams`. A reorder between fields of different widths fails
/// loudly on the declared-width check. A reorder between same-width
/// neighbours (chain_id and nonce, the two fees, algo and dest) decodes
/// cleanly and surfaces only as an rid-mismatch drop on every record, which
/// reads like caller fault rather than our bug.

[Pessina]

Adopted in 2f4388c, tightened: cites Signet.compact and names the same-width neighbour pairs.

[BRBussy]

We need to support path being any arbitrary 32 byte array here. render_padded_ascii requires that the bytes, minus trailing zeros, are already valid utf8, which a real path like the erc20-vault's raw commitment hash essentially never is, so those requests get dropped and never signed. We cannot enforce printable bytes on the compact side (well we chose not to for performance), so the path must be treated as opaque bytes and hex encoded directly:

// No trimming first on the path (trimming would make 0xab..00 and 0xab.. derive the same key).
let path = hex::encode(record.path);

Params may eventually have the same problem, although we are enforcing that is blank for now, so possibly fine to ignore for now? Perhaps just a comment there, or asserting again that it is blank here.

[Pessina]

Fixed in e88a975: full 32 bytes, lowercase hex, never trimmed; all-NUL renders as 64 zeros. Nothing is deployed, so the TS reference (epsilon-derivation.ts, fakenet getPath, vault tooling) and the epsilon goldens move to the same rule in lockstep before first deploy. params asserted blank per your side note.

[BRBussy]

The gap-fill here is a direct copy of the catchup walk above so to tidy this up the shared part could be put into a helper like walk_heights(range) that could be used by both? CatchupCompleted event emit could stay at the call site between the two helper calls. Definitely a nit though.

[Pessina]

Done in 121416b: index_height shared by catchup and the live gap walk, CatchupCompleted emitted between the call sites.

[BRBussy]

This classification is a substring match on error text that we generate and return ourselves in contract_state_over in rpc.rs. It would be more robust to return a typed error instead, something like a StateUnservable type we recover here with err.downcast_ref::(). If protects against someone rewording strings in that error string later.

[Pessina]

Deferred: retry_rpc! rebuilds the error from its Display, severing the source chain, so the downcast returns None on exactly the exhausted-budget path. Matcher and attacher already share the one STATE_UNSERVABLE_MSG constant; the typed error needs a core retry_rpc! change, folded into the planned core consolidation.

[BRBussy]

An error reading the caller state here (one that survives the rpc retries) escalates to a block level retry. A caller whose contract state is too large to serve inside request_timeout triggers that, so one bloated contract could halt indexing. The checkpoint will freeze and the watchdog restart comes back to the same block. The node being down does not reach this line (the central read in process_block fails first), so a persistent error here is very likely specific to this caller's state. Perhaps after a bounded number of retries this should become a per-entry drop, charged to the caller like caller-state-undecodable?

[Pessina]

Fixed in a98ecb4: caller-read failures that survive the transport budget (unservable included) drop per entry as caller-state-unreadable; block-level retries stay reserved for central reads. The transient-loss trade is as you stated, and right against a chain-wide halt.

[BRBussy]

It might be worth a one liner here noting that blocks processed after the last consumer checkpoint re-emit their requests on a restart, i.e. a comment indicating that this side of the channel intentionally provides at least once delivery, and that that is intended behaviour, with dedup handled downstream by SignId in the node's request handling. So nobody should try to dedup emissions in this file.

[Pessina]

Added in a98ecb4: emit_block's doc states at-least-once delivery, SignId dedup downstream, never here.

[isSerge]

Do you plan to add integration tests in this or following PR?