Skip to content

Release v0.15.0 - xchain-e2e-test - #9

Merged
jdogresorg merged 110 commits into
masterfrom
release/v0.15.0
Sep 7, 2026
Merged

Release v0.15.0 - xchain-e2e-test#9
jdogresorg merged 110 commits into
masterfrom
release/v0.15.0

Conversation

@jdogresorg

Copy link
Copy Markdown
Contributor

Cuts xchain-e2e-test for the v0.15.0 release train. 106 commits from master.

Added

  • The ATTEST response-mirror acceptance venue: five real hubs and two mirror-fed indexers, a per-table mirror fault injection, and the AT0 to AT6 drills with a shared staking prologue and wedge-clearing waits.
  • The ROLLCALL acceptance suites drive eviction, rollback, cooldown re-entry and the publish paths on a shared two-chain regtest venue, with a seeding tool and an outsider-clearing tool.
  • Responsible-set widening is driven on a chain from both sides of the ladder, and the ATTEST per-block and per-contract admission caps are driven live.
  • Capability and contract slashes are asserted to release escrow, and capability slash coverage aligns with the buried snapshot height.
  • Template DEPLOYs route through the size-aware chunked path, and chained MINTs from one address are pinned to distinct transactions.
  • A swallowed wait fails instead of passing on a missing row, and CI lints for the pattern.
  • Fixture stakes are now released when a run ends, so a run leaves the venue's capability set no larger than it found it.
  • The suite reports, and can fail on, any staking key a run leaves behind in the shared venue.
  • A gate that fails any hand-built STAKE broadcast which bypasses the release ledger.

…ank ladder

Driving the eviction suite against a real venue found four things, three of them
in this harness and one of them a fact about the protocol that the harness now
has to respect.

THE IDLE KEY IS PER-VENUE. An eviction stamps deactivation exactly as an unstake
does, and the STAKE v1 admission rule refuses a pubkey that carries any valid
stake row ever - it asks for the active stake with a null block index, which
drops the activation and deactivation clause entirely, and it is keyed on the
pubkey with no source term. So an evicted signing key can never be staked again,
and a fixed idle seed made the eviction test one-shot per venue: the first
successful run burned the key and every later run failed to seed. The three
SIGNING seeds stay hard-coded, because they are the frozen vector's own signers
and a harness that can sign for them is a harness that agrees with the vector.
The idle one now resolves from an explicit seed, else from the federation
mnemonic and a generation counter, else the legacy value with a warning saying
the venue gets one run.

THE IDLE SOURCE ADDRESS ROTATES WITH IT, which is a second requirement rather
than tidiness. Absences are recorded per source, and the lookup has no term
excluding rows that predate the source's current stake, while the streak window
is simply the last few rolled epochs. A source that re-enters at the same
address therefore inherits its own history and one further absence evicts it
again immediately. A new generation gets a new address as well as a new key.

THE PUBLISH PHASE NOW CLIMBS. The rank ladder unlocks with chain height, and the
drive was ticking three times at a fixed height, so only the lowest ranks could
ever publish. With the elected leader being the hub the test deliberately
silences, and the hub holding every signature sitting at a rank that never
unlocked, exactly one signature reached the chain and the epoch closed below
threshold - which read as a protocol failure when it was the harness holding the
chain still. It now mines forward through the phase and stops when the far side
actually holds every signature it expects, which is also what a live venue does.
Two smaller repairs come with it: the coverage check ignores null entries,
because that read echoes every key it was asked about and puts null against the
ones it has no signature for, so counting keys counted absences as presences;
and a stalled tip whose next block is a close now mines the other chain and
retries, because both stamps are wall clock and the far side can be short by as
little as one second - which is also how a failed drive wedges the venue for
every later run.

Adds an opt-in per-hub trace, because every one of those explanations produces
the same single log line, and the engines already hold the answer. Adds a
precondition that refuses a venue whose idle source already carries absences,
rather than reporting a correct eviction as a failure.
The indexer's match-scoped status writer clears a fully paid COINPAY match to
'valid'. This assertion pinned the opposite, so it failed on the first bitcoin
run carrying that writer, while every other assertion in the case passed.

The load-bearing point is restated rather than dropped: the fill is selected
from the coinpays row, never from the match's own status. The clearing is
forward-only, so a match settled before it holds 'pending_coinpay' permanently,
and a selection keyed on match status drops exactly the historical fills a
price window depends on.
Levelling develop with master, the ceremony step that closes a train. Commits
that reach master without coming back leave the branch all work lands on missing
what shipped, so the next diff off develop reads as though it never happened.
…t names

The staged snapshot named xchain-hub 0.12.0 while the sibling repo declares 0.12.3, so the advisory check that compares the two refused the push. The snapshot cannot be re-staged from here, so the version field is corrected to what the sibling carries.
…king keys

A fixture STAKE joins the venue's real capability set, so a run that never gives it back grows the set permanently and dilutes the operator hub's weight share until checkpoint quorum is unreachable. A release ledger books every stake created through stakeHelper, a root afterAll broadcasts an UNSTAKE per outstanding key, and a gate script refuses hand-built STAKE payloads that bypass the ledger.
…ead of minting

The capability and VM-contract slash paths had no end-to-end coverage, so the first real run of either would have been in production. Both now drive a live chain and assert the escrow is released rather than minted.
…eight

Membership and the delegated owner resolve at the buried snapshot height, so the coverage reads the same height the indexer does.
…ed election and assert the whole attestation fee
…ladder

Reproduces the failure the ladder exists for rather than a model of it. The
regtest chain already carries staked attestation validators from earlier runs
whose keys nobody holds, so a base set can genuinely contain a member that never
signs, exactly as a live one did.

Both phases run against the same request, the same signers and the same payload,
and only the chain height differs, because a test that showed only the second
half would pass just as well against a set that was wide all along. Inside the
first segment a response carrying one out-of-set signature must be rejected and
leave the request pending; past the segment boundary the identical signer set
must be accepted, the request fulfilled and the contract callback fired.
…eads

The HUB_DB_* environment this process gets is a model of the indexer's
configuration, not the configuration itself, and a venue that sets HUB_DB_NAME on
the indexer alone moves every price lookup to the hub database while the fixtures
keep writing to the indexer's own. Ask the indexer where its price reads land and
pin that answer, then check once after the bootstrap seed that it can see it.
…l clocks

The three standalone DOGE setup drivers each seeded one price row at the indexer's
parsed tip, which on an idle regtest chain is already outside the freshness window
by the time the driver's own blocks are mined, so every priced action indexed as
having no current oracle price. Share one helper that writes both anchors, retracts
the fixture rounds that would otherwise shadow them, and re-seeds before each submit.
…e roster

A staked oracle_publish member the harness does not run wins the leader election
some of the time and never publishes, and its weight counts against quorum in every
epoch, so the acceptance suites report on the wrong subsystem. Fail the precondition
with what is extra and what it costs, arm the regtest rail for the harness itself,
and keep the seeding tool from releasing the stakes it just placed.
Three MINTs fired a second apart can be handed the same input twice by the
utxo-tracker, which rebuilds a byte-identical transaction with the same txid. Drive
both the indexed chain and the rapid rebuild against a stale tracker view.
…s responses

The indexer's response verification sits in its own module beside the handler,
so the gate call and the handler are in different files. The case asserts the
input the gate receives, so it reads both.
…apability

A hub refuses to start when a capability the coins registry names has no
configured MIN_STAKE, so the harness config carries full_node at its canonical
value.
The setup hook requires the sibling checkout and builds an isolate, which
costs seconds on an idle box and far more on a busy venue. The contract cases
themselves finish in milliseconds.
The existing multi-validator helper federates hubs over P2P only and
serves no hub API, so nothing in the e2e tree could exercise an indexer
following a hub through the mirror. This venue spawns five real hub API
children at attestation redundancy three, each fronted by a byte-level
P2P proxy so one hub's gossip can be delayed without touching its HTTP
surface, and two real BTC regtest indexers each following a different hub
with the regtest grace and forward-margin overrides set so a response
binds in seconds. The smoke test boots it, checks every snapshot route
answers at the current schema version, and checks both mirrors caught up.
The venue hubs run their own oracle and it publishes a coin price about ninety
times the harness's canonical one, measured on the venue: 7,679,017 against
79,666, with the XCHAIN pair off by more still. A price that wrong is worse than
no price at all. The venue node computes a native fee two orders of magnitude
above what the harness paid, since the harness prices its fees off the standing
node, and rejects the deploy for an insufficient fee.

That is the same divergence class as the missing price, one layer down: two
nodes disagreeing about the same block because they hold different price data.

The readiness barrier now requires the price to fall inside a wide band around
the canonical value rather than merely to exist, and names the offending value
when it does not. The band is deliberately loose: the drill does not care what
the price is, only that both nodes compute comparable fees from it.
…he venue crossing two runs

Five defects, every one of them in the instrument rather than in protocol code.

ROLLCALL. A hub may publish only at rank <= floor((btcTip - E) / ELECTION_TOLERANCE),
so the AT9 leg's two ticks at E + 6 unlocked ranks 0..2 and then waited two minutes
for a rank-3 signature no hub was allowed to send. driveEpoch's rank climb is
extracted as climbPublishLadder and both callers use it, with a height ceiling so
AT9 can still mine its window end after the DOGE freeze.

The gossip wait could not see an indexer that had HALTED: a hub signs only once its
own indexer can answer ledger_hash(E), and that indexer parks on any close whose
DOGE evidence it cannot prove. It now mines DOGE while it waits, the cure mineBtcTo
already applies to the same halt.

AT10's reward-pool funding never ran. The balance read joined a table that does not
exist and called a method the DB module does not export, and its catch turned both
into null, which the caller read as "no funding needed": the pool held 2 XCHAIN
against a 120 XCHAIN claim with nothing in the log about the pool. The read is now
loud, the funding sizes on the leader's whole unclaimed total (a COLLECT with no
AMOUNT claims all of it), and both legs mine while they wait for their row.

ATTEST. Each run gets a fresh hub database whose price_snapshots ids restart at 1
while a reused venue keeps its mirror, and the mirror's price upsert excludes the
natural key from its ODKU. Colliding on the PRIMARY key instead makes MariaDB write
the new row's PRICE onto the OLD row's PAIR: the mirror served BTC/USD = 7,676,531
while all five venue hubs held 79,603 for that round, and the seeded values sat
under BTC/AUD and BTC/CAD. The venue node priced its deploy off that and refused the
fee. The mirror database is now rebuilt per run, and the readiness barrier reads by
round_number, the ordering getLatestPrice actually uses.

Driven on the regtest venue: AT9 and AT10 green, 2 passing / 0 failing, and the
attest venue's price barrier passes for both pairs on both indexers.
…g to the block cadence

Both helpers broadcast and then poll for their row on a fixed budget, and that
budget extends only when the indexer is visibly behind or still writing. A
transaction waiting for a BLOCK trips neither: the indexer is at the tip, idle,
and right. Measured on the venue, the deploy leg gave up at 60s with the indexer
one block behind and the indexer logged that very transaction valid moments
later, which reads as a failed deploy and is a lost race.

Mining under the wait removes the dependency without touching the wait's budget
or its diagnostics, and re-broadcasts nothing, so the paragraph above the deploy
still holds: there is exactly one contract.
…of replaying the chain

A venue indexer that replayed this chain from genesis produced a different
ledger. The same contract in the same block 3895 was action index 791 on the
standing node and 586 on the venue one, both counters dense, 139 blocks
disagreeing. By action type the divergence was almost entirely ATTEST, 119
against 17, with the callbacks, rewards and evictions those responses caused
following downstream and every purely chain-derived type identical.

Neither node is wrong. A mirror-era response is a system-synthesized action with
no transaction of its own, derived from a mirror row the node held at the time,
so a node replaying today cannot re-derive one and every action index after the
first divergence shifts. The on-chain batches exist precisely so a chain-only
node can rebuild that history, and none has ever been published on this chain.

The cost was the whole ladder: the drill composes its EXECUTE with the standing
node's contract index, the venue node answered that the contract index was
unknown, no request existed there for a response to bind to, and the resulting
silence read as an applier that does not work.

So the chain database is copied from the standing node, across two connections
because the venue account and the indexer account each own only one of the two
databases and neither should be widened. Reuse is kept, but the test for it is
agreement rather than mere presence: a venue database that holds a different
number of actions at a common height is a different ledger, not lag, and is
rebuilt once rather than carried into another run.

Driven: both venue indexers agree at 1233 actions and catch the chain in five
seconds, the EXECUTE is valid on both, and both apply the same tx-less response
actions at the same block indices.
…med to do

mineWhileWaiting was parsed into minePerPoll, mineCap and mined at the top of the
wait loop and then never read again. Three paragraphs of comment described the
behaviour; no line implemented it. Every drill that asked for it waited on a
chain nobody was moving.

On an idle regtest chain that is fatal rather than slow. Applying a mirror
response happens inside the block loop, so a response can be delivered, valid and
applicable, and still never apply because there is no next block to apply it in.
That is the applied [null,null] this ladder has been reading as a broken applier.

Two smaller things in the same path. AT1 was the only drill passing a bare number
where the option is read as an object, so even once the loop mines it would have
mined nothing; the wait now refuses the wrong shape loudly rather than silently
disabling itself. And the fixture's own mine-while helper no longer adds blocks
to a node that is already behind, which had pushed the indexer 49 blocks back.
… its reward read requires

Two defects found by auditing the five undriven drills instead of discovering
them one failed run at a time.

Every drill broadcasts its EXECUTE through a helper that polls on a fixed 60s
budget and extends only when the indexer is lagging or still writing. A
transaction waiting for a BLOCK trips neither, so the wait expires while the
transaction is valid - the same race already measured and fixed for the deploy in
the fixture. All eight call sites now mine under the wait.

And at2 passed a bare block number where readAttestRewards reads
{blockIndex, roundReference} and refuses an unscoped sweep. The guard fires
before any reward row is compared, so the case that rewards are paid identically
on both indexers was never actually checked; it died on an unrelated assertion
every run. This is the third instance of the same shape this week, after
mineWhileWaiting and the balance read, which is why the wait now refuses a
malformed option loudly rather than disabling itself.
…instrument faults that hid them

Every red in the ladder since the venue first booted was the instrument or
the venue host, never the mirror, the applier or the hubs, until AT5's
last one (a hub signer-order defect, landing separately with AT5). Found by
running all drills back to back and reading the venue indexer's and hubs'
own logs first:

- waits: the assertion message on an applied row threw on a BigInt (mariadb
  BIGINT), reporting the first success as a TypeError; jsonSafe everywhere a
  row is serialized. feeLines prints the indexer's fee-settlement lines on a
  reward assertion, so an empty split and a settle that never ran read apart.
- at2, at5, at2b: the applied wait mined nothing on an idle chain (the AT1
  shape); now mined under, capped by the request's own window.
- at1: per-provider deadlines (llm allows 20 blocks, not 60), no mining under
  the row wait, the applied cap inside the window, and the hub OAuth token
  forwarded to the hub children when the harness has it.
- at2, at6: requests escrow a fee the split can divide. Regtest XCHAIN is
  issued with 0 decimals, so 2 XCHAIN over 3 signers floored to 0 per signer
  and wrote no reward row while the indexer logged the split; 6 leaves a
  whole share at every set size the widening ladder reaches. at6's residual
  bound now floors at the chain's decimals, and it read the broadcast hash
  off a string as .txHash.
- at4: the remove-reorg exceeded the standing utxo-tracker's 12-block undo
  window and halted it for every drill after; burial 4, nudges only onto a
  level indexer, no mining under the row wait, and a guard that refuses a
  reorg deeper than the window instead of halting the venue.
- at2b: the delayed indexer parked by name before the drill polled and the
  poll never saw it; no backlog under the prompt wait and a grace wide
  enough to read.
- at5: the venue wait ran before the venue existed; the DOGE rail was read
  out of scope; the signer probe reads process.env, so the staged variables
  are placed there for the load only; the test server's body is now
  deterministic per URL (random bytes per fetch gave every hub its own body
  and no round could reach quorum); the chain tip is pushed as BTC, judged
  on the hub's status rather than a null check, and re-pushed every 3 s from
  a BTC connector captured before the first rail switch; DOGE is mined one
  block per poll under the marker wait so each window's change confirms;
  the publisher is funded with 20 DOGE.
- at3/at4 teardown: a failed miner resume is printed, not swallowed.
- venue: keyset pagination on the primary key for the seed instead of
  LIMIT/OFFSET, which re-read every table from row zero per page; and with
  ATTEST_VENUE_LOG_DIR set every hub and indexer child also logs to disk, so
  a run killed by an outer timeout keeps its explanation.
- db: row waits gave up at a lag of 1 or 2 blocks (threshold 2), most often
  when nearly done, and parseInt(env) || 2 swallowed an explicit 0.
  Threshold 0, explicit 0 honoured, boundary cases under test.

SCOREBOARD (BTC/DOGE regtest, serialized attest venue, 2026-09-05)
- AT1 http_get + llm: GREEN 2/2 (pass 5, 17:06Z)
- AT2 dissemination/determinism: GREEN 1 passing + 1 pending by design
  (pass 7, 18:01Z); at2b forward margin GREEN 2/2 (16:26Z)
- AT3 deadline: GREEN 2/2 (15:39Z)
- AT4 reorg: GREEN 2/2 incl. the harness barrier (pass 5, 17:10Z)
- AT5 batch/replay: reaches its subject (rounds finalize on 5/5, windows
  publish on DOGE); blocked on the hub's signer order, next commit
- AT6 flag day: GREEN 2 passing + 1 pending by design (pass 6, 17:58Z)
…e the venue

The sixth drill, after the hub fixes it found (signer order on the mirror
row, capability snapshot at the batch anchor). What the drill and the
venue needed on top:

- at5: the publisher is funded as forty independent outputs, because the
  encoder spends confirmed outputs only and every empty window re-spends
  one large output into unconfirmed change; the chain tip is pushed from a
  BTC connector captured before the first rail switch, and DOGE is mined
  one block per poll under the marker wait, in sequence, never on a timer
  that races the rail switch; the forward margin is 90 s, above a round
  that runs through a leader-slot timeout, because a row is written at
  finalization and a window closes a margin after the leader's clock.
- venue ATTACH MODE: a second venue for another coin spawns indexers that
  follow an existing venue's hubs and share its hub database, spawning and
  stopping no hubs of its own. AT5 attaches one DOGE indexer, seeded from
  the standing DOGE indexer inside the rail switch, and reads its batch
  actions there: only a node mirroring the federation holds the capability
  snapshot a v5 head is judged against, and the standing DOGE indexer
  follows the standing hub.
- decoder credential: the environment's DECODER_DB_* apply only when their
  user is the decoder's own (the rail switch swaps COIN and leaves them),
  and the sidecar search continues past a `.local` that lacks the key to
  the coin config beside it.

Driven: rounds finalize on 5 of 5 hubs, the window co-signs, lands on DOGE
as a valid v5 head plus a v6 continuation, links back onto both mirror
rows, and an empty window lands as a valid row_count 0 head (the second
case is green). The first case is not yet green as one run: when a round
re-runs under more than one leader slot, hubs finalize on proposals whose
signed effective_time differs, and the batch co-sign refuses the window
(`differs on effective_time`). That is a protocol convergence rule the
spec has to decide, recorded on the ledger; the drill is complete up to it.
…t-side

Every one of the four had an indexer verdict already written for it, so
none was the missing-row pipeline gap they looked like from the suite log
alone. Read from the dogecoin leg's indexer diagnostics:

- capability SLASH: `STAKE : amount=400.00000000 : invalid: ACTION (BTC
  only)`. Capability staking is BTC-only by protocol design and this suite
  runs on all three coins, so its bond can never land off Bitcoin. It now
  skips off BTC, the same guard and the same reason the COLLECT suite
  carries. This file was created on develop, which is why it had never
  passed rather than regressed.
- gated FILE: the transfer LANDED VALID, with its BATCH and MESSAGE, and
  the test's own wait missed it. `checkSend` turned `memo: ''` into
  `im.memo = ''`, which never matches the NULL the indexer stores for an
  absent memo; checkMint and checkList have carried the IS NULL branch for
  as long as the parameter has existed and checkSend never got it. Fixed in
  the helper, not the caller, so the trap is closed for every future
  caller. The wait is also pinned to the batch's txHash: the suite sends a
  permitted transfer and a refused one with the same source, destination,
  tick and amount, so those fields alone name two rows with opposite
  verdicts.
- OWNERSHIP, both cases: the actions were refused exactly as the tests
  intend (`invalid: TICK (ownership escrowed)`), and each test asserts on
  that refusal itself. They sent it through helpers that demand
  status=valid, so the row requirement threw inside the helper before the
  test reached its own assertion. They now use raw senders, the pattern the
  same file already uses for a refused child ISSUE, and issueHelper gains
  sendIssueV1Raw and sendIssueV5Raw beside the existing sendIssueV0Raw.

No product code changed. The row requirement itself is correct and is what
surfaced the last two.
…racts

Review-round fixes.

The encoder connector could not send an API key, so a keyed encoder killed the
harness at bootstrap rather than at a corner case; the key is now wired at all
four real construction sites, not the two the recommendation named.

The sleep-flake scanner read only the direct call forms, so the inline
awaited-timer promise shape used by four helpers was ungated, and the committed
baseline had been written from that blind count. Replaced the pattern match with
paren-balanced parsing and re-derived the baseline from the widened detector.

Suite: unit lane green; the stubbed integration lane's failures are pre-existing
and identical under the old and new globs, which is how they were ruled out.
The blocks this case mines are stamped at expireAt, so they are future-dated by
however far ahead of wall clock it sits, and the indexer's anchor-attest barrier
holds a block until the hub's wall-clock watermark reaches that stamp. Every
second of margin here is a second the whole bitcoin parse pipeline stalls for
real.

At the shared-ledger 7200s window plus the old 600s margin that was 2h10m, and
run 34015867460 spent 2h08m50s of a 289-minute leg deferring one block on it,
which is also what made thirty downstream waits give up. The venue now sets a
300s window, and sixty seconds past the deadline is ample to carry a block over
it.

The reason is written next to the number, because the number looks arbitrary and
the cost of raising it is invisible from here.
Nine cases across the attestation suites wait for an attestation_responses row
produced by an on-chain ATTEST v1 broadcast. Above the response-mirror activation
height the indexer refuses that transaction outright, so the row can never
appear and the case dies on a timeout that says nothing about the product.

Regtest is armed at 0 and the resolver is a plain height comparison over a
module-level constant with no environment variable, no config key and no
injected seam, so every regtest request that can exist is mirror-era. There is
no legal block index below 0. These cases are legacy-era by construction.

They cannot be migrated either. In the mirror era a response is not broadcast at
all: it finalizes over P2P across a hub federation with a staked responsible set,
which is the venue the AT0 to AT6 ladder runs on. The three-coin matrix stack has
one hub and no validators, so nothing there could mint a response row. The
mirror-era coverage already lives in that ladder.

So they skip, with the reason stated where the skip happens, matching what the
flag-day drill already does for its own unreachable case. Measured on the
2026-09-06 matrix: these nine were 9 of the bitcoin leg's 40 failures, and the
indexer log carried ten refusals against zero response rows.

The activation map is copied locally because it decides only whether to skip and
cannot fork settlement, and a unit test asserts it equal to the indexer's
vendored copy so a flag-day edit cannot silently re-enable cases that still
cannot pass.
…d at what the positive direction budgets

controllerPolicy and vmContractCustody wait for the indexer verdict on the rejected
action (its actions row) instead of sleeping, with the bound taken from the same
file positive-direction wait (25000ms for orders, 20000ms for sends, 30000ms for
the custody cancel) rather than the 3-7s the old settles guessed. oracleBatchTamper
waits on the mined-height condition. consensusWait gains waitFor, a rejecting
poll that reports its last observation. The two oracleBatchReplay settles are kept
on purpose: the later extra.length === 0 assertions observe an absence, and a poll
that returns at the >= floor closes that window. Baseline 38 -> 31.
…olds

A request whose round finalized under two leader slots holds two honest mirror
rows, each of which rides a window and gets its own link, so the drill checks
all of them instead of whichever the planner returns first.
…arrier cycle

The six waitForTxIndexed bounds in controllerPolicy and vmContractCustody go from
20 to 30 seconds to 120 seconds. Measured on two venues: the indexer defers a freshly
mined block for a full 60-second barrier cycle plus retry (cross-chain match sync on
the regtest venue, attestation mirror on the hosted runner, 21 deferrals and 22
blocks over 30s in one run), and an untouched 90-second loop in the same file failed
under it. The bounds are bounds, not floors, so a healthy venue pays nothing.
…ive helper

The case asserts that a second v1 stake on an already-active pubkey is
rejected, but it sent that stake through the positive-path helper, which waits
for a row at status=valid and throws when none lands. So the case could only
pass while the rejection was broken, and it failed the moment the rejection
worked: it was the single failure in an otherwise green bitcoin matrix leg.

The poll's own line separated the two candidate explanations without guessing,
and is quoted in the test so the next reader does not have to re-derive it:
the indexer was current at zero blocks of lag and had refused the duplicate,
rather than a slow venue that never indexed it. Rewritten on the pattern the
sibling rejection case in this file already uses.
The lockfile snapshots each staged sibling's own version, and the hub's develop
moved to 0.14.0 when last train's master was levelled into it. The snapshot
still read 0.12.3, so the sibling-tree advisory that exists to catch exactly
this drift was the last failure standing on this repo's gate.
…w wait that admits variants

The batch drill capped mining per wait, so its four waits could mine three
times the request deadline; on a slow venue that expired the second request
while its round was still finalizing, and a row that landed seconds later
could never bind. The budget is now measured against the earliest emitted
deadline from the current tip at each wait, with the same headroom as before.

The mirror-row wait asked for exactly one row per request on both venue
indexers. A round that finalizes under two leader slots now legitimately
leaves two, and which one binds is the applier's tie-break, so the wait
accepts one or more and the applied wait proves the choice.
…nd a row

The one mining budget the batch drill shares was spent almost entirely by the
stage that waits for mirror rows, leaving the stage that waits for them to
apply with nothing to mine. That looks like a passing budget and is not: a row
binds at the first block whose protocol time reaches its signed effective
time, and off mainnet protocol time is median-time-past, which trails the tip
by half its window, so several blocks must be mined after the row exists
before any node can apply it. The mirror stage now stops twelve blocks short
of the deadline and the applied stage may mine the rest, up to the deadline
block itself, which still binds.
…and give it room

Two corrections from pass 24, which ended with the rows eligible and unbound.

The applied stage reserved one block below the deadline. A row satisfied at the
deadline block still binds, since the expiry sweep's predicate is strictly less
than, so that reserve was the only thing standing between the run and a bind.

The deadline itself was the larger error. Sixty blocks is ample for the
widening ladder but that is not what spends them: a row becomes applicable on
protocol time, which off mainnet is median-time-past, so the applied stage must
mine slowly enough for a median over eleven blocks to climb past a stamp set a
forward margin ahead, while every transaction the drill sends mines a block of
its own. Pass 24 reached the deadline with the stamp 269 seconds overdue and
every block in the median window stamped seconds apart.
@jdogresorg jdogresorg changed the title Release v0.15.0 Release v0.15.0 - xchain-e2e-test Sep 7, 2026
…d for

The unit tier could not start: mocha failed to load mathjs, and five more
modules were in the same position. Each is imported by this repository's own
test code and none was declared, so they resolved only because a sibling
checkout happened to provide them. A clean install has no sibling, which is why
this reds on a hosted runner and passes on a venue.

The versions pinned are the ones the tree already resolved, so nothing moves;
the declaration only makes the existing resolution the one that is asked for.
…code from

The unit tier died at module load on `Cannot find module 'mathjs'`, which reads
as a missing dependency in this repository and is not one. A unit test reaches
the attest-mirror venue helper, which reaches the attestation helper, which
requires the indexer's stake-weighted quorum module, and Node resolves that
file's dependencies from the indexer's own tree. The indexer was checked out
but never installed, so the whole tier failed before a single test ran.

The checkout comment predicted this precisely, naming a loud module error from
the indexer's tree as the cue to add it to the install loop. The live tier
already installed all three, which is why it stayed green throughout.
…it reaches

Installing the indexer fixed the module error it was named for and uncovered
the next one: the sdk helper resolves the SDK from paths beside this repository
and throws its own error when none answers, and a unit test reaches it through
the attest-mirror venue helper. The tier died at module load again, on a
message that reads like a missing dependency here and is not one.

This is the last of them rather than the next in a series, and that is measured
rather than hoped: run against a checkout with every sibling present and
installed, the tier reports 1193 passing and 7 pending. The one remaining
failure is a history-scanning guard that cannot see anything on a shallow CI
clone, and it is unrelated to module resolution.

The sdk needs no deploy key because this repository is public now, unlike when
the two keys above it were added.
The lockfile pinned xchain-hub at 0.12.3 and xchain-sdk at 0.12.0 while both
siblings now declare 0.15.0, which freezes the nested ranges those snapshots
carry. The trees cannot be re-staged on a Mac, so the recorded versions are
corrected, which is the remedy the failing check names.
Operator ruling 2026-09-07: the point of this train is to exercise the response
mirror and roll call on testnet, and a train that ships them dark there is not
worth cutting. The mirror is armed at block 151324, the chain tip when the
ruling was made, so it is active the moment a node updates rather than waiting
on a future height. Roll call needed no change: it was already armed at 151200,
which the chain passed some time ago.

Mainnet stays unratified for both, so its behaviour is byte for byte unchanged.
On testnet this changes state derived from existing bytes, so the changelogs now
carry an Activation section saying so, and every hub and the indexers following
it must update together rather than one at a time.

The activation map is mirrored in five places and all five move together: both
service copies, the documented canonical, the vendored copy the test helper
reads, and the assertion that used testnet as its example of an unratified
network, which it no longer is.
@jdogresorg
jdogresorg merged commit 33f797f into master Sep 7, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant