feat(request): deterministic proposer rotation with scheduled round timeout - #1051
feat(request): deterministic proposer rotation with scheduled round timeout#1051yvonneanne wants to merge 3 commits into
Conversation
3951ff6 to
382bfe1
Compare
8834349 to
950e9d6
Compare
382bfe1 to
0390ebb
Compare
950e9d6 to
41a5ffb
Compare
6593179 to
c728123
Compare
Thanks! I don't think #1050 points at a bug in The fair concern underneath is that divergent views are normal in production. So the line here is: peer-identical decisions (proposer, round, deadline) derive only from (round, participants, entropy); local liveness is still fine for single-decider or local choices (mesh gate, presignature intersection, addressing). Maybe we should make divergence observable (export active-set size + membership hash per node) so we can see how often views actually disagree? |
|
@yvonneanne Agreed. Yes, we need to understand whether the active set actually differs in practice and, if it does, why. We have an endpoint that shows the node's state, but a long that is triggered only on active set change is the simplest option. |
16b0be6 to
833f045
Compare
098d2ae to
2086b5b
Compare
…imeout Proposer election read the local active set to skip over peers it believed were down. Nodes with divergent active views then elected different proposers for the same round, so accepts never reached threshold and the request never settled (#907). Elect from inputs every node shares instead — round, membership, entropy — and drop the search window. Rotation is a plain offset-plus-round modulo `n`, which visits every participant before repeating, so a request still reaches a live proposer within `f + 1` rounds. Round deadlines now come from `round_timeout(r)`, a pure function of the round: round 0 keeps the full `ORGANIZE_POSIT_TIMEOUT`, later rounds start at a floor derived from `ACCEPT_POSIT_TIMEOUT` and grow back up to it. Peers that agree on the round agree on the deadline. Un-ignores test_divergent_active_views_still_converge.
Rounds grew by a fixed 1s step and stopped at ORGANIZE_POSIT_TIMEOUT. Both parts break recovery from staggered indexing: nodes that observe a request at different times enter the same round at different moments and can only transact while both are inside it, so a round must outgrow that skew. A ceiling below the skew leaves them permanently offset, since both advance at the same rate; and an additive step clears the skew by one step however long it has been growing, leaving a window too short to complete a round. Grow by 23/20 instead and cap at a separate, much higher ROUND_TIMEOUT_CEILING. Any ratio above 1 eventually crosses the skew, so the factor is set by what it costs the other case: rotating past dead proposers wants rounds short. At 23/20 a round only doubles after five rotations (2, 2.3, 2.6, 3, 3.5 against the previous 2, 3, 4, 5, 6), so early rotation is cheaper than before, while a 60s skew is crossed at round 26 with 5.8s of overlap rather than never.
… wait min_viable was the deliberator's post-Accept obligation, not the cost of completing a round — the assertion let a round through that cleared the skew but couldn't fit a Propose broadcast plus accept gathering. Bar on ROUND_TIMEOUT_FLOOR instead, and check that a larger skew gets a wider settling window, which a fixed step can't do but a ratio can. Also tighten the window-scaling guard to the ceiling minus a full round, the actual bound settling_overlap needs to succeed, and document the tolerable-skew bound on ROUND_TIMEOUT_CEILING itself.
1630f29 to
3ef8353
Compare
jakmeier
left a comment
There was a problem hiding this comment.
Overall great changes, as discussed in our call
Closes #907. Realizes the design in #228; original intent of #182.
What
organize.rs): drop the forward scan for a locally-active proposer, the entropy-seeded fallback, and thestate.round = selected_roundjump.proposer(r)is now a pure function of(round, participants, entropy);state.roundadvances only throughbump_round.mod.rs,state.rs):T(0)keeps the generous inherited budget;T(r ≥ 1)starts at a short floor and grows by23/20up to a separate ceiling, so rotating past a dead proposer no longer costs a full round-0 wait.wait_for_active_participantsis untouched — it gates supply, not election.proposer_per_roundreduces both operands before adding:roundderives fromhighest_seen_round, which a peer sets unbounded, so the bare sum overflows.500msliteral (ACCEPT_POSIT_TIMEOUT + 500ms); it is now2 * ACCEPT_POSIT_TIMEOUT, and the schedule's floor is asserted to exceed it.Why
Cost, deliberately paid
T(r)before rotating, where the local filter used to skip it free. That is the price of peer-identical election; skipping needs the local knowledge being removed.T(r ≥ 1)being short is what buys the cost down. Rounds 1..18 are shorter than the old uniform 20s, so an alive-but-contended proposer (permit or presignature wait) has less time than before.23/20over doubling: most failed rounds are dead-or-unable proposers, where the cheapest early rounds waste least, and rotation changes proposer every round so doubling would penalise a healthy node for its predecessors. A round only doubles after five rotations. Multiplicative over additive, though: a fixed step clears a late node's indexing skew by one step however long it has been growing, leaving a window too short to complete a round in, and a ceiling below that skew leaves the two permanently offset since both advance at the same rate.Open question: per-chain
T(0)?T(0)decides whether this change is free or a latency regression, and one global value serves budgets differing by three orders of magnitude.Chain::expected_response_time_secs: NEAR/Solana 11s, Hydration 29, Canton 35, Ethereum 3605, Bitcoin ~4805. AtT(0) = 20sa single dead round-0 proposer is over budget on NEAR and Solana; the regression fires with probability ≈ (nodes down)/n.chainis on-chain request data, so a per-chainT(0)stays peer-identicalT(0)sized for the tightest chain / derive fromexpected_response_time_secs/ a per-chain constant besideexpected_finality_time_secs. Flagging, not choosing.Known exposure, not introduced here
entropyis requester-supplied, andentropy[0]alone fixes the rotation offset, so a requester can grind for a cycle that front-loads nodes it believes are down. Bounded (one round each) but repeatable per request. The local filter used to mask it.Using an input fixed after submission works (block hash), which
IndexedSignRequestdoes not carry. This is a separate issue, not addressed in this PR.Tests
round_timeout_schedule— every round outlasts the post-Accept wait, rotation is cheaper than round 0, the cap is reached.rotation_reaches_live_proposer_within_f_plus_1_rounds— exhaustive over down-sets and start rounds forn = 4, 5.n = 4is load-bearing: a stride sharing a factor withncovers only half the participants, which an oddnmasks.