diff --git a/CLAUDE.md b/CLAUDE.md index 42ca59e7..4e8e699b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -333,10 +333,51 @@ npm run clean participation opens reveal vote gates late or never (withheld at slot end) -4. **Lifecycle Manager** (`pkg/lifecycle/`) - - Builder registration on beacon chain - - Balance monitoring and auto top-ups - - Deposit and exit operations +3b. **Builder Key Registry** (`pkg/builder_keys/`) — the managed set of builder + BLS keys and the identity dependency of every module that used to hold a + single `*signer.BLSSigner`. + - **Derivation**: keys come from the operator's ENTRY key (`--builder-privkey` + or `--builder-mnemonic` + `--builder-key-index`). Internal index 0 IS the + entry key; index n ≥ 1 is `derive_child_SK(entry_sk, n)` — one EIP-2333 node + deeper than any other participant's account path, which is what makes the + set collision-free with other builders sharing a mnemonic. Derivation is + cached for the process lifetime (`signer.DeriveInternalKey`). + - **Two index spaces, never conflated**: `KeyIndex` is our derivation index + (stable forever); `BuilderIndex` is the beacon registry index, assigned at + deposit and REUSED by other builders after an exit. + - **State** per key (`Status`: unused / depositing / pending / active / + exiting / exited / withdrawn) is resolved from the epoch snapshot in ONE + pass over `chainSvc.GetBuilders()`, so monitoring hundreds of keys costs the + same beacon query as one. Only transactions scale with the fleet size. + - **Usage history** (`kv_store` namespace `builder_keys`) survives restarts so + a key deposited in an earlier run is recognised before the beacon state + confirms it, and a key whose pubkey has left the registry is reused instead + of pushing the highest index up forever. A persisted pubkey that disagrees + with the derived one is a hard startup error (the entry key changed). + - **Discovery** scans past the target for keys we used before and stops after + `builder_keys.discovery_gap` never-used indices. Scanned-but-unused indices + are derived, not tracked, so the fleet view is the size of the fleet. + - **Selection** (`SelectForBid`) orders active keys per strategy + (`round_robin` default, `single`, `random`, `least_used`) excluding keys + already committed for the slot. Balance is a PREFERENCE, not a filter: + underfunded keys sort last but are still offered when nothing else can + cover the bid — deliberately underfunded bids are a scenario worth testing. + +4. **Lifecycle Manager** (`pkg/lifecycle/`) — the fleet reconciler + - Keeps the managed key count at `builder_keys.target_count`: deposits keys + below it (`auto_deposit`), exits surplus keys above it (`auto_exit`, + highest index first, skipping keys with pending payments the chain would + silently ignore), and tops up whichever key fell below the threshold + - At most ONE lifecycle transaction per reconcile pass: they all go through + the single funding wallet and each waits for its receipt, so the fleet + ramps instead of flooding the EIP-8282 deposit queue (whose fee grows with + its length). `deposit_max_fee` then backs the ramp off on its own; a wallet + that cannot cover the next deposit reports once instead of failing per key + - Target changes wake the reconciler immediately (`Manager.Reconcile()`, + wired from the settings `OnChange`) + - Early onboarding covers the WHOLE target set: the pre-Gloas deposits sit in + the pending queue together and the fork transition converts them all, so + `pending_deposit_sim.go` asks whether the batch's LAST entry survives - Optional component (only active with `--lifecycle` flag) 4b. **Slot Results Tracker** (`pkg/slot_results/`) — generic per-slot outcome history @@ -451,7 +492,25 @@ Configuration is managed via: - Environment variables (auto-loaded by viper) Key config sections: -- **Builder keys**: `--builder-privkey` (BLS), `--wallet-privkey` (ECDSA) +- **Builder keys**: `--builder-privkey` (BLS) or `--builder-mnemonic` + + `--builder-key-index` (the ENTRY key), `--wallet-privkey` (ECDSA) +- **Managed key set**: `--builder-keys-target` (keys kept registered and + funded; default 1 = the entry key alone, byte-identical to single-key + behaviour), `--builder-keys-max-index` (derivation cap, default 1000), + `--builder-keys-discovery-gap` (unused indices ending the startup scan, + default 100), `--builder-keys-auto-deposit` / `--builder-keys-auto-exit` + (both default true; auto-exit is irreversible — an exited key cannot be + reactivated). All but the discovery gap are mutable via + `builder_keys.*` settings keys +- **Key selection**: `--epbs-key-strategy` (round_robin | single | random | + least_used), `--epbs-bid-keys-per-slot` (cap per slot, 0 = no cap beyond the + fleet), `--epbs-bid-keys-per-step` (keys bidding a payload per interval step, + each one increment higher; 1 = walk the fleet up the ladder, 0 = spend every + remaining key at once), `--builder-api-key-strategy` (empty = follow the ePBS + strategy). A key is SPENT once one of its bids reaches the network — the + gossip rules ignore a builder's later bids for a slot — so every bid, an + escalated re-bid of the same payload included, takes a key that has not bid + yet, and the submissions of one step go out concurrently - **Clients**: `--cl-client`, `--el-engine-api`, `--el-rpc` - **Schedule**: `--schedule-mode` (all/every_nth/next_n), `--schedule-every-nth`, `--schedule-next-n` - **ePBS timing**: `--build-start-time`, `--epbs-bid-start`, `--epbs-bid-end` diff --git a/cmd/deposit.go b/cmd/deposit.go index 6d9e749e..1ed74fd1 100644 --- a/cmd/deposit.go +++ b/cmd/deposit.go @@ -11,7 +11,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/lifecycle" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" "github.com/ethpandaops/buildoor/pkg/rpc/execution" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/wallet" ) @@ -53,10 +52,17 @@ var depositCmd = &cobra.Command{ } defer rpcClient.Close() - // Initialize BLS signer (raw hex key or mnemonic-derived) - blsSigner, err := signer.NewBuilderSigner(cfg.BuilderPrivkey, cfg.BuilderMnemonic, cfg.BuilderKeyIndex) + // Initialize the managed builder key set and select the key to deposit for + registry, err := newKeyRegistry(cfg, logger) if err != nil { - return fmt.Errorf("invalid builder key: %w", err) + return err + } + + keyIndex, _ := cmd.Flags().GetUint64("key-index") + + key, err := registry.Key(keyIndex) + if err != nil { + return err } // Initialize wallet @@ -92,7 +98,7 @@ var depositCmd = &cobra.Command{ defer chainSvc.Stop() //nolint:errcheck // cleanup // Check if builder already registered - pubkey := blsSigner.PublicKey() + pubkey := key.Pubkey() builderInfo := chainSvc.GetBuilderByPubkey(pubkey) if builderInfo != nil { @@ -110,25 +116,26 @@ var depositCmd = &cobra.Command{ timeout, _ := cmd.Flags().GetDuration("timeout") // Initialize lifecycle manager - lifecycleMgr, err := lifecycle.NewManager(cfg, clClient, chainSvc, blsSigner, w, logger) + lifecycleMgr, err := lifecycle.NewManager(cfg, clClient, chainSvc, registry, w, logger) if err != nil { return fmt.Errorf("failed to initialize lifecycle manager: %w", err) } logger.WithFields(map[string]any{ + "key": key.String(), "pubkey": fmt.Sprintf("%x", pubkey[:8]), "amount_gwei": amount, }).Info("Creating builder deposit") // Ensure builder is registered - if err := lifecycleMgr.EnsureBuilderRegistered(ctx); err != nil { + if err := lifecycleMgr.EnsureBuilderRegistered(ctx, key); err != nil { return fmt.Errorf("failed to ensure builder registered: %w", err) } if waitForInclusion { logger.Info("Waiting for registration...") - if err := lifecycleMgr.WaitForRegistration(ctx, timeout); err != nil { + if err := lifecycleMgr.WaitForRegistration(ctx, key, timeout); err != nil { return fmt.Errorf("registration wait failed: %w", err) } @@ -147,6 +154,7 @@ func init() { rootCmd.AddCommand(depositCmd) depositCmd.Flags().Uint64("amount", 10000000000, "Deposit amount in Gwei") + depositCmd.Flags().Uint64("key-index", 0, "Internal builder key index to deposit for (0 = the entry key)") depositCmd.Flags().Bool("wait", true, "Wait for deposit to be included") depositCmd.Flags().Duration("timeout", 5*time.Minute, "Timeout for waiting") } diff --git a/cmd/exit.go b/cmd/exit.go index 8189b0dc..85add338 100644 --- a/cmd/exit.go +++ b/cmd/exit.go @@ -10,7 +10,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/lifecycle" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" "github.com/ethpandaops/buildoor/pkg/rpc/execution" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/wallet" ) @@ -52,13 +51,20 @@ var exitCmd = &cobra.Command{ } defer rpcClient.Close() - // Initialize BLS signer (raw hex key or mnemonic-derived) - blsSigner, err := signer.NewBuilderSigner(cfg.BuilderPrivkey, cfg.BuilderMnemonic, cfg.BuilderKeyIndex) + // Initialize the managed builder key set and select the key to exit + registry, err := newKeyRegistry(cfg, logger) if err != nil { - return fmt.Errorf("invalid builder key: %w", err) + return err } - pubkey := blsSigner.PublicKey() + keyIndex, _ := cmd.Flags().GetUint64("key-index") + + key, err := registry.Key(keyIndex) + if err != nil { + return err + } + + pubkey := key.Pubkey() // Initialize wallet (its address is the exit source; must match the builder's // registered execution address) @@ -97,12 +103,13 @@ var exitCmd = &cobra.Command{ } logger.WithFields(map[string]any{ + "key": key.String(), "builder_index": builderInfo.Index, "pubkey": fmt.Sprintf("%x", pubkey[:8]), }).Info("Submitting builder exit") - exitSvc := lifecycle.NewExitService(chainSvc, blsSigner, w, logger) - if err := exitSvc.CreateExit(ctx); err != nil { + exitSvc := lifecycle.NewExitService(chainSvc, w, logger) + if err := exitSvc.CreateExit(ctx, key); err != nil { return fmt.Errorf("failed to submit builder exit: %w", err) } @@ -114,4 +121,6 @@ var exitCmd = &cobra.Command{ func init() { rootCmd.AddCommand(exitCmd) + + exitCmd.Flags().Uint64("key-index", 0, "Internal builder key index to exit (0 = the entry key)") } diff --git a/cmd/keys.go b/cmd/keys.go new file mode 100644 index 00000000..95702ef7 --- /dev/null +++ b/cmd/keys.go @@ -0,0 +1,28 @@ +package cmd + +import ( + "fmt" + + "github.com/sirupsen/logrus" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" + "github.com/ethpandaops/buildoor/pkg/signer" +) + +// newKeyRegistry builds the managed builder key set from the configured entry +// key (raw private key or mnemonic + account index). Internal key 0 is the entry +// key itself, so a single-key deployment keeps its identity. +func newKeyRegistry(cfg *config.Config, log logrus.FieldLogger) (*builder_keys.Registry, error) { + entryPrivkey, err := signer.ResolveEntryPrivkey(cfg.BuilderPrivkey, cfg.BuilderMnemonic, cfg.BuilderKeyIndex) + if err != nil { + return nil, fmt.Errorf("invalid builder key: %w", err) + } + + registry, err := builder_keys.NewRegistry(cfg, entryPrivkey, log) + if err != nil { + return nil, fmt.Errorf("failed to initialize builder key registry: %w", err) + } + + return registry, nil +} diff --git a/cmd/root.go b/cmd/root.go index ef4eb36d..63f1b5f2 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -65,6 +65,11 @@ func init() { rootCmd.PersistentFlags().Bool("builder-api-on-demand-build", defaults.BuilderAPI.OnDemandBuild, "Build a payload on the fly when a bid request asks for a legal parent no candidate covers yet") rootCmd.PersistentFlags().String("builder-api-url", defaults.BuilderAPI.BuilderURL, "Publicly reachable URL of this builder (e.g. https://builder.example.com); used to validate builder_url in SignedRequestAuthV1") rootCmd.PersistentFlags().Bool("builder-api-require-auth", defaults.BuilderAPI.RequireRequestAuth, "Require SignedRequestAuthV1 on getExecutionPayloadBid requests; reject unauthenticated requests with 401") + rootCmd.PersistentFlags().Uint64("builder-keys-target", defaults.BuilderKeys.TargetCount, "Number of builder keys to keep registered and funded (derived from the entry key; index 0 is the entry key itself)") + rootCmd.PersistentFlags().Uint64("builder-keys-max-index", defaults.BuilderKeys.MaxIndex, "Highest internal builder key index that may be derived") + rootCmd.PersistentFlags().Uint64("builder-keys-discovery-gap", defaults.BuilderKeys.DiscoveryGap, "Number of consecutive unused indices that ends the startup scan for previously deposited keys") + rootCmd.PersistentFlags().Bool("builder-keys-auto-deposit", defaults.BuilderKeys.AutoDeposit, "Deposit new builder keys to reach the target count") + rootCmd.PersistentFlags().Bool("builder-keys-auto-exit", defaults.BuilderKeys.AutoExit, "Exit surplus builder keys when the managed count exceeds the target (irreversible)") rootCmd.PersistentFlags().Uint64("deposit-amount", defaults.DepositAmount, "Builder deposit amount in Gwei") rootCmd.PersistentFlags().Uint64("topup-threshold", defaults.TopupThreshold, "Balance threshold for auto top-up in Gwei") rootCmd.PersistentFlags().Uint64("topup-amount", defaults.TopupAmount, "Amount to top-up in Gwei") @@ -92,6 +97,10 @@ func init() { rootCmd.PersistentFlags().Uint64("epbs-bid-value-override", defaults.EPBS.BidValueOverride, "Absolute p2p bid base value in gwei, replacing max(blockValue, bid-min) + subsidy (0 = disabled); allows underbidding the block value for testing") rootCmd.PersistentFlags().Uint64("epbs-vote-threshold", defaults.EPBS.HeadVoteThresholdPct, "Head-vote participation threshold in percent; crossing it fires an immediate threshold_met update (0 = disabled)") rootCmd.PersistentFlags().String("epbs-bid-candidate", defaults.EPBS.BidCandidate, "Which built candidate payload p2p bids commit to: auto, parent_full, parent_empty, grandparent_full, grandparent_empty or all") + rootCmd.PersistentFlags().String("epbs-key-strategy", defaults.EPBS.KeyStrategy, "Which managed builder key signs each bid: round_robin, single, random or least_used") + rootCmd.PersistentFlags().Uint64("epbs-bid-keys-per-slot", defaults.EPBS.BidKeysPerSlot, "Max distinct builder keys bidding one slot (0 = no cap beyond the fleet)") + rootCmd.PersistentFlags().Uint64("epbs-bid-keys-per-step", defaults.EPBS.BidKeysPerStep, "Builder keys bidding a payload per interval step, each one increment higher (0 = every remaining key at once)") + rootCmd.PersistentFlags().String("builder-api-key-strategy", defaults.BuilderAPI.KeyStrategy, "Which managed builder key signs served Builder API bids (empty = follow --epbs-key-strategy)") rootCmd.PersistentFlags().Bool("epbs-bid-candidate-switch", defaults.EPBS.BidCandidateSwitch, "Allow the auto bid candidate selection to switch mid-slot when the chain view changes") // Payload build candidates (reorg / payload-miss preparedness) @@ -199,6 +208,14 @@ func initConfig() error { ValueOverrideGwei: v.GetUint64("builder-api-value-override"), ServeCandidates: v.GetString("builder-api-serve-candidates"), OnDemandBuild: v.GetBool("builder-api-on-demand-build"), + KeyStrategy: v.GetString("builder-api-key-strategy"), + }, + BuilderKeys: config.BuilderKeysConfig{ + TargetCount: v.GetUint64("builder-keys-target"), + MaxIndex: v.GetUint64("builder-keys-max-index"), + DiscoveryGap: v.GetUint64("builder-keys-discovery-gap"), + AutoDeposit: v.GetBool("builder-keys-auto-deposit"), + AutoExit: v.GetBool("builder-keys-auto-exit"), }, DepositMaxFeeGwei: v.GetUint64("deposit-max-fee"), DepositAmount: v.GetUint64("deposit-amount"), @@ -223,6 +240,9 @@ func initConfig() error { HeadVoteThresholdPct: v.GetUint64("epbs-vote-threshold"), BidCandidate: v.GetString("epbs-bid-candidate"), BidCandidateSwitch: v.GetBool("epbs-bid-candidate-switch"), + KeyStrategy: v.GetString("epbs-key-strategy"), + BidKeysPerSlot: v.GetUint64("epbs-bid-keys-per-slot"), + BidKeysPerStep: v.GetUint64("epbs-bid-keys-per-step"), }, Reveal: config.RevealConfig{ Enabled: v.GetBool("reveal-enabled"), diff --git a/cmd/run.go b/cmd/run.go index 0542d89a..b08a6a0d 100644 --- a/cmd/run.go +++ b/cmd/run.go @@ -29,7 +29,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" "github.com/ethpandaops/buildoor/pkg/rpc/execution" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/slot_results" "github.com/ethpandaops/buildoor/pkg/validatorranges" "github.com/ethpandaops/buildoor/pkg/wallet" @@ -84,13 +83,14 @@ and begins building blocks according to configuration.`, return fmt.Errorf("failed to connect to EL engine API: %w", err) } - // 3. Initialize BLS signer (raw hex key or mnemonic-derived) - blsSigner, err := signer.NewBuilderSigner(cfg.BuilderPrivkey, cfg.BuilderMnemonic, cfg.BuilderKeyIndex) + // 3. Initialize the managed builder key set (derived from the raw hex key + // or the mnemonic; internal key 0 is that entry key itself) + keyRegistry, err := newKeyRegistry(cfg, logger) if err != nil { - return fmt.Errorf("invalid builder key: %w", err) + return err } - pubkey := blsSigner.PublicKey() + pubkey := keyRegistry.Primary().Pubkey() logger.WithField("pubkey", fmt.Sprintf("%x", pubkey[:8])).Info("Builder key loaded") // 4. Initialize RPC client and wallet (if lifecycle enabled) @@ -217,6 +217,14 @@ and begins building blocks according to configuration.`, } defer chainSvc.Stop() //nolint:errcheck // cleanup + // 7a. Start the builder key registry: it resolves every managed key's + // on-chain state from the chain service's epoch snapshots and persists + // the usage history that makes withdrawn keys reusable. + if err := keyRegistry.Start(ctx, chainSvc, stateDB); err != nil { + return fmt.Errorf("failed to start builder key registry: %w", err) + } + defer keyRegistry.Stop() + // 7b. Initialize the per-slot action plan service. Decision points // (build/bid/serve/reveal) freeze the slot's plan on first use; plans // persist in the state-db's kv_store when --state-db is set. @@ -234,7 +242,7 @@ and begins building blocks according to configuration.`, var lifecycleMgr *lifecycle.Manager if lifecycleAvailable { - lifecycleMgr, err = lifecycle.NewManager(cfg, clClient, chainSvc, blsSigner, w, logger) + lifecycleMgr, err = lifecycle.NewManager(cfg, clClient, chainSvc, keyRegistry, w, logger) if err != nil { return fmt.Errorf("failed to initialize lifecycle: %w", err) } @@ -299,7 +307,11 @@ and begins building blocks according to configuration.`, if epbsAvailable { paymentTracker = payload_bidder.NewPaymentTracker(chainSvc, logger) - revealSvc = payload_bidder.NewRevealService(cfg, payload_bidder.NewSigner(blsSigner), + // Effective balances (bid readiness, top-up decisions) must account + // for payments settled since the last epoch snapshot. + keyRegistry.SetBalanceAdjuster(paymentTracker) + + revealSvc = payload_bidder.NewRevealService(cfg, keyRegistry, clClient, chainSvc, builderSvc, paymentTracker, planSvc, chainSvc.GetHeadVoteTracker(), logger) if err := revealSvc.Start(ctx); err != nil { @@ -308,7 +320,8 @@ and begins building blocks according to configuration.`, defer revealSvc.Stop() } - inclusionTracker := payload_bidder.NewInclusionTracker(clClient, chainSvc, builderSvc, revealSvc, paymentTracker, logger) + inclusionTracker := payload_bidder.NewInclusionTracker(clClient, chainSvc, builderSvc, keyRegistry, + revealSvc, paymentTracker, logger) if err := inclusionTracker.Start(ctx); err != nil { return fmt.Errorf("failed to start inclusion tracker: %w", err) @@ -340,7 +353,7 @@ and begins building blocks according to configuration.`, gloasForkEpoch := chainSpec.GetForkEpoch(version.DataVersionGloas) logger.WithField("gloas_fork_epoch", gloasForkEpoch).Info("Initializing p2p bidder service...") - epbsSvc, err = p2p_bidder.NewService(clClient, chainSvc, blsSigner, propPrefSvc.GetStore(), planSvc, logger) + epbsSvc, err = p2p_bidder.NewService(clClient, chainSvc, keyRegistry, propPrefSvc.GetStore(), planSvc, logger) if err != nil { return fmt.Errorf("failed to initialize p2p bidder: %w", err) } @@ -368,7 +381,7 @@ and begins building blocks according to configuration.`, "genesis_validators_root": fmt.Sprintf("0x%x", genesisValidatorsRoot[:]), }).Info("Using genesis parameters from beacon node") - builderAPISrv = builderapi.NewServer(&cfg.BuilderAPI, logger, chainSvc, planSvc, builderSvc.GetPayloadCache(), blsSigner, validatorStore) + builderAPISrv = builderapi.NewServer(&cfg.BuilderAPI, logger, chainSvc, planSvc, builderSvc.GetPayloadCache(), keyRegistry, validatorStore) builderAPISrv.SetCLClient(clClient) builderAPISrv.SetEnabled(cfg.BuilderAPIEnabled) @@ -393,7 +406,7 @@ and begins building blocks according to configuration.`, // subscriptions never miss an event; SetPersistence migrates any // legacy won_blocks namespace into slot results. resultTracker := slot_results.NewTracker(cfg, chainSvc, stateDB, planSvc, - builderSvc, epbsSvc, revealSvc, inclusionTracker, logger) + builderSvc, epbsSvc, revealSvc, inclusionTracker, keyRegistry, logger) resultTracker.SetPersistence(ctx, stateDB) if err := resultTracker.Start(ctx); err != nil { @@ -434,7 +447,13 @@ and begins building blocks according to configuration.`, if lifecycleMgr != nil { lifecycleMgr.SetEnabled(cfg.LifecycleEnabled) + // A target key count change must act now, not at the next + // reconcile tick. + lifecycleMgr.Reconcile() } + + // The derived key set follows the target/derivation settings. + keyRegistry.Refresh() }) // 15. Start WebUI/API server (if configured) @@ -452,7 +471,7 @@ and begins building blocks according to configuration.`, AuthProviderURL: cfg.AuthProviderURL, InjectHeadHTML: cfg.InjectHeadHTML, OverviewURL: cfg.OverviewURL, - }, settingsSvc, stateDB, builderSvc, epbsSvc, lifecycleMgr, chainSvc, validatorStore, builderAPISrv, propPrefSvc, valRanges, revealSvc, inclusionTracker, paymentTracker, planSvc, resultTracker) + }, settingsSvc, stateDB, builderSvc, epbsSvc, lifecycleMgr, keyRegistry, chainSvc, validatorStore, builderAPISrv, propPrefSvc, valRanges, revealSvc, inclusionTracker, paymentTracker, planSvc, resultTracker) // Connect Builder API server to event stream (if both are enabled) if builderAPISrv != nil && apiHandler != nil { @@ -471,12 +490,6 @@ and begins building blocks according to configuration.`, }) lifecycleMgr.SetRegistrationCallback(func(index uint64) { epbsSvc.SetBuilderRegistered(index) - if builderAPISrv != nil { - builderAPISrv.SetBuilderIndex(index) - } - if revealSvc != nil { - revealSvc.SetBuilderIndex(index) - } }) } diff --git a/pkg/action_plan/frozen.go b/pkg/action_plan/frozen.go index a3e2d74d..a1088119 100644 --- a/pkg/action_plan/frozen.go +++ b/pkg/action_plan/frozen.go @@ -115,11 +115,39 @@ type ResolvedBidSettings struct { // auto, all, or a specific candidate key. BidCandidate string `json:"bid_candidate,omitempty"` + // KeyStrategy is the effective builder key selection strategy for the + // slot's bids. + KeyStrategy string `json:"key_strategy,omitempty"` + + // BidKeysPerSlot caps how many distinct builder keys bid the slot + // (0 = no cap beyond the fleet). + BidKeysPerSlot uint64 `json:"bid_keys_per_slot,omitempty"` + + // BidKeysPerStep is how many keys bid a payload per interval step + // (0 = every remaining key at once). + BidKeysPerStep uint64 `json:"bid_keys_per_step,omitempty"` + // Forced marks that the plan activated bidding although the module is // globally disabled. Forced bool `json:"forced,omitempty"` } +// EffectiveKeysPerStep returns how many keys one step may spend: the configured +// count, or every key the fleet has left when unset. +func (s *ResolvedBidSettings) EffectiveKeysPerStep() uint64 { + if s.BidKeysPerStep > 0 { + return s.BidKeysPerStep + } + + // Unset means "as many as are left"; the per-slot cap and the fleet size + // bound the loop, so a generous ceiling is enough. + return maxKeysPerStep +} + +// maxKeysPerStep bounds an unset per-step count so a step can never loop +// unbounded; no fleet reaches it in practice. +const maxKeysPerStep = 1024 + // ResolvedBuilderAPISettings are the effective Builder API bid-serving // parameters for the slot. type ResolvedBuilderAPISettings struct { @@ -135,6 +163,10 @@ type ResolvedBuilderAPISettings struct { // (all, canonical_only, or a comma-separated candidate key list). ServeCandidates string `json:"serve_candidates,omitempty"` + // KeyStrategy is the effective builder key selection strategy for the + // slot's served bids. + KeyStrategy string `json:"key_strategy,omitempty"` + // Forced marks that the plan activated serving although the module is // globally disabled. Forced bool `json:"forced,omitempty"` @@ -373,14 +405,17 @@ func resolveBid(plan *SlotPlan, cfg *config.Config, fork version.DataVersion) *R } resolved := &ResolvedBidSettings{ - StartMs: cfg.EPBS.BidStartTime, - EndMs: cfg.EPBS.BidEndTime, - IntervalMs: cfg.EPBS.BidInterval, - MinGwei: cfg.EPBS.BidMinAmount, - IncreaseGwei: cfg.EPBS.BidIncrease, - SubsidyGwei: cfg.EPBS.BidSubsidy, - BidCandidate: cfg.EPBS.BidCandidate, - Forced: forced, + StartMs: cfg.EPBS.BidStartTime, + EndMs: cfg.EPBS.BidEndTime, + IntervalMs: cfg.EPBS.BidInterval, + MinGwei: cfg.EPBS.BidMinAmount, + IncreaseGwei: cfg.EPBS.BidIncrease, + SubsidyGwei: cfg.EPBS.BidSubsidy, + BidCandidate: cfg.EPBS.BidCandidate, + KeyStrategy: cfg.EPBS.KeyStrategy, + BidKeysPerSlot: cfg.EPBS.BidKeysPerSlot, + BidKeysPerStep: cfg.EPBS.BidKeysPerStep, + Forced: forced, } if cfg.EPBS.BidValueOverride > 0 { @@ -406,6 +441,13 @@ func resolveBid(plan *SlotPlan, cfg *config.Config, fork version.DataVersion) *R if bid.BidCandidate != nil { resolved.BidCandidate = *bid.BidCandidate } + + if bid.KeyStrategy != nil { + resolved.KeyStrategy = *bid.KeyStrategy + } + + applyOverride(&resolved.BidKeysPerSlot, bid.BidKeysPerSlot) + applyOverride(&resolved.BidKeysPerStep, bid.BidKeysPerStep) } return resolved @@ -433,6 +475,7 @@ func resolveBuilderAPI(plan *SlotPlan, cfg *config.Config) *ResolvedBuilderAPISe resolved := &ResolvedBuilderAPISettings{ SubsidyGwei: cfg.BuilderAPI.BlockValueSubsidyGwei, ServeCandidates: cfg.BuilderAPI.ServeCandidates, + KeyStrategy: builderAPIKeyStrategy(cfg), Forced: forced, } @@ -453,11 +496,25 @@ func resolveBuilderAPI(plan *SlotPlan, cfg *config.Config) *ResolvedBuilderAPISe if api.ServeCandidates != nil { resolved.ServeCandidates = *api.ServeCandidates } + + if api.KeyStrategy != nil { + resolved.KeyStrategy = *api.KeyStrategy + } } return resolved } +// builderAPIKeyStrategy resolves the Builder API's key selection strategy, +// which falls back to the ePBS one when left unset. +func builderAPIKeyStrategy(cfg *config.Config) string { + if cfg.BuilderAPI.KeyStrategy != "" { + return cfg.BuilderAPI.KeyStrategy + } + + return cfg.EPBS.KeyStrategy +} + func resolveReveal(plan *SlotPlan, cfg *config.Config) *ResolvedRevealSettings { resolved := &ResolvedRevealSettings{ Suppressed: !cfg.Reveal.Enabled, diff --git a/pkg/action_plan/types.go b/pkg/action_plan/types.go index 24ee188a..ef542664 100644 --- a/pkg/action_plan/types.go +++ b/pkg/action_plan/types.go @@ -15,6 +15,7 @@ import ( "github.com/ethpandaops/go-eth2-client/spec/phase0" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/jqtransform" @@ -61,6 +62,18 @@ type BidPlan struct { // bids commit to: auto, all, or a specific candidate key. BidCandidate *string `json:"bid_candidate,omitempty"` + // KeyStrategy overrides which managed builder key signs this slot's bids: + // round_robin, single, random or least_used. + KeyStrategy *string `json:"key_strategy,omitempty"` + + // BidKeysPerSlot overrides how many distinct builder keys may bid this + // slot (0 = no cap beyond the fleet). + BidKeysPerSlot *uint64 `json:"bid_keys_per_slot,omitempty"` + + // BidKeysPerStep overrides how many keys bid a payload per interval step + // (0 = every remaining key at once). + BidKeysPerStep *uint64 `json:"bid_keys_per_step,omitempty"` + // BidValueGwei is an absolute bid base value replacing // max(blockValue, min) + subsidy; BidIncrease still applies per re-bid. // Allows underbidding the block value for testing. @@ -84,6 +97,9 @@ func (p *BidPlan) clone() *BidPlan { c.BidInterval = cloneScalar(p.BidInterval) c.BidSubsidy = cloneScalar(p.BidSubsidy) c.BidValueGwei = cloneScalar(p.BidValueGwei) + c.KeyStrategy = cloneScalar(p.KeyStrategy) + c.BidKeysPerSlot = cloneScalar(p.BidKeysPerSlot) + c.BidKeysPerStep = cloneScalar(p.BidKeysPerStep) return &c } @@ -91,7 +107,8 @@ func (p *BidPlan) clone() *BidPlan { func (p *BidPlan) hasOverrides() bool { return p.BidStartTime != nil || p.BidEndTime != nil || p.BidMinAmount != nil || p.BidIncrease != nil || p.BidInterval != nil || p.BidSubsidy != nil || - p.BidValueGwei != nil || p.IgnoreMissingPrefs || p.BidCandidate != nil + p.BidValueGwei != nil || p.IgnoreMissingPrefs || p.BidCandidate != nil || + p.KeyStrategy != nil || p.BidKeysPerSlot != nil || p.BidKeysPerStep != nil } func (p *BidPlan) validate(slotMs int64) error { @@ -127,6 +144,13 @@ func (p *BidPlan) validate(slotMs int64) error { } } + if p.KeyStrategy != nil { + if strategy := *p.KeyStrategy; strategy != builder_keys.NormalizedStrategy(strategy) { + return fmt.Errorf( + "bid: key_strategy must be round_robin, single, random or least_used, got %q", strategy) + } + } + return nil } @@ -147,6 +171,10 @@ type BuilderAPIPlan struct { // (context-cancellable, capped at one slot). ResponseDelayMs *int64 `json:"response_delay_ms,omitempty"` + // KeyStrategy overrides which managed builder key signs this slot's served + // bids: round_robin, single, random or least_used. + KeyStrategy *string `json:"key_strategy,omitempty"` + // ServeCandidates overrides which built candidate payloads bid requests // for this slot may be answered from: all, canonical_only, or a // comma-separated candidate key list. @@ -168,7 +196,7 @@ func (p *BuilderAPIPlan) clone() *BuilderAPIPlan { func (p *BuilderAPIPlan) hasOverrides() bool { return p.ValueSubsidyGwei != nil || p.TotalValueOverrideGwei != nil || p.ResponseDelayMs != nil || - p.ServeCandidates != nil + p.ServeCandidates != nil || p.KeyStrategy != nil } func (p *BuilderAPIPlan) validate(slotMs int64) error { @@ -191,6 +219,13 @@ func (p *BuilderAPIPlan) validate(slotMs int64) error { } } + if p.KeyStrategy != nil { + if strategy := *p.KeyStrategy; strategy != builder_keys.NormalizedStrategy(strategy) { + return fmt.Errorf( + "builder_api: key_strategy must be round_robin, single, random or least_used, got %q", strategy) + } + } + return nil } diff --git a/pkg/builder_keys/key.go b/pkg/builder_keys/key.go new file mode 100644 index 00000000..f8cabca4 --- /dev/null +++ b/pkg/builder_keys/key.go @@ -0,0 +1,91 @@ +// Package builder_keys owns buildoor's managed set of builder BLS keys: the +// internal derivation from the operator's entry key, each key's on-chain state +// (registration, balance, pending payments), the persisted usage history that +// makes withdrawn keys reusable, and the selection of a ready key per bid. +// +// Two index spaces meet here and must never be conflated: +// +// - the KEY INDEX is our internal derivation index. It is stable forever; +// index 0 is the operator's entry key, so a single-key deployment keeps its +// identity when the fleet grows. +// - the BUILDER INDEX is the beacon registry index assigned at deposit time. +// It only exists once a key is registered and is reused by other builders +// after an exit. +package builder_keys + +import ( + "fmt" + "sync/atomic" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" + + "github.com/ethpandaops/buildoor/pkg/signer" +) + +// Key is one derived builder key: a stable identity plus the latest snapshot of +// its on-chain state. Instances are created once by the Registry and are safe +// for concurrent use; the state snapshot is swapped atomically on every refresh, +// so readers on the bid/reveal hot paths never block or observe a torn value. +type Key struct { + keyIndex uint64 + pubkey phase0.BLSPubKey + bls *signer.BLSSigner + + state atomic.Pointer[State] +} + +// newKey derives the key at the given internal index from the entry private key. +func newKey(entryPrivkeyHex string, keyIndex uint64) (*Key, error) { + privkeyHex, err := signer.DeriveInternalKey(entryPrivkeyHex, keyIndex) + if err != nil { + return nil, fmt.Errorf("failed to derive builder key %d: %w", keyIndex, err) + } + + blsSigner, err := signer.NewBLSSigner(privkeyHex) + if err != nil { + return nil, fmt.Errorf("failed to create signer for builder key %d: %w", keyIndex, err) + } + + k := &Key{ + keyIndex: keyIndex, + pubkey: blsSigner.PublicKey(), + bls: blsSigner, + } + + k.state.Store(&State{ + KeyIndex: keyIndex, + Pubkey: k.pubkey, + Status: StatusUnused, + }) + + return k, nil +} + +// KeyIndex returns the internal derivation index (0 = the entry key). +func (k *Key) KeyIndex() uint64 { return k.keyIndex } + +// Pubkey returns the key's BLS public key. +func (k *Key) Pubkey() phase0.BLSPubKey { return k.pubkey } + +// BLSSigner returns the underlying signer for bid, envelope and deposit signing. +func (k *Key) BLSSigner() *signer.BLSSigner { return k.bls } + +// State returns the latest state snapshot. Never nil. +func (k *Key) State() *State { return k.state.Load() } + +// BuilderIndex returns the on-chain builder index and whether the key is +// registered at all. Index 0 is a valid builder index, so the boolean — not a +// zero check — decides whether the value may be used. +func (k *Key) BuilderIndex() (uint64, bool) { + state := k.state.Load() + + return state.BuilderIndex, state.HasBuilderIndex +} + +// Status returns the key's current lifecycle status. +func (k *Key) Status() Status { return k.state.Load().Status } + +// String renders the key for logs: internal index plus a pubkey prefix. +func (k *Key) String() string { + return fmt.Sprintf("#%d/%x", k.keyIndex, k.pubkey[:4]) +} diff --git a/pkg/builder_keys/mockchain_test.go b/pkg/builder_keys/mockchain_test.go new file mode 100644 index 00000000..31483e47 --- /dev/null +++ b/pkg/builder_keys/mockchain_test.go @@ -0,0 +1,91 @@ +package builder_keys + +import ( + "context" + "time" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" + "github.com/ethpandaops/go-eth2-client/spec/version" + + "github.com/ethpandaops/buildoor/pkg/chain" + "github.com/ethpandaops/buildoor/pkg/rpc/beacon" + "github.com/ethpandaops/buildoor/pkg/utils" +) + +// stubChainService is a minimal chain.Service exposing a fixed builder registry +// and finality, which is all the registry reads (shape copied from +// pkg/payload_bidder/mockchain_test.go). +type stubChainService struct { + builders []*chain.BuilderInfo + currentEpoch phase0.Epoch + finalizedEpoch phase0.Epoch + genesis beacon.Genesis + + epochStatsDispatch utils.Dispatcher[*chain.EpochStats] +} + +var _ chain.Service = (*stubChainService)(nil) + +func (m *stubChainService) Start(context.Context) error { return nil } +func (m *stubChainService) Stop() error { return nil } + +func (m *stubChainService) GetChainSpec() *chain.ChainSpec { + return &chain.ChainSpec{SecondsPerSlot: 12 * time.Second, SlotsPerEpoch: 32} +} +func (m *stubChainService) GetGenesis() *beacon.Genesis { return &m.genesis } + +func (m *stubChainService) SlotToTime(phase0.Slot) time.Time { return time.Time{} } +func (m *stubChainService) TimeToSlot(time.Time) phase0.Slot { return 0 } +func (m *stubChainService) GetCurrentEpoch() phase0.Epoch { return m.currentEpoch } +func (m *stubChainService) GetCurrentSlot() phase0.Slot { return 0 } +func (m *stubChainService) GetFinalizedEpoch() phase0.Epoch { return m.finalizedEpoch } +func (m *stubChainService) GetCurrentFork() version.DataVersion { + return version.DataVersionGloas +} + +func (m *stubChainService) ActiveForkAtEpoch(phase0.Epoch) version.DataVersion { + return version.DataVersionGloas +} +func (m *stubChainService) GetForkVersion() (phase0.Version, error) { return phase0.Version{}, nil } + +func (m *stubChainService) GetEpochOfSlot(slot phase0.Slot) phase0.Epoch { + return phase0.Epoch(uint64(slot) / 32) +} + +func (m *stubChainService) GetCurrentEpochStats() *chain.EpochStats { return nil } +func (m *stubChainService) GetEpochStats(phase0.Epoch) *chain.EpochStats { return nil } + +func (m *stubChainService) SubscribeEpochStats() *utils.Subscription[*chain.EpochStats] { + return m.epochStatsDispatch.Subscribe(4, false) +} + +func (m *stubChainService) GetHeadVoteTracker() *chain.HeadVoteTracker { return nil } +func (m *stubChainService) GetHeadTracker() *chain.HeadTracker { return nil } + +func (m *stubChainService) GetBuilderByIndex(index uint64) *chain.BuilderInfo { + for _, info := range m.builders { + if info.Index == index { + return info + } + } + + return nil +} + +func (m *stubChainService) GetBuilderByPubkey(pubkey phase0.BLSPubKey) *chain.BuilderInfo { + for _, info := range m.builders { + if info.Pubkey == pubkey { + return info + } + } + + return nil +} + +func (m *stubChainService) GetBuilders() []*chain.BuilderInfo { return m.builders } + +func (m *stubChainService) GetValidatorPubkeyByIndex(phase0.ValidatorIndex) *phase0.BLSPubKey { + return nil +} + +func (m *stubChainService) RefreshBuilders(context.Context) error { return nil } diff --git a/pkg/builder_keys/operations.go b/pkg/builder_keys/operations.go new file mode 100644 index 00000000..4c959222 --- /dev/null +++ b/pkg/builder_keys/operations.go @@ -0,0 +1,210 @@ +package builder_keys + +import ( + "fmt" + "time" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" + "github.com/sirupsen/logrus" +) + +// NextDepositCandidate returns the lowest-index key eligible for a fresh +// deposit: never used, or used before and since gone from the builder registry. +// Picking the lowest index keeps the highest derivation index bounded as the +// operator ramps the target up and down over a devnet's lifetime. +// +// It returns nil when the derivation cap leaves no eligible key. +func (r *Registry) NextDepositCandidate() *Key { + for keyIndex := uint64(0); keyIndex <= r.maxIndex(); keyIndex++ { + key, err := r.derive(keyIndex) + if err != nil { + r.log.WithError(err).WithField("key_index", keyIndex). + Error("Failed to derive builder key for deposit") + + return nil + } + + if key.Status().Depositable() { + return key + } + } + + return nil +} + +// NextExitCandidate returns the highest-index key that can be exited: active on +// chain and free of pending payments, since the beacon chain silently ignores an +// exit request while a builder still owes one. +// +// It returns nil while a higher-index key is still on its way to active. Those +// in-flight keys are the actual surplus — exiting a lower, usable key in their +// place would burn a key we just paid for and leave the fleet no smaller once +// they register. +func (r *Registry) NextExitCandidate() *Key { + keys := r.Keys() + + for i := len(keys) - 1; i >= 0; i-- { + state := keys[i].State() + + switch state.Status { + case StatusDepositing, StatusPending: + return nil + case StatusActive: + if state.PendingPayments == 0 { + return keys[i] + } + + // The payment settles within a couple of epochs; waiting keeps the + // exit order intact instead of skipping down to a lower key. + return nil + case StatusUnused, StatusExiting, StatusExited, StatusWithdrawn: + // Never exitable; keep looking further down. + } + } + + return nil +} + +// MarkDepositPending claims a key for a deposit that is being submitted, so a +// batch picking several candidates in a row does not pick the same key twice and +// a concurrent pass does not deposit for it again. Release it with +// ReleaseDepositPending when the submission never reached the chain, or confirm +// it with MarkDepositSubmitted. +func (r *Registry) MarkDepositPending(keyIndex uint64) { + r.mu.Lock() + + if runtime, ok := r.runtimes[keyIndex]; ok { + runtime.depositPendingUntil = time.Now().Add(depositPendingTTL) + } + + r.mu.Unlock() + + r.setTracked(keyIndex, true) + r.Refresh() +} + +// ReleaseDepositPending clears the in-flight marker of a key whose deposit never +// reached the chain, so it becomes a candidate again immediately instead of +// waiting out the TTL. +func (r *Registry) ReleaseDepositPending(keyIndex uint64) { + r.mu.Lock() + + if runtime, ok := r.runtimes[keyIndex]; ok { + runtime.depositPendingUntil = time.Time{} + } + + r.mu.Unlock() + + r.Refresh() +} + +// MarkDepositSubmitted records a confirmed deposit transaction for a key: it +// bumps the persisted use count (the key has consumed a deposit generation) and +// holds the key in the depositing state until the beacon state catches up. +func (r *Registry) MarkDepositSubmitted(keyIndex uint64) { + key, err := r.Key(keyIndex) + if err != nil { + r.log.WithError(err).WithField("key_index", keyIndex).Error("Cannot record deposit for unknown builder key") + + return + } + + now := time.Now() + + usage, ok := r.usage.Get(keyIndex) + if !ok || usage == nil { + usage = &Usage{ + KeyIndex: keyIndex, + Pubkey: fmt.Sprintf("%#x", key.Pubkey()), + FirstUsedAt: now.UnixMilli(), + } + } else { + copied := *usage + usage = &copied + } + + usage.UseCount++ + usage.LastDepositAt = now.UnixMilli() + + r.usage.Put(keyIndex, usage) + + r.mu.Lock() + + if runtime, ok := r.runtimes[keyIndex]; ok { + runtime.depositPendingUntil = now.Add(depositPendingTTL) + } + + r.mu.Unlock() + + // The key belongs to the fleet from here on, wherever it sits relative to + // the target — the discovery scan must keep visiting it. + r.setTracked(keyIndex, true) + + r.log.WithFields(logrus.Fields{ + "key": key.String(), + "use_count": usage.UseCount, + }).Info("Recorded builder key deposit") + + r.Refresh() +} + +// MarkExitSubmitted records a submitted exit request for a key and holds it in +// the exiting state until the beacon state carries the withdrawable epoch. +// Without that marker the key keeps reading active for the couple of epochs the +// exit needs to appear, and the reconciler re-submits it — paying the queue fee +// — on every pass. +func (r *Registry) MarkExitSubmitted(keyIndex uint64) { + now := time.Now() + + if usage, ok := r.usage.Get(keyIndex); ok && usage != nil { + copied := *usage + copied.LastExitAt = now.UnixMilli() + + r.usage.Put(keyIndex, &copied) + } + + r.mu.Lock() + + if runtime, ok := r.runtimes[keyIndex]; ok { + runtime.exitPendingUntil = now.Add(exitPendingTTL) + } + + r.mu.Unlock() + + r.Refresh() +} + +// PrimeKeyState derives the key at the given index, applies mutate to a copy of +// its state snapshot and publishes the result. It bypasses the chain refresh, so +// it is only for tests and for priming a key before any beacon state is +// available; the next Refresh overwrites whatever it set. +func (r *Registry) PrimeKeyState(keyIndex uint64, mutate func(*State)) (*Key, error) { + key, err := r.Key(keyIndex) + if err != nil { + return nil, err + } + + state := *key.State() + mutate(&state) + key.state.Store(&state) + + r.setTracked(keyIndex, true) + r.rebuildBuilderIndex() + + return key, nil +} + +// MarkToppedUp records the epoch of a submitted top-up, arming the per-key +// cooldown that keeps a low balance from queueing duplicate deposits before the +// pending one lands. +func (r *Registry) MarkToppedUp(keyIndex uint64, epoch phase0.Epoch) { + r.mu.Lock() + + if runtime, ok := r.runtimes[keyIndex]; ok { + runtime.lastTopupEpoch = epoch + } + + r.mu.Unlock() + + r.Refresh() +} diff --git a/pkg/builder_keys/persistence_test.go b/pkg/builder_keys/persistence_test.go new file mode 100644 index 00000000..970c53c3 --- /dev/null +++ b/pkg/builder_keys/persistence_test.go @@ -0,0 +1,112 @@ +package builder_keys + +import ( + "context" + "path/filepath" + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/config" + "github.com/ethpandaops/buildoor/pkg/db" + "github.com/ethpandaops/buildoor/pkg/signer" +) + +// otherEntryKey is a second, unrelated entry key used to simulate an operator +// pointing the same state-db at different builder key material. +const otherEntryKey = "5c1d7a4b2e9f30586ca1d4b7e29f0563a8d1c4b7e29f0536a8d1c4b7e29f0536" + +func testDB(t *testing.T) *db.Database { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + stateDB := db.NewDatabase(&db.Config{File: filepath.Join(t.TempDir(), "state.db")}, log) + require.NoError(t, stateDB.Init()) + + t.Cleanup(func() { _ = stateDB.Close() }) + + return stateDB +} + +func newTestRegistry(t *testing.T, entryKey string) *Registry { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + cfg := &config.Config{BuilderKeys: config.BuilderKeysConfig{ + TargetCount: 1, + DiscoveryGap: 2, + MaxIndex: 50, + }} + + registry, err := NewRegistry(cfg, entryKey, log) + require.NoError(t, err) + + return registry +} + +// Usage history must survive a restart: a key deposited in an earlier run is +// recognised as ours before the beacon state confirms it, so the reconciler +// does not deposit for it twice. +func TestUsageSurvivesRestart(t *testing.T) { + stateDB := testDB(t) + + first := newTestRegistry(t, testEntryKey) + require.NoError(t, first.Start(context.Background(), nil, stateDB)) + + first.MarkDepositSubmitted(3) + require.NoError(t, first.usage.Flush()) + + first.Stop() + + second := newTestRegistry(t, testEntryKey) + require.NoError(t, second.Start(context.Background(), nil, stateDB)) + + defer second.Stop() + + state := second.State(3) + require.NotNil(t, state, "the key deposited before the restart must be tracked") + require.Equal(t, uint32(1), state.UseCount) + // The in-flight marker is process-local, so after a restart the key reads as + // withdrawn until the beacon state proves otherwise. + require.Equal(t, StatusWithdrawn, state.Status) + + // It is the lowest reusable index, so it is picked before any fresh index + // above it — that is what keeps the highest derivation index bounded. + require.Equal(t, uint64(0), second.NextDepositCandidate().KeyIndex()) +} + +// Pointing a state-db at different key material must fail loudly: acting on +// those records would deposit for, top up, or exit keys we do not control. +func TestStartRejectsForeignUsageRecords(t *testing.T) { + stateDB := testDB(t) + + first := newTestRegistry(t, testEntryKey) + require.NoError(t, first.Start(context.Background(), nil, stateDB)) + + first.MarkDepositSubmitted(2) + require.NoError(t, first.usage.Flush()) + + first.Stop() + + foreign := newTestRegistry(t, otherEntryKey) + + err := foreign.Start(context.Background(), nil, stateDB) + require.ErrorContains(t, err, "builder key source changed") +} + +func TestRegistryWithoutStateDBStillDerives(t *testing.T) { + registry := newTestRegistry(t, testEntryKey) + require.NoError(t, registry.Start(context.Background(), nil, nil)) + + defer registry.Stop() + + entrySigner, err := signer.NewBLSSigner(testEntryKey) + require.NoError(t, err) + require.Equal(t, entrySigner.PublicKey(), registry.Primary().Pubkey()) + require.Len(t, registry.Keys(), 1) +} diff --git a/pkg/builder_keys/registry.go b/pkg/builder_keys/registry.go new file mode 100644 index 00000000..0b7f9b2e --- /dev/null +++ b/pkg/builder_keys/registry.go @@ -0,0 +1,787 @@ +package builder_keys + +import ( + "context" + "fmt" + "slices" + "sync" + "time" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" + "github.com/sirupsen/logrus" + + "github.com/ethpandaops/buildoor/pkg/chain" + "github.com/ethpandaops/buildoor/pkg/config" + "github.com/ethpandaops/buildoor/pkg/db" + "github.com/ethpandaops/buildoor/pkg/memstore" + "github.com/ethpandaops/buildoor/pkg/utils" +) + +const ( + // defaultDiscoveryGap ends the startup scan after this many consecutive + // never-used indices when the config leaves it at zero. + defaultDiscoveryGap = 100 + // defaultMaxIndex bounds derivation when the config leaves it at zero. + defaultMaxIndex = 1000 + // depositPendingTTL is how long a submitted deposit keeps its key in the + // depositing state without any on-chain evidence. Beyond it the key falls + // back to unused/withdrawn so the reconciler retries instead of waiting + // forever on a deposit that never made it into the queue. + depositPendingTTL = 30 * time.Minute + // exitPendingTTL is the same grace period for a submitted exit request. + exitPendingTTL = 30 * time.Minute +) + +// BalanceAdjuster supplies the local balance delta of a key: credits from +// top-ups and debits from revealed payments that the latest beacon state +// snapshot does not reflect yet. Implemented by payload_bidder.PaymentTracker. +type BalanceAdjuster interface { + GetBalanceAdjustment(keyIndex uint64) int64 +} + +// ChangeEvent carries the full key set whenever any key's state changes, so +// subscribers (the WebUI bridge) can push a complete snapshot without querying. +type ChangeEvent struct { + States []*State + Aggregate Aggregate +} + +// keyRuntime is the registry-owned mutable state of one key that does not come +// from the beacon state: our own in-flight operations and rolling counters. +type keyRuntime struct { + key *Key + usage *Usage + + // depositPendingUntil keeps the key in the depositing state after we + // submitted a deposit but before the beacon state shows it. + depositPendingUntil time.Time + // exitPendingUntil keeps the key in the exiting state after we submitted an + // exit request but before the beacon state shows the withdrawable epoch. + // Without it the key keeps reading active and the reconciler re-submits the + // exit every pass, paying the queue fee each time. + exitPendingUntil time.Time + lastTopupEpoch phase0.Epoch + bidsSubmitted uint64 + bidsWon uint64 +} + +// Registry is the single owner of the builder key set: derivation, per-key +// on-chain state, usage history and selection. It is the identity dependency of +// every module that used to hold a single *signer.BLSSigner. +type Registry struct { + cfg *config.Config + entrySK string + log logrus.FieldLogger + adjuster BalanceAdjuster + + mu sync.RWMutex + // runtimes holds every derived key (the derivation cache); tracked/order + // hold the subset that forms the visible fleet. + runtimes map[uint64]*keyRuntime + tracked map[uint64]struct{} + order []uint64 + byPubkey map[phase0.BLSPubKey]*Key + byBuilderIndex map[uint64]*Key + + usage *memstore.Store[uint64, *Usage] + chainSvc chain.Service + changes utils.Dispatcher[*ChangeEvent] + + cancel context.CancelFunc + wg sync.WaitGroup +} + +// NewRegistry creates the key set rooted at the operator's entry key. It derives +// key 0 eagerly, which validates the supplied key material; every other key is +// derived on demand and cached for the process lifetime. +func NewRegistry(cfg *config.Config, entryPrivkeyHex string, log logrus.FieldLogger) (*Registry, error) { + r := &Registry{ + cfg: cfg, + entrySK: entryPrivkeyHex, + log: log.WithField("component", "builder-keys"), + runtimes: make(map[uint64]*keyRuntime, 8), + tracked: make(map[uint64]struct{}, 8), + order: make([]uint64, 0, 8), + byPubkey: make(map[phase0.BLSPubKey]*Key, 8), + byBuilderIndex: make(map[uint64]*Key, 8), + usage: memstore.New[uint64, *Usage](), + } + + if _, err := r.derive(0); err != nil { + return nil, err + } + + r.setTracked(0, true) + + return r, nil +} + +// SetBalanceAdjuster wires the source of local balance deltas. Optional: without +// it, effective balances reflect the last beacon state snapshot only. +func (r *Registry) SetBalanceAdjuster(adjuster BalanceAdjuster) { + r.mu.Lock() + r.adjuster = adjuster + r.mu.Unlock() +} + +// Start attaches usage persistence, runs the initial discovery scan and keeps +// key states in sync with every epoch's beacon state. +func (r *Registry) Start(ctx context.Context, chainSvc chain.Service, stateDB *db.Database) error { + r.mu.Lock() + r.chainSvc = chainSvc + r.mu.Unlock() + + if stateDB != nil { + r.usage.SetPersistence(ctx, db.NewKVPersistence(stateDB, Namespace, UsageCodec{}), r.log) + + if err := r.verifyPersistedKeys(); err != nil { + return err + } + } + + r.Refresh() + + runCtx, cancel := context.WithCancel(ctx) + r.cancel = cancel + + r.wg.Add(1) + + go r.run(runCtx) + + r.log.WithFields(logrus.Fields{ + "target": r.cfg.BuilderKeys.EffectiveTargetCount(), + "known_keys": len(r.order), + "primary_key": r.Primary().String(), + }).Info("Builder key registry started") + + return nil +} + +// Stop stops the registry's refresh loop and flushes usage persistence. +func (r *Registry) Stop() { + if r.cancel != nil { + r.cancel() + } + + r.wg.Wait() + r.usage.Stop() +} + +// run refreshes key states on every epoch transition: the beacon state snapshot +// that backs balances, registrations and pending payments is replaced there. +func (r *Registry) run(ctx context.Context) { + defer r.wg.Done() + + r.mu.RLock() + chainSvc := r.chainSvc + r.mu.RUnlock() + + // A nil channel simply never fires, which is what a registry without a chain + // service (tests, pre-chain startup) wants. + var epochCh <-chan *chain.EpochStats + + if chainSvc != nil { + epochSub := chainSvc.SubscribeEpochStats() + defer epochSub.Unsubscribe() + + epochCh = epochSub.Channel() + } + + // A slower standalone tick keeps locally-driven transitions (a submitted + // deposit, an expired deposit TTL) visible between epochs. + ticker := time.NewTicker(30 * time.Second) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case _, ok := <-epochCh: + if !ok { + return + } + + r.Refresh() + case <-ticker.C: + r.Refresh() + } + } +} + +// verifyPersistedKeys checks that every persisted usage record still matches the +// key its index derives to. A mismatch means the entry key changed, in which +// case the whole record set describes a different fleet and acting on it would +// deposit against, top up, or exit the wrong keys. +func (r *Registry) verifyPersistedKeys() error { + maxIndex := r.maxIndex() + + for keyIndex, usage := range r.usage.Entries() { + if keyIndex > maxIndex { + r.log.WithFields(logrus.Fields{ + "key_index": keyIndex, + "max_index": maxIndex, + }).Warn("Persisted builder key usage above the derivation cap; ignoring") + + continue + } + + key, err := r.derive(keyIndex) + if err != nil { + return err + } + + // A key we have deposited for before is part of the fleet regardless of + // where it sits relative to the current target. + r.setTracked(keyIndex, true) + + if usage.Pubkey == "" { + continue + } + + if derived := fmt.Sprintf("%#x", key.Pubkey()); usage.Pubkey != derived { + return fmt.Errorf( + "persisted builder key %d has pubkey %s but the configured entry key derives %s: "+ + "the builder key source changed, refusing to manage a foreign key set", + keyIndex, usage.Pubkey, derived) + } + } + + return nil +} + +// derive derives the key at the given internal index and caches it for the +// process lifetime. Derived is not the same as tracked: the discovery scan +// derives well past the target to look for keys we used before, but only the +// keys that matter (within the target, or known) join the visible set. +func (r *Registry) derive(keyIndex uint64) (*Key, error) { + r.mu.RLock() + runtime, ok := r.runtimes[keyIndex] + r.mu.RUnlock() + + if ok { + return runtime.key, nil + } + + key, err := newKey(r.entrySK, keyIndex) + if err != nil { + return nil, err + } + + r.mu.Lock() + defer r.mu.Unlock() + + if existing, ok := r.runtimes[keyIndex]; ok { + return existing.key, nil + } + + r.runtimes[keyIndex] = &keyRuntime{key: key} + r.byPubkey[key.Pubkey()] = key + + return key, nil +} + +// setTracked adds or removes a key from the visible set. Tracked keys are the +// ones the fleet consists of: everything within the target count plus every key +// with an on-chain entry, a deposit in flight, or a usage history. +func (r *Registry) setTracked(keyIndex uint64, tracked bool) { + r.mu.Lock() + defer r.mu.Unlock() + + _, known := r.tracked[keyIndex] + if known == tracked { + return + } + + if tracked { + r.tracked[keyIndex] = struct{}{} + r.order = append(r.order, keyIndex) + + slices.Sort(r.order) + + return + } + + delete(r.tracked, keyIndex) + + r.order = slices.DeleteFunc(r.order, func(index uint64) bool { return index == keyIndex }) +} + +// highestTracked returns the highest tracked key index, or 0 when none is +// tracked yet. +func (r *Registry) highestTracked() uint64 { + r.mu.RLock() + defer r.mu.RUnlock() + + if len(r.order) == 0 { + return 0 + } + + return r.order[len(r.order)-1] +} + +// maxIndex returns the effective derivation cap. +func (r *Registry) maxIndex() uint64 { + if r.cfg.BuilderKeys.MaxIndex > 0 { + return r.cfg.BuilderKeys.MaxIndex + } + + return defaultMaxIndex +} + +// discoveryGap returns the effective number of consecutive unused indices that +// ends the discovery scan. +func (r *Registry) discoveryGap() uint64 { + if r.cfg.BuilderKeys.DiscoveryGap > 0 { + return r.cfg.BuilderKeys.DiscoveryGap + } + + return defaultDiscoveryGap +} + +// Refresh recomputes every key's state from the current beacon state and rescans +// for keys we have used before. +// +// The scan always covers the target count and everything already tracked, and +// keeps going past that while it finds keys of ours — stopping only after a full +// run of never-used indices (the discovery gap). Unused indices above the target +// are derived but not tracked, so the fleet view stays the size of the fleet +// rather than the size of the scan. Fires a ChangeEvent when anything changed. +func (r *Registry) Refresh() { + snapshot := r.chainSnapshot() + + target := r.cfg.BuilderKeys.EffectiveTargetCount() + maxIndex := r.maxIndex() + gapLimit := r.discoveryGap() + + // Never stop before the target or before the highest key we already know + // about: a key touched by an operation must be refreshed even when it sits + // far above the target. + scanFloor := max(target, r.highestTracked()+1) + + changed := false + gap := uint64(0) + + for keyIndex := uint64(0); keyIndex <= maxIndex; keyIndex++ { + if keyIndex >= scanFloor && gap >= gapLimit { + break + } + + key, err := r.derive(keyIndex) + if err != nil { + r.log.WithError(err).WithField("key_index", keyIndex).Error("Failed to derive builder key") + + break + } + + state, dirty := r.refreshKey(key, snapshot) + changed = changed || dirty + + known := state.Status != StatusUnused + r.setTracked(keyIndex, known || keyIndex < target) + + switch { + case keyIndex < target: + // Always covered; these do not count toward the discovery gap. + case known: + gap = 0 + scanFloor = max(scanFloor, keyIndex+1) + default: + gap++ + } + } + + r.rebuildBuilderIndex() + + if changed { + r.changes.Fire(&ChangeEvent{States: r.States(), Aggregate: r.Aggregate()}) + } +} + +// chainState is the per-refresh view of the beacon state, resolved once so a +// refresh over hundreds of keys stays a single pass. +type chainState struct { + builders map[phase0.BLSPubKey]*chain.BuilderInfo + pendingDeposits map[phase0.BLSPubKey]struct{} + currentEpoch phase0.Epoch + finalizedEpoch uint64 +} + +// chainSnapshot builds the per-refresh beacon state view. +func (r *Registry) chainSnapshot() *chainState { + r.mu.RLock() + chainSvc := r.chainSvc + r.mu.RUnlock() + + snapshot := &chainState{ + builders: map[phase0.BLSPubKey]*chain.BuilderInfo{}, + pendingDeposits: map[phase0.BLSPubKey]struct{}{}, + } + + if chainSvc == nil { + return snapshot + } + + snapshot.currentEpoch = chainSvc.GetCurrentEpoch() + snapshot.finalizedEpoch = uint64(chainSvc.GetFinalizedEpoch()) + + for _, info := range chainSvc.GetBuilders() { + if info != nil { + snapshot.builders[info.Pubkey] = info + } + } + + if stats := chainSvc.GetCurrentEpochStats(); stats != nil { + for i := range stats.PendingDeposits { + snapshot.pendingDeposits[stats.PendingDeposits[i].Pubkey] = struct{}{} + } + } + + return snapshot +} + +// refreshKey recomputes one key's state snapshot, returning it plus whether it +// differs from the previous one. +func (r *Registry) refreshKey(key *Key, snapshot *chainState) (*State, bool) { + r.mu.Lock() + + runtime := r.runtimes[key.KeyIndex()] + if runtime == nil { + r.mu.Unlock() + + return key.State(), false + } + + usage, _ := r.usage.Get(key.KeyIndex()) + runtime.usage = usage + adjuster := r.adjuster + + state := &State{ + KeyIndex: key.KeyIndex(), + Pubkey: key.Pubkey(), + PubkeyHex: fmt.Sprintf("%#x", key.Pubkey()), + LastTopupEpoch: runtime.lastTopupEpoch, + BidsSubmitted: runtime.bidsSubmitted, + BidsWon: runtime.bidsWon, + } + + if usage != nil { + state.UseCount = usage.UseCount + state.LastDepositAt = usage.LastDepositAt + state.LastExitAt = usage.LastExitAt + } + + info := snapshot.builders[key.Pubkey()] + _, queued := snapshot.pendingDeposits[key.Pubkey()] + depositInFlight := queued || time.Now().Before(runtime.depositPendingUntil) + exitInFlight := time.Now().Before(runtime.exitPendingUntil) + + r.mu.Unlock() + + if info != nil { + state.BuilderIndex = info.Index + state.HasBuilderIndex = true + state.Balance = info.Balance + state.PendingPayments = info.PendingPayments + state.DepositEpoch = info.DepositEpoch + state.WithdrawableEpoch = info.WithdrawableEpoch + } + + state.Status = resolveStatus(info, depositInFlight, exitInFlight, state.UseCount, + snapshot.currentEpoch, snapshot.finalizedEpoch) + + if adjuster != nil { + state.BalanceAdjustment = adjuster.GetBalanceAdjustment(key.KeyIndex()) + } + + state.EffectiveBalance = effectiveBalance(state) + + previous := key.State() + if *previous == *state { + return previous, false + } + + key.state.Store(state) + + if previous.Status != state.Status { + r.log.WithFields(logrus.Fields{ + "key": key.String(), + "from": previous.Status, + "to": state.Status, + }).Info("Builder key status changed") + } + + return state, true +} + +// resolveStatus decides a key's lifecycle position from its on-chain registry +// entry, whether an operation of ours is in flight, and how often the key has +// been used before. +// +// The in-flight checks must precede the on-chain ones. A key whose first deposit +// was just submitted already has a non-zero use count while its pubkey is still +// absent from the registry, and reading that as "withdrawn" would let the +// reconciler deposit for it a second time. A key whose exit was just submitted +// still reads active until the beacon state carries the withdrawable epoch, +// which would have the reconciler re-submit the exit — paying the queue fee — +// on every pass until then. +func resolveStatus( + info *chain.BuilderInfo, + depositInFlight, exitInFlight bool, + useCount uint32, + currentEpoch phase0.Epoch, + finalizedEpoch uint64, +) Status { + switch { + case info == nil && depositInFlight: + return StatusDepositing + case info == nil && useCount > 0: + // Used before and gone from the registry: depositable again. + return StatusWithdrawn + case info == nil: + return StatusUnused + case chain.HasBuilderExited(info): + if uint64(currentEpoch) >= info.WithdrawableEpoch { + return StatusExited + } + + return StatusExiting + case exitInFlight: + return StatusExiting + case chain.IsBuilderActive(info, finalizedEpoch): + return StatusActive + default: + return StatusPending + } +} + +// effectiveBalance applies the local adjustment and pending payments to the +// snapshot balance, flooring at zero. +func effectiveBalance(state *State) uint64 { + live := max(int64(state.Balance)+state.BalanceAdjustment, 0) //nolint:gosec // gwei balances stay far below int64 + + balance := uint64(live) + if state.PendingPayments >= balance { + return 0 + } + + return balance - state.PendingPayments +} + +// rebuildBuilderIndex refreshes the builder-index lookup used to resolve the key +// behind a block's winning bid. +func (r *Registry) rebuildBuilderIndex() { + r.mu.Lock() + defer r.mu.Unlock() + + byIndex := make(map[uint64]*Key, len(r.runtimes)) + + for _, runtime := range r.runtimes { + if builderIndex, ok := runtime.key.BuilderIndex(); ok { + byIndex[builderIndex] = runtime.key + } + } + + r.byBuilderIndex = byIndex +} + +// Key returns the derived key at the given internal index, deriving it if +// needed. +func (r *Registry) Key(keyIndex uint64) (*Key, error) { + if keyIndex > r.maxIndex() { + return nil, fmt.Errorf("builder key index %d exceeds the derivation cap %d", keyIndex, r.maxIndex()) + } + + return r.derive(keyIndex) +} + +// Keys returns every tracked key, key-index ascending. +func (r *Registry) Keys() []*Key { + r.mu.RLock() + defer r.mu.RUnlock() + + keys := make([]*Key, 0, len(r.order)) + for _, keyIndex := range r.order { + keys = append(keys, r.runtimes[keyIndex].key) + } + + return keys +} + +// Primary returns the key that stands in wherever a single builder identity is +// required (the pre-Gloas Builder API, the legacy lifecycle endpoints): the +// lowest-index active key, or the entry key when none is active. +func (r *Registry) Primary() *Key { + r.mu.RLock() + defer r.mu.RUnlock() + + var fallback *Key + + for _, keyIndex := range r.order { + key := r.runtimes[keyIndex].key + if key.Status() == StatusActive { + return key + } + + if fallback == nil { + fallback = key + } + } + + return fallback +} + +// ByPubkey returns the key with the given public key, or nil. +func (r *Registry) ByPubkey(pubkey phase0.BLSPubKey) *Key { + r.mu.RLock() + defer r.mu.RUnlock() + + return r.byPubkey[pubkey] +} + +// ByBuilderIndex returns the key registered under the given on-chain builder +// index, or nil when the index is not ours. This is how a won block's bid is +// traced back to the key that must sign the reveal. +func (r *Registry) ByBuilderIndex(builderIndex uint64) *Key { + r.mu.RLock() + defer r.mu.RUnlock() + + return r.byBuilderIndex[builderIndex] +} + +// State returns the state snapshot of one derived key, or nil when the index has +// not been derived. +func (r *Registry) State(keyIndex uint64) *State { + r.mu.RLock() + runtime, ok := r.runtimes[keyIndex] + r.mu.RUnlock() + + if !ok { + return nil + } + + return runtime.key.State() +} + +// EffectiveBalance returns the key's live spendable balance in gwei: the latest +// beacon snapshot combined with the current local adjustment, rather than the +// adjustment captured at the last refresh. Bid readiness and top-up decisions +// read this — a payment settled seconds ago must not still look spendable. +func (r *Registry) EffectiveBalance(keyIndex uint64) uint64 { + state := r.State(keyIndex) + if state == nil { + return 0 + } + + r.mu.RLock() + adjuster := r.adjuster + r.mu.RUnlock() + + live := *state + if adjuster != nil { + live.BalanceAdjustment = adjuster.GetBalanceAdjustment(keyIndex) + } + + return effectiveBalance(&live) +} + +// States returns a snapshot of every derived key's state, key-index ascending. +func (r *Registry) States() []*State { + keys := r.Keys() + + states := make([]*State, 0, len(keys)) + for _, key := range keys { + states = append(states, key.State()) + } + + return states +} + +// Aggregate summarises the key set for the dashboard. +func (r *Registry) Aggregate() Aggregate { + aggregate := Aggregate{Target: r.cfg.BuilderKeys.EffectiveTargetCount()} + + for _, state := range r.States() { + switch state.Status { + case StatusUnused: + aggregate.Unused++ + case StatusDepositing: + aggregate.Depositing++ + case StatusPending: + aggregate.Pending++ + case StatusActive: + aggregate.Active++ + case StatusExiting: + aggregate.Exiting++ + case StatusExited: + aggregate.Exited++ + case StatusWithdrawn: + aggregate.Withdrawn++ + } + + if state.Status.Managed() { + aggregate.Managed++ + } + + if state.HasBuilderIndex { + aggregate.TotalBalance += state.Balance + aggregate.TotalPendingPayments += state.PendingPayments + aggregate.TotalEffective += state.EffectiveBalance + } + } + + return aggregate +} + +// AnyActive reports whether at least one key is active on chain — the fleet-wide +// availability gate that replaces the single-builder registration check. +func (r *Registry) AnyActive() bool { + for _, key := range r.Keys() { + if key.Status() == StatusActive { + return true + } + } + + return false +} + +// SubscribeChanges subscribes to key set changes. +func (r *Registry) SubscribeChanges(capacity int, blocking bool) *utils.Subscription[*ChangeEvent] { + return r.changes.Subscribe(capacity, blocking) +} + +// RecordBid counts a submitted bid against a key. The counter feeds the +// least-used selection strategy and the UI, so it is published to the key's +// state snapshot immediately rather than at the next refresh — selection +// happens many times per slot, refreshes once per epoch. +func (r *Registry) RecordBid(keyIndex uint64) { + r.bumpCounters(keyIndex, 1, 0) +} + +// RecordWin counts a won slot against a key. +func (r *Registry) RecordWin(keyIndex uint64) { + r.bumpCounters(keyIndex, 0, 1) +} + +// bumpCounters advances a key's rolling counters in both the runtime (which +// survives refreshes) and the published state snapshot. +func (r *Registry) bumpCounters(keyIndex, bids, wins uint64) { + r.mu.Lock() + + runtime, ok := r.runtimes[keyIndex] + if !ok { + r.mu.Unlock() + return + } + + runtime.bidsSubmitted += bids + runtime.bidsWon += wins + + key := runtime.key + state := *key.State() + state.BidsSubmitted = runtime.bidsSubmitted + state.BidsWon = runtime.bidsWon + + key.state.Store(&state) + + r.mu.Unlock() +} diff --git a/pkg/builder_keys/registry_test.go b/pkg/builder_keys/registry_test.go new file mode 100644 index 00000000..408c45e8 --- /dev/null +++ b/pkg/builder_keys/registry_test.go @@ -0,0 +1,445 @@ +package builder_keys + +import ( + "fmt" + "testing" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/chain" + "github.com/ethpandaops/buildoor/pkg/config" + "github.com/ethpandaops/buildoor/pkg/signer" +) + +const testEntryKey = "3f2b8e1c9d4a6f70b5c8e2a1d7943f6058ac2be91d3f5074a6b8c2e1d9f30475" + +func testRegistry(t *testing.T, keys config.BuilderKeysConfig) *Registry { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + registry, err := NewRegistry(&config.Config{BuilderKeys: keys}, testEntryKey, log) + require.NoError(t, err) + + return registry +} + +func TestRegistryPrimaryIsTheEntryKey(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 4}) + + entrySigner, err := signer.NewBLSSigner(testEntryKey) + require.NoError(t, err) + + primary := registry.Primary() + require.NotNil(t, primary) + require.Equal(t, uint64(0), primary.KeyIndex()) + require.Equal(t, entrySigner.PublicKey(), primary.Pubkey()) +} + +func TestRegistryDerivesUpToTarget(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 5, DiscoveryGap: 3, MaxIndex: 100}) + registry.Refresh() + + keys := registry.Keys() + // The scan reaches past the target looking for used keys, but unused + // indices above it are not tracked: the fleet is the size of the target. + require.Len(t, keys, 5) + + pubkeys := map[phase0.BLSPubKey]uint64{} + for _, key := range keys { + previous, seen := pubkeys[key.Pubkey()] + require.False(t, seen, "key %d duplicates key %d", key.KeyIndex(), previous) + + pubkeys[key.Pubkey()] = key.KeyIndex() + require.Equal(t, StatusUnused, key.Status()) + } +} + +// Discovery must keep scanning past the target for keys we deposited in an +// earlier run, and stop only after a full gap of never-used indices. +func TestRegistryDiscoveryFindsUsedKeysAboveTarget(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 1, DiscoveryGap: 4, MaxIndex: 100}) + + // A key deposited in a previous run, well above the current target. + registry.MarkDepositSubmitted(9) + + registry.Refresh() + + state := registry.State(9) + require.NotNil(t, state) + require.Equal(t, StatusDepositing, state.Status, "a just-submitted deposit stays in flight") + require.Equal(t, uint32(1), state.UseCount) + + // The fleet is the target (key 0) plus the key we used before, not the whole + // scanned range. + indices := make([]uint64, 0, 2) + for _, key := range registry.Keys() { + indices = append(indices, key.KeyIndex()) + } + + require.Equal(t, []uint64{0, 9}, indices) +} + +// A used key far above the target must survive every later refresh: the scan +// floor follows the highest key we know about, not the target. +func TestRegistryKeepsUsedKeysAcrossRefreshes(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 1, DiscoveryGap: 2, MaxIndex: 100}) + + registry.MarkDepositSubmitted(7) + + for range 3 { + registry.Refresh() + } + + state := registry.State(7) + require.NotNil(t, state) + require.Equal(t, StatusDepositing, state.Status) + require.Len(t, registry.Keys(), 2) +} + +func TestRegistryRespectsMaxIndex(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 20, DiscoveryGap: 5, MaxIndex: 3}) + registry.Refresh() + + require.Len(t, registry.Keys(), 4, "the target is clamped to the cap, inclusive") + + _, err := registry.Key(4) + require.ErrorContains(t, err, "derivation cap") +} + +func TestRegistryDepositCandidatePrefersLowestReusableIndex(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 4, DiscoveryGap: 2, MaxIndex: 100}) + registry.Refresh() + + candidate := registry.NextDepositCandidate() + require.NotNil(t, candidate) + require.Equal(t, uint64(0), candidate.KeyIndex()) +} + +func TestRegistryStatusResolution(t *testing.T) { + const finalized = uint64(10) + + active := &chain.BuilderInfo{Index: 3, DepositEpoch: 5, WithdrawableEpoch: chain.FarFutureEpoch} + unfinalized := &chain.BuilderInfo{Index: 4, DepositEpoch: 12, WithdrawableEpoch: chain.FarFutureEpoch} + exiting := &chain.BuilderInfo{Index: 5, DepositEpoch: 5, WithdrawableEpoch: 40} + exited := &chain.BuilderInfo{Index: 6, DepositEpoch: 5, WithdrawableEpoch: 20} + + tests := []struct { + name string + info *chain.BuilderInfo + depositInFlight bool + exitInFlight bool + useCount uint32 + want Status + }{ + {name: "never used", want: StatusUnused}, + {name: "deposit in flight", depositInFlight: true, want: StatusDepositing}, + { + name: "first deposit in flight outranks its own use count", + depositInFlight: true, + useCount: 1, + want: StatusDepositing, + }, + {name: "used and gone from the registry", useCount: 2, want: StatusWithdrawn}, + {name: "registered but unfinalized", info: unfinalized, want: StatusPending}, + {name: "registered and finalized", info: active, want: StatusActive}, + {name: "exit initiated", info: exiting, want: StatusExiting}, + {name: "withdrawable epoch reached", info: exited, want: StatusExited}, + { + // Until the beacon state carries the withdrawable epoch the key + // still reads active, and the reconciler would re-submit the exit — + // paying the queue fee — on every pass. + name: "exit submitted but not yet on chain", + info: active, + exitInFlight: true, + want: StatusExiting, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + got := resolveStatus(test.info, test.depositInFlight, test.exitInFlight, + test.useCount, phase0.Epoch(25), finalized) + require.Equal(t, test.want, got) + }) + } +} + +func TestEffectiveBalanceFloorsAtZero(t *testing.T) { + tests := []struct { + name string + state State + want uint64 + }{ + {name: "plain balance", state: State{Balance: 100}, want: 100}, + {name: "credit", state: State{Balance: 100, BalanceAdjustment: 25}, want: 125}, + {name: "debit", state: State{Balance: 100, BalanceAdjustment: -40}, want: 60}, + {name: "debit past zero", state: State{Balance: 10, BalanceAdjustment: -40}, want: 0}, + {name: "pending payments", state: State{Balance: 100, PendingPayments: 30}, want: 70}, + {name: "pending payments exceed balance", state: State{Balance: 10, PendingPayments: 30}, want: 0}, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + require.Equal(t, test.want, effectiveBalance(&test.state)) + }) + } +} + +func TestStateReadyRequiresActiveAndFunded(t *testing.T) { + require.True(t, (&State{Status: StatusActive, EffectiveBalance: 100}).Ready(100)) + require.False(t, (&State{Status: StatusActive, EffectiveBalance: 99}).Ready(100)) + require.False(t, (&State{Status: StatusPending, EffectiveBalance: 1000}).Ready(100)) + require.False(t, (&State{Status: StatusExiting, EffectiveBalance: 1000}).Ready(100)) +} + +func TestRegistryAggregateCountsStatuses(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 3, DiscoveryGap: 1, MaxIndex: 100}) + registry.Refresh() + + aggregate := registry.Aggregate() + require.Equal(t, uint64(3), aggregate.Target) + require.Equal(t, uint64(0), aggregate.Managed) + require.Equal(t, uint64(len(registry.Keys())), aggregate.Unused) + require.False(t, registry.AnyActive()) +} + +func TestRegistryChangeEventsOnStatusTransition(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 2, DiscoveryGap: 1, MaxIndex: 100}) + registry.Refresh() + + sub := registry.SubscribeChanges(4, false) + defer sub.Unsubscribe() + + registry.MarkDepositSubmitted(1) + + select { + case event := <-sub.Channel(): + require.NotNil(t, event) + + var found bool + + for _, state := range event.States { + if state.KeyIndex == 1 { + found = true + + require.Equal(t, StatusDepositing, state.Status) + } + } + + require.True(t, found, "change event must carry the whole key set") + default: + t.Fatal("expected a change event after a deposit was recorded") + } +} + +func TestRegistryRefreshIsIdempotent(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 2, DiscoveryGap: 2, MaxIndex: 100}) + registry.Refresh() + + before := len(registry.Keys()) + + sub := registry.SubscribeChanges(4, false) + defer sub.Unsubscribe() + + registry.Refresh() + registry.Refresh() + + require.Len(t, registry.Keys(), before) + + select { + case event := <-sub.Channel(): + t.Fatalf("unchanged refresh must not fire an event, got %v", event) + default: + } +} + +func TestRegistryLookupsByPubkey(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 3, DiscoveryGap: 1, MaxIndex: 100}) + registry.Refresh() + + for _, key := range registry.Keys() { + require.Same(t, key, registry.ByPubkey(key.Pubkey())) + } + + require.Nil(t, registry.ByPubkey(phase0.BLSPubKey{0xde, 0xad})) + require.Nil(t, registry.ByBuilderIndex(42), "no key is registered on chain in this fixture") +} + +// Every consumer that signs — p2p bids, Builder API bids, reveal envelopes — +// takes its builder index from the key's state, and a key that reads +// unregistered is refused rather than signed with the zero index. So the whole +// fleet's ability to bid rests on the registry resolving indexes from beacon +// state on its own, with no registration callback involved: a buildoor started +// without --lifecycle (no --el-rpc / --wallet-privkey) never sees one, and a +// fleet deposited in an earlier run must come up bidding regardless. +func TestRegistryResolvesBuilderIndexesFromChainState(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 3, DiscoveryGap: 1, MaxIndex: 100}) + + // Three keys already registered on chain, at indexes that do not line up + // with our derivation indexes — the two spaces are unrelated. + chainSvc := &stubChainService{currentEpoch: 30, finalizedEpoch: 20} + for keyIndex, builderIndex := range []uint64{11, 7, 25} { + key, err := registry.Key(uint64(keyIndex)) + require.NoError(t, err) + + chainSvc.builders = append(chainSvc.builders, &chain.BuilderInfo{ + Index: builderIndex, + Pubkey: key.Pubkey(), + Balance: 2_000_000_000, + DepositEpoch: 5, + WithdrawableEpoch: chain.FarFutureEpoch, + }) + } + + require.NoError(t, registry.Start(t.Context(), chainSvc, nil)) + defer registry.Stop() + + for keyIndex, want := range []uint64{11, 7, 25} { + key, err := registry.Key(uint64(keyIndex)) + require.NoError(t, err) + + builderIndex, registered := key.BuilderIndex() + require.True(t, registered, "key %d must resolve its index from chain state alone", keyIndex) + require.Equal(t, want, builderIndex) + require.Equal(t, StatusActive, key.Status()) + require.Same(t, key, registry.ByBuilderIndex(want), "reveals resolve the winning key by index") + } + + require.Equal(t, uint64(3), registry.Aggregate().Active) +} + +func TestKeyStringIdentifiesTheDerivationIndex(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 2}) + + key, err := registry.Key(1) + require.NoError(t, err) + + pubkey := key.Pubkey() + require.Equal(t, fmt.Sprintf("#1/%x", pubkey[:4]), key.String()) +} + +// The beacon chain silently ignores an exit request while the builder still owes +// a payment, so a key with pending payments is not exitable — and the exit waits +// for it rather than skipping down to a lower key, which would burn a usable key +// while leaving the surplus one in place. +func TestRegistryExitCandidateWaitsOnPendingPayments(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 4, DiscoveryGap: 1, MaxIndex: 32}) + + active := func(state *State) { + state.Status = StatusActive + state.HasBuilderIndex = true + state.BuilderIndex = state.KeyIndex + 10 + } + + for keyIndex := range uint64(3) { + _, err := registry.PrimeKeyState(keyIndex, active) + require.NoError(t, err) + } + + // The highest key still owes a payment. + _, err := registry.PrimeKeyState(2, func(state *State) { + active(state) + state.PendingPayments = 500 + }) + require.NoError(t, err) + + require.Nil(t, registry.NextExitCandidate(), + "a lower key must not be exited in place of the highest one") + + // Once it settles, it is the one to go. + _, err = registry.PrimeKeyState(2, func(state *State) { + active(state) + state.PendingPayments = 0 + }) + require.NoError(t, err) + + candidate := registry.NextExitCandidate() + require.NotNil(t, candidate) + require.Equal(t, uint64(2), candidate.KeyIndex()) +} + +// Deposits reuse the lowest withdrawn index instead of extending the set, which +// is what keeps the highest derivation index bounded across target ramps. +func TestRegistryDepositCandidateReusesWithdrawnKeys(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 3, DiscoveryGap: 1, MaxIndex: 32}) + + for keyIndex := range uint64(3) { + _, err := registry.PrimeKeyState(keyIndex, func(state *State) { + state.Status = StatusActive + state.HasBuilderIndex = true + state.BuilderIndex = state.KeyIndex + 10 + }) + require.NoError(t, err) + } + + // With every key in use the next deposit extends the set. + require.Equal(t, uint64(3), registry.NextDepositCandidate().KeyIndex()) + + // Once key 1 leaves the registry it becomes the preferred candidate again. + _, err := registry.PrimeKeyState(1, func(state *State) { + state.Status = StatusWithdrawn + state.HasBuilderIndex = false + state.UseCount = 1 + }) + require.NoError(t, err) + + require.Equal(t, uint64(1), registry.NextDepositCandidate().KeyIndex()) +} + +func TestStatusClassification(t *testing.T) { + managed := map[Status]bool{ + StatusUnused: false, StatusDepositing: true, StatusPending: true, + StatusActive: true, StatusExiting: false, StatusExited: false, + StatusWithdrawn: false, + } + depositable := map[Status]bool{ + StatusUnused: true, StatusDepositing: false, StatusPending: false, + StatusActive: false, StatusExiting: false, StatusExited: false, + StatusWithdrawn: true, + } + + for status, want := range managed { + require.Equal(t, want, status.Managed(), "Managed(%s)", status) + } + + for status, want := range depositable { + require.Equal(t, want, status.Depositable(), "Depositable(%s)", status) + } +} + +// Keys still on their way to active are the real surplus after a target cut: +// exiting a lower, usable key in their place burns a key we just paid for and +// leaves the fleet no smaller once they register. +func TestRegistryExitCandidateWaitsOnInFlightKeys(t *testing.T) { + registry := testRegistry(t, config.BuilderKeysConfig{TargetCount: 3, DiscoveryGap: 1, MaxIndex: 32}) + + for keyIndex := range uint64(2) { + _, err := registry.PrimeKeyState(keyIndex, func(state *State) { + state.Status = StatusActive + state.HasBuilderIndex = true + state.BuilderIndex = state.KeyIndex + 10 + }) + require.NoError(t, err) + } + + _, err := registry.PrimeKeyState(2, func(state *State) { state.Status = StatusPending }) + require.NoError(t, err) + + require.Nil(t, registry.NextExitCandidate(), "wait for the in-flight key instead of exiting a usable one") + + // Once it activates, it is the one to go. + _, err = registry.PrimeKeyState(2, func(state *State) { + state.Status = StatusActive + state.HasBuilderIndex = true + state.BuilderIndex = 12 + }) + require.NoError(t, err) + + candidate := registry.NextExitCandidate() + require.NotNil(t, candidate) + require.Equal(t, uint64(2), candidate.KeyIndex()) +} diff --git a/pkg/builder_keys/selection.go b/pkg/builder_keys/selection.go new file mode 100644 index 00000000..9aec132c --- /dev/null +++ b/pkg/builder_keys/selection.go @@ -0,0 +1,142 @@ +package builder_keys + +import ( + "math/rand/v2" + "slices" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" +) + +// Key selection strategies: which of the ready keys a bid is signed with when +// more keys are ready than the slot needs. +const ( + // StrategyRoundRobin rotates through the ready keys by slot, spreading bids + // and their payments evenly across the fleet. + StrategyRoundRobin = "round_robin" + // StrategySingle always uses the lowest-index ready key. One key bids per + // slot, which is how a single-key deployment behaves. + StrategySingle = "single" + // StrategyRandom shuffles the ready keys per slot (deterministically, so a + // slot's assignment is reproducible). + StrategyRandom = "random" + // StrategyLeastUsed prefers the keys that have bid least, which keeps + // balance drain even when slots are not evenly distributed. + StrategyLeastUsed = "least_used" +) + +// NormalizedStrategy returns the strategy, falling back to round-robin for +// unknown values (config and plan overrides are free-form strings). +func NormalizedStrategy(strategy string) string { + switch strategy { + case StrategyRoundRobin, StrategySingle, StrategyRandom, StrategyLeastUsed: + return strategy + default: + return StrategyRoundRobin + } +} + +// SelectRequest describes the keys a caller needs for one slot. +// +// A key may only bid once per slot: the gossip rules ignore every bid after a +// builder's first for a slot, so a second bid from the same key never +// propagates. Callers therefore ask for distinct keys and pass the ones they +// already committed in Exclude. +type SelectRequest struct { + // Strategy is the selection strategy; unknown values fall back to + // round-robin. + Strategy string + // RequiredGwei is the bid value a returned key should be able to cover: a + // builder whose effective balance is below its bid has the bid rejected on + // chain. It is a preference, not a filter — keys that can cover it come + // first, and an underfunded key is still returned when nothing else is + // available, because deliberately underfunded bids are a scenario buildoor + // exists to test. + RequiredGwei uint64 + // Count caps how many keys to return; 0 means all ready keys. + Count uint64 + // Exclude holds key indices already committed for this slot. + Exclude map[uint64]struct{} +} + +// SelectForBid returns the keys to sign a slot's bids with, ordered by the +// requested strategy. It returns nil when no ready key qualifies. +func (r *Registry) SelectForBid(slot phase0.Slot, req SelectRequest) []*Key { + funded := make([]*Key, 0, 4) + underfunded := make([]*Key, 0, 2) + + for _, key := range r.Keys() { + if _, excluded := req.Exclude[key.KeyIndex()]; excluded { + continue + } + + if key.State().Status != StatusActive { + continue + } + + // Balances move between refreshes (a payment settles, a top-up lands), + // so this reads the live effective balance rather than the snapshot's. + if r.EffectiveBalance(key.KeyIndex()) >= req.RequiredGwei { + funded = append(funded, key) + } else { + underfunded = append(underfunded, key) + } + } + + if len(funded) == 0 && len(underfunded) == 0 { + return nil + } + + strategy := NormalizedStrategy(req.Strategy) + + // Funded keys first, so an underfunded one is only reached once nothing + // else can cover the bid. + ordered := make([]*Key, 0, len(funded)+len(underfunded)) + ordered = append(ordered, orderForStrategy(funded, strategy, slot)...) + ordered = append(ordered, orderForStrategy(underfunded, strategy, slot)...) + + if req.Count > 0 && uint64(len(ordered)) > req.Count { + ordered = ordered[:req.Count] + } + + return ordered +} + +// orderForStrategy orders ready keys (key-index ascending on input) per the +// strategy. The order decides which keys win when there are more ready keys +// than the slot needs. +func orderForStrategy(ready []*Key, strategy string, slot phase0.Slot) []*Key { + if len(ready) == 0 { + return nil + } + + switch strategy { + case StrategySingle: + return ready[:1] + + case StrategyRandom: + // Seeded from the slot so a slot's assignment is reproducible across + // re-evaluations within the slot (and across a restart mid-slot). + shuffled := slices.Clone(ready) + rng := rand.New(rand.NewPCG(uint64(slot), 0x6275696c64)) + + rng.Shuffle(len(shuffled), func(i, j int) { + shuffled[i], shuffled[j] = shuffled[j], shuffled[i] + }) + + return shuffled + + case StrategyLeastUsed: + sorted := slices.Clone(ready) + slices.SortStableFunc(sorted, func(a, b *Key) int { + return int(a.State().BidsSubmitted) - int(b.State().BidsSubmitted) //nolint:gosec // counters stay far below int range + }) + + return sorted + + default: // StrategyRoundRobin + rotated := slices.Clone(ready) + offset := int(uint64(slot) % uint64(len(rotated))) //nolint:gosec // bounded by len + + return append(rotated[offset:], rotated[:offset]...) + } +} diff --git a/pkg/builder_keys/selection_test.go b/pkg/builder_keys/selection_test.go new file mode 100644 index 00000000..54851887 --- /dev/null +++ b/pkg/builder_keys/selection_test.go @@ -0,0 +1,154 @@ +package builder_keys + +import ( + "testing" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/config" +) + +// activeRegistry builds a registry with `count` active keys, each holding +// balanceGwei. +func activeRegistry(t *testing.T, count, balanceGwei uint64) *Registry { + t.Helper() + + registry := testRegistry(t, config.BuilderKeysConfig{ + TargetCount: count, DiscoveryGap: 1, MaxIndex: 64, + }) + + for keyIndex := range count { + _, err := registry.PrimeKeyState(keyIndex, func(state *State) { + state.Status = StatusActive + state.HasBuilderIndex = true + state.BuilderIndex = state.KeyIndex + 100 + state.Balance = balanceGwei + state.EffectiveBalance = balanceGwei + }) + require.NoError(t, err) + } + + return registry +} + +func selectedIndices(keys []*Key) []uint64 { + indices := make([]uint64, 0, len(keys)) + for _, key := range keys { + indices = append(indices, key.KeyIndex()) + } + + return indices +} + +func TestSelectForBidStrategies(t *testing.T) { + registry := activeRegistry(t, 4, 1_000) + + tests := []struct { + name string + strategy string + slot phase0.Slot + want []uint64 + }{ + {name: "single always takes the lowest index", strategy: StrategySingle, slot: 7, want: []uint64{0}}, + {name: "round robin rotates by slot", strategy: StrategyRoundRobin, slot: 6, want: []uint64{2, 3, 0, 1}}, + {name: "round robin wraps", strategy: StrategyRoundRobin, slot: 8, want: []uint64{0, 1, 2, 3}}, + {name: "unknown falls back to round robin", strategy: "bogus", slot: 5, want: []uint64{1, 2, 3, 0}}, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + got := registry.SelectForBid(test.slot, SelectRequest{Strategy: test.strategy}) + require.Equal(t, test.want, selectedIndices(got)) + }) + } +} + +func TestSelectForBidRandomIsDeterministicPerSlot(t *testing.T) { + registry := activeRegistry(t, 5, 1_000) + + first := selectedIndices(registry.SelectForBid(11, SelectRequest{Strategy: StrategyRandom})) + again := selectedIndices(registry.SelectForBid(11, SelectRequest{Strategy: StrategyRandom})) + require.Equal(t, first, again, "the same slot must yield the same assignment") + + other := selectedIndices(registry.SelectForBid(12, SelectRequest{Strategy: StrategyRandom})) + require.ElementsMatch(t, first, other) +} + +func TestSelectForBidLeastUsedPrefersIdleKeys(t *testing.T) { + registry := activeRegistry(t, 3, 1_000) + + registry.RecordBid(0) + registry.RecordBid(0) + registry.RecordBid(1) + + got := registry.SelectForBid(1, SelectRequest{Strategy: StrategyLeastUsed}) + require.Equal(t, []uint64{2, 1, 0}, selectedIndices(got)) +} + +// A key already committed to a candidate must not be handed out again: its +// second bid for the slot would be ignored by the gossip rules. +func TestSelectForBidExcludesCommittedKeys(t *testing.T) { + registry := activeRegistry(t, 3, 1_000) + + got := registry.SelectForBid(0, SelectRequest{ + Strategy: StrategyRoundRobin, + Exclude: map[uint64]struct{}{0: {}, 2: {}}, + }) + + require.Equal(t, []uint64{1}, selectedIndices(got)) +} + +// Balance is a preference, not a filter: an underfunded key is still offered +// once nothing else can cover the bid, because deliberately underfunded bids +// are a scenario buildoor exists to test. +func TestSelectForBidPrefersFundedKeys(t *testing.T) { + registry := activeRegistry(t, 3, 1_000) + + _, err := registry.PrimeKeyState(1, func(state *State) { + state.Status = StatusActive + state.HasBuilderIndex = true + state.BuilderIndex = 101 + state.Balance = 10 + state.EffectiveBalance = 10 + }) + require.NoError(t, err) + + got := registry.SelectForBid(0, SelectRequest{Strategy: StrategyRoundRobin, RequiredGwei: 500}) + require.Equal(t, []uint64{0, 2, 1}, selectedIndices(got), "the underfunded key sorts last") + + // With only the underfunded key left it is still offered. + got = registry.SelectForBid(0, SelectRequest{ + Strategy: StrategyRoundRobin, + RequiredGwei: 500, + Exclude: map[uint64]struct{}{0: {}, 2: {}}, + }) + require.Equal(t, []uint64{1}, selectedIndices(got)) +} + +func TestSelectForBidSkipsInactiveKeys(t *testing.T) { + registry := activeRegistry(t, 3, 1_000) + + for keyIndex, status := range map[uint64]Status{1: StatusExiting, 2: StatusPending} { + _, err := registry.PrimeKeyState(keyIndex, func(state *State) { + state.Status = status + }) + require.NoError(t, err) + } + + got := registry.SelectForBid(0, SelectRequest{Strategy: StrategyRoundRobin}) + require.Equal(t, []uint64{0}, selectedIndices(got)) + + // Nothing active at all yields no key rather than an unusable one. + _, err := registry.PrimeKeyState(0, func(state *State) { state.Status = StatusExited }) + require.NoError(t, err) + + require.Nil(t, registry.SelectForBid(0, SelectRequest{Strategy: StrategyRoundRobin})) +} + +func TestSelectForBidHonoursCount(t *testing.T) { + registry := activeRegistry(t, 4, 1_000) + + got := registry.SelectForBid(0, SelectRequest{Strategy: StrategyRoundRobin, Count: 2}) + require.Len(t, got, 2) +} diff --git a/pkg/builder_keys/state.go b/pkg/builder_keys/state.go new file mode 100644 index 00000000..63ab8bb9 --- /dev/null +++ b/pkg/builder_keys/state.go @@ -0,0 +1,112 @@ +package builder_keys + +import ( + "github.com/ethpandaops/go-eth2-client/spec/phase0" +) + +// Status is a builder key's lifecycle position, derived on every refresh from +// the beacon state, the pending-deposit queue and our own usage history. +type Status string + +const ( + // StatusUnused: derived but never deposited. Available for a first deposit. + StatusUnused Status = "unused" + // StatusDepositing: a deposit was submitted (our own transaction or an entry + // in the pending-deposit queue) but the key is not in the builder registry yet. + StatusDepositing Status = "depositing" + // StatusPending: registered, but the deposit epoch is not finalized, so the + // key may not bid yet. + StatusPending Status = "pending" + // StatusActive: registered, finalized and not exiting — usable for bidding. + StatusActive Status = "active" + // StatusExiting: an exit was initiated; the withdrawable epoch is set but not + // reached. The key can never be reactivated. + StatusExiting Status = "exiting" + // StatusExited: the withdrawable epoch has passed while the entry is still in + // the registry. + StatusExited Status = "exited" + // StatusWithdrawn: the key was used before and its pubkey has left the builder + // registry, so it is depositable again. + StatusWithdrawn Status = "withdrawn" +) + +// Managed reports whether the status counts toward the target key count: keys +// that are registered or on their way there. Exiting/exited keys do not count — +// they can never bid again, so the reconciler must replace them. +func (s Status) Managed() bool { + switch s { + case StatusDepositing, StatusPending, StatusActive: + return true + default: + return false + } +} + +// Depositable reports whether a fresh deposit may be submitted for the key. +// Exited entries are excluded: per the Gloas spec a deposit cannot reactivate +// them, it is only swept back to the wallet. +func (s Status) Depositable() bool { + return s == StatusUnused || s == StatusWithdrawn +} + +// State is an immutable snapshot of one builder key. It doubles as the API wire +// shape for the builder keys endpoint and SSE event. +type State struct { + KeyIndex uint64 `json:"key_index"` + Pubkey phase0.BLSPubKey `json:"-"` + // PubkeyHex is the 0x-prefixed public key, for JSON consumers. + PubkeyHex string `json:"pubkey"` + Status Status `json:"status"` + + // BuilderIndex is the on-chain registry index; only meaningful when + // HasBuilderIndex is set (index 0 is a valid builder index). + BuilderIndex uint64 `json:"builder_index"` + HasBuilderIndex bool `json:"has_builder_index"` + + Balance uint64 `json:"balance_gwei"` + PendingPayments uint64 `json:"pending_payments_gwei"` + BalanceAdjustment int64 `json:"balance_adjustment_gwei"` + // EffectiveBalance is the balance the key can actually bid against: + // chain balance plus local adjustments minus pending payments. + EffectiveBalance uint64 `json:"effective_balance_gwei"` + + DepositEpoch uint64 `json:"deposit_epoch"` + WithdrawableEpoch uint64 `json:"withdrawable_epoch"` + + // UseCount is how many deposit generations this key has gone through. + UseCount uint32 `json:"use_count"` + LastDepositAt int64 `json:"last_deposit_at,omitempty"` + LastExitAt int64 `json:"last_exit_at,omitempty"` + + // LastTopupEpoch guards the per-key top-up cooldown (0 = never topped up). + LastTopupEpoch phase0.Epoch `json:"last_topup_epoch,omitempty"` + + BidsSubmitted uint64 `json:"bids_submitted"` + BidsWon uint64 `json:"bids_won"` +} + +// Ready reports whether the key may be used for a bid of the given value: it +// must be active on chain and able to cover the payment. A builder whose +// effective balance is below the bid value has its bid rejected by the +// consensus layer, so bidding from it is pure noise. +func (s *State) Ready(requiredGwei uint64) bool { + return s.Status == StatusActive && s.EffectiveBalance >= requiredGwei +} + +// Aggregate summarises the whole key set for the dashboard. +type Aggregate struct { + Target uint64 `json:"target"` + Managed uint64 `json:"managed"` + + Unused uint64 `json:"unused"` + Depositing uint64 `json:"depositing"` + Pending uint64 `json:"pending"` + Active uint64 `json:"active"` + Exiting uint64 `json:"exiting"` + Exited uint64 `json:"exited"` + Withdrawn uint64 `json:"withdrawn"` + + TotalBalance uint64 `json:"total_balance_gwei"` + TotalPendingPayments uint64 `json:"total_pending_payments_gwei"` + TotalEffective uint64 `json:"total_effective_gwei"` +} diff --git a/pkg/builder_keys/usage.go b/pkg/builder_keys/usage.go new file mode 100644 index 00000000..b98a974a --- /dev/null +++ b/pkg/builder_keys/usage.go @@ -0,0 +1,69 @@ +package builder_keys + +import ( + "encoding/json" + "fmt" + "strconv" + + "github.com/ethpandaops/buildoor/pkg/db" +) + +// Namespace is the kv_store namespace holding per-key usage history. +const Namespace = "builder_keys" + +// Usage is the persisted history of one builder key. It survives restarts so a +// key that was deposited in an earlier run is recognised as ours even before the +// beacon state confirms it, and so a key that has cycled out of the registry can +// be reused instead of pushing the highest derivation index up forever. +type Usage struct { + KeyIndex uint64 `json:"key_index"` + // Pubkey is the derived public key at the time of writing. A mismatch on + // load means the entry key changed and the whole record set is about a + // different fleet. + Pubkey string `json:"pubkey"` + // UseCount is how many deposit generations this key has gone through. + UseCount uint32 `json:"use_count"` + FirstUsedAt int64 `json:"first_used_at,omitempty"` + LastDepositAt int64 `json:"last_deposit_at,omitempty"` + LastExitAt int64 `json:"last_exit_at,omitempty"` +} + +// UsageCodec translates key usage records to their persisted kv_store form: +// decimal key-index keys, JSON-encoded values. +type UsageCodec struct{} + +var _ db.KVCodec[uint64, *Usage] = UsageCodec{} + +// EncodeKey encodes an internal key index as its decimal string form. +func (UsageCodec) EncodeKey(keyIndex uint64) string { + return strconv.FormatUint(keyIndex, 10) +} + +// DecodeKey parses a decimal key index. +func (UsageCodec) DecodeKey(key string) (uint64, error) { + keyIndex, err := strconv.ParseUint(key, 10, 64) + if err != nil { + return 0, fmt.Errorf("invalid builder key usage key %q: %w", key, err) + } + + return keyIndex, nil +} + +// EncodeValue JSON-encodes a usage record. +func (UsageCodec) EncodeValue(usage *Usage) ([]byte, error) { + if usage == nil { + return nil, fmt.Errorf("cannot encode nil builder key usage") + } + + return json.Marshal(usage) +} + +// DecodeValue JSON-decodes a usage record. +func (UsageCodec) DecodeValue(value []byte) (*Usage, error) { + usage := &Usage{} + if err := json.Unmarshal(value, usage); err != nil { + return nil, fmt.Errorf("failed to decode builder key usage: %w", err) + } + + return usage, nil +} diff --git a/pkg/builderapi/epbs/beacon_block.go b/pkg/builderapi/epbs/beacon_block.go index c00cda93..9bd82599 100644 --- a/pkg/builderapi/epbs/beacon_block.go +++ b/pkg/builderapi/epbs/beacon_block.go @@ -142,6 +142,20 @@ func (h *Handler) HandleSubmitBeaconBlock(w http.ResponseWriter, r *http.Request return } + // The block's bid names the builder index the proposer committed to. That is + // the key that must sign the envelope: the block is the authority, not + // whichever key we would pick now. + revealKey := h.registry.ByBuilderIndex(uint64(bid.BuilderIndex)) + if revealKey == nil { + log.WithField("builder_index", bid.BuilderIndex). + Info("submitBeaconBlock: 400 — bid builder index does not belong to this builder") + h.recordSubmission(bid.Slot, submissionStatusFailed, + "bid builder index does not belong to this builder") + writeError(w, http.StatusBadRequest, "bid builder index does not belong to this builder") + + return + } + beaconBlockRoot, err := dynssz.GetGlobalDynSsz().HashTreeRoot(block.Message) if err != nil { log.WithError(err).Warn("submitBeaconBlock: failed to compute beacon block hash tree root") @@ -182,11 +196,14 @@ func (h *Handler) HandleSubmitBeaconBlock(w http.ResponseWriter, r *http.Request // and per-slot dedup with the p2p flow — nothing is published here. h.revealSvc.RequestReveal(&payload_bidder.RevealRequest{ Payload: event, + Key: revealKey, BlockInfo: &beacon.BlockInfo{ Slot: bid.Slot, Root: blockRoot, ParentRoot: block.Message.ParentRoot, ExecutionBlockHash: bid.BlockHash, + BuilderIndex: uint64(bid.BuilderIndex), + BuilderIndexKnown: true, }, Transport: payload_builder.BidTransportBuilderAPI, }) diff --git a/pkg/builderapi/epbs/handler.go b/pkg/builderapi/epbs/handler.go index 35cdd2a0..146445c6 100644 --- a/pkg/builderapi/epbs/handler.go +++ b/pkg/builderapi/epbs/handler.go @@ -17,12 +17,12 @@ import ( "github.com/sirupsen/logrus" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/memstore" "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/payload_builder" - "github.com/ethpandaops/buildoor/pkg/signer" ) // EventBroadcaster is the narrow WebUI event surface the post-Gloas dialect needs @@ -87,8 +87,9 @@ type Handler struct { log logrus.FieldLogger chainSvc chain.Service payloadCache *payload_builder.PayloadCache - bidderSigner *payload_bidder.Signer // shared Gloas bid signer (wraps blsSigner) - blsSigner *signer.BLSSigner // IsBuilderActive pubkey check + // registry is the managed builder key set: it decides which key signs each + // served bid and supplies the builder index that goes into it. + registry *builder_keys.Registry // planSvc is the mandatory per-slot scheduling/settings authority: bid // serving is decided exclusively by the slot's frozen plan. @@ -105,7 +106,13 @@ type Handler struct { lastBidMu sync.Mutex lastBids map[phase0.Slot]recordedBid // dedupe of repeated identical bid records - builderIndex atomic.Uint64 // builder index used in Gloas bids; set after lifecycle registration + // bidKeyMu guards bidKeys, which pins the builder key each (slot, parent + // tuple) is served from. There is no gossip first-seen rule here, but a + // polling proposer must keep seeing the same builder rather than a new + // index per request. + bidKeyMu sync.Mutex + bidKeys map[bidKeyTarget]uint64 + enabled atomic.Bool bidsRequested atomic.Uint64 // count of getExecutionPayloadBid requests received blocksAccepted atomic.Uint64 // count of accepted signed beacon blocks @@ -117,18 +124,71 @@ type Handler struct { // (via Freeze) on every getExecutionPayloadBid request. func NewHandler(cfg *config.BuilderAPIConfig, log logrus.FieldLogger, chainSvc chain.Service, planSvc *action_plan.PlanService, payloadCache *payload_builder.PayloadCache, - blsSigner *signer.BLSSigner) *Handler { + registry *builder_keys.Registry) *Handler { return &Handler{ cfg: cfg, log: log.WithField("component", "builderapi-epbs"), chainSvc: chainSvc, planSvc: planSvc, payloadCache: payloadCache, - bidderSigner: payload_bidder.NewSigner(blsSigner), - blsSigner: blsSigner, + registry: registry, prefsStore: NewBuilderPreferencesStore(), lastBids: make(map[phase0.Slot]recordedBid, maxRecordedBidSlots), + bidKeys: make(map[bidKeyTarget]uint64, maxRecordedBidSlots), + } +} + +// bidKeyTarget identifies what a served bid commits to, so repeated polls for +// the same slot and parent are answered from the same builder key. +type bidKeyTarget struct { + slot phase0.Slot + parentHash phase0.Hash32 + parentRoot phase0.Root +} + +// bidKey returns the builder key to serve a bid for the given target from, +// selecting one on first request and reusing it afterwards. It returns nil when +// no key is ready to cover the value. +func (h *Handler) bidKey(target bidKeyTarget, strategy string, requiredGwei uint64) *builder_keys.Key { + h.bidKeyMu.Lock() + defer h.bidKeyMu.Unlock() + + if keyIndex, ok := h.bidKeys[target]; ok { + if key, err := h.registry.Key(keyIndex); err == nil && key.State().Ready(requiredGwei) { + return key + } + + // The committed key can no longer cover the bid (payment settled, exit + // initiated); fall through and pick another. + delete(h.bidKeys, target) + } + + selected := h.registry.SelectForBid(target.slot, builder_keys.SelectRequest{ + Strategy: strategy, + RequiredGwei: requiredGwei, + Count: 1, + }) + + if len(selected) == 0 { + return nil } + + h.bidKeys[target] = selected[0].KeyIndex() + + // Bound the map: only the current and next slot are ever requested. + for len(h.bidKeys) > maxRecordedBidSlots { + oldest := target + + for candidate := range h.bidKeys { + if candidate.slot < oldest.slot { + oldest = candidate + } + } + + delete(h.bidKeys, oldest) + } + + return selected[0] } // SetResultRecorder wires the optional per-slot result recorder. @@ -245,12 +305,6 @@ func (h *Handler) SetEventBroadcaster(b EventBroadcaster) { h.events = b } -// SetBuilderIndex sets the on-chain builder index inserted into Gloas bids. -// Called from the lifecycle manager once registration is observed. -func (h *Handler) SetBuilderIndex(index uint64) { - h.builderIndex.Store(index) -} - // SetEnabled sets the enabled state of the post-Gloas Builder API dialect. // Only the non-slot-scoped endpoints (beacon block submission, builder // preferences) follow this flag; getExecutionPayloadBid bid serving is diff --git a/pkg/builderapi/epbs/handler_test.go b/pkg/builderapi/epbs/handler_test.go index f7e31109..6c0f36a8 100644 --- a/pkg/builderapi/epbs/handler_test.go +++ b/pkg/builderapi/epbs/handler_test.go @@ -37,7 +37,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/utils" ) @@ -182,14 +181,17 @@ type beaconBlockTestEnv struct { // newBeaconBlockTestEnv creates an enabled post-Gloas handler wired to a real // RevealService (with a stub envelope publisher). Slot 1 starts "now"; the // reveal is due revealTimeMs into the slot. +// testBuilderIndex is the on-chain builder index the test key set is primed +// with; blocks must name it for the reveal to bind to our key. +const testBuilderIndex = uint64(1) + func newBeaconBlockTestEnv(t *testing.T, slotDuration time.Duration, revealTimeMs int64) *beaconBlockTestEnv { t.Helper() log := logrus.New() log.SetLevel(logrus.PanicLevel) - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) cfg := &config.Config{APIPort: 8080, BuilderAPIEnabled: true} cfg.Reveal = config.DefaultConfig().Reveal @@ -211,12 +213,12 @@ func newBeaconBlockTestEnv(t *testing.T, slotDuration time.Duration, revealTimeM publisher := &stubEnvelopePublisher{} revealSvc := payload_bidder.NewRevealService( - cfg, payload_bidder.NewSigner(blsSigner), publisher, chainSvc, builderSvc, nil, planSvc, nil, log) + cfg, registry, publisher, chainSvc, builderSvc, nil, planSvc, nil, log) broadcaster := &stubBlockBroadcaster{} h := NewHandler(&cfg.BuilderAPI, log, chainSvc, planSvc, - payload_builder.NewPayloadCache(10), blsSigner) + payload_builder.NewPayloadCache(10), registry) h.SetBlockBroadcaster(broadcaster) h.SetRevealService(revealSvc) h.SetEnabled(true) @@ -251,7 +253,9 @@ func seedGloasPayload(h *Handler, slot phase0.Slot, blockHash phase0.Hash32) *pa // signedBeaconBlockJSON builds a fully populated Gloas SignedBeaconBlock whose // bid commits to blockHash, and returns its JSON encoding. -func signedBeaconBlockJSON(t *testing.T, slot phase0.Slot, blockHash phase0.Hash32) []byte { +func signedBeaconBlockJSON(t *testing.T, slot phase0.Slot, blockHash phase0.Hash32, + builderIndex uint64, +) []byte { t.Helper() block := &gloasspec.SignedBeaconBlock{ @@ -276,6 +280,7 @@ func signedBeaconBlockJSON(t *testing.T, slot phase0.Slot, blockHash phase0.Hash Message: &gloasspec.ExecutionPayloadBid{ BlockHash: blockHash, Slot: slot, + BuilderIndex: gloasspec.BuilderIndex(builderIndex), BlobKZGCommitments: []deneb.KZGCommitment{}, }, }, @@ -314,7 +319,7 @@ func TestHandleSubmitBeaconBlock_Success(t *testing.T) { blockHash := phase0.Hash32{0xab} payload := seedGloasPayload(env.handler, slot, blockHash) - rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash)) + rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex)) require.Equal(t, http.StatusAccepted, rec.Code, "submitBeaconBlock should return 202") assert.Equal(t, 1, env.broadcaster.callCount(), "beacon block must be broadcast exactly once") @@ -349,7 +354,7 @@ func TestHandleSubmitBeaconBlock_ProposalVersion(t *testing.T) { seedGloasPayload(env.handler, slot, blockHash) // Default: the chain's current fork (Gloas). - rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash)) + rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex)) require.Equal(t, http.StatusAccepted, rec.Code) proposal := env.broadcaster.lastProposal() @@ -360,7 +365,7 @@ func TestHandleSubmitBeaconBlock_ProposalVersion(t *testing.T) { // Heze via the Eth-Consensus-Version header (same wire schema as Gloas). req := httptest.NewRequest(http.MethodPost, "/eth/v1/builder/beacon_block", - bytes.NewReader(signedBeaconBlockJSON(t, slot, blockHash))) + bytes.NewReader(signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex))) req.Header.Set("Content-Type", "application/json") req.Header.Set("Eth-Consensus-Version", "heze") rec = httptest.NewRecorder() @@ -376,6 +381,28 @@ func TestHandleSubmitBeaconBlock_ProposalVersion(t *testing.T) { assert.Nil(t, proposal.Gloas) } +// TestHandleSubmitBeaconBlock_ForeignBuilderIndex rejects a block whose bid +// names a builder we do not control: signing the envelope with a key of ours +// would produce a validly-signed reveal for somebody else's bid, which the +// beacon chain rejects while we still owe nothing and the proposer loses the +// slot silently. +func TestHandleSubmitBeaconBlock_ForeignBuilderIndex(t *testing.T) { + env := newBeaconBlockTestEnv(t, 4*time.Second, 3500) + + require.NoError(t, env.revealSvc.Start(context.Background())) + defer env.revealSvc.Stop() + + slot := phase0.Slot(1) + blockHash := phase0.Hash32{0xab} + seedGloasPayload(env.handler, slot, blockHash) + + rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex+99)) + + assert.Equal(t, http.StatusBadRequest, rec.Code) + assert.Equal(t, 0, env.broadcaster.callCount(), "block must not be broadcast") + assert.Equal(t, uint64(0), env.handler.BlocksAccepted()) +} + // TestHandleSubmitBeaconBlock_NoCachedPayload returns 400 and neither // broadcasts the block nor requests a reveal. func TestHandleSubmitBeaconBlock_NoCachedPayload(t *testing.T) { @@ -385,7 +412,7 @@ func TestHandleSubmitBeaconBlock_NoCachedPayload(t *testing.T) { defer env.revealSvc.Stop() // No payload seeded. - rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, 1, phase0.Hash32{0xab})) + rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, 1, phase0.Hash32{0xab}, testBuilderIndex)) assert.Equal(t, http.StatusBadRequest, rec.Code, "missing cached payload should return 400") assert.Equal(t, 0, env.broadcaster.callCount(), "block must not be broadcast") @@ -409,7 +436,7 @@ func TestHandleSubmitBeaconBlock_BroadcastFailure(t *testing.T) { blockHash := phase0.Hash32{0xab} payload := seedGloasPayload(env.handler, slot, blockHash) - rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash)) + rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex)) assert.Equal(t, http.StatusInternalServerError, rec.Code, "broadcast failure should return 500") assert.Equal(t, 1, env.broadcaster.callCount()) @@ -434,7 +461,7 @@ func TestHandleSubmitBeaconBlock_SSZBody(t *testing.T) { seedGloasPayload(env.handler, slot, blockHash) block := eth2all.SignedBeaconBlock{Version: version.DataVersionGloas} - require.NoError(t, json.Unmarshal(signedBeaconBlockJSON(t, slot, blockHash), &block)) + require.NoError(t, json.Unmarshal(signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex), &block)) body, err := block.MarshalSSZ() require.NoError(t, err) diff --git a/pkg/builderapi/epbs/keyset_test.go b/pkg/builderapi/epbs/keyset_test.go new file mode 100644 index 00000000..45fe775f --- /dev/null +++ b/pkg/builderapi/epbs/keyset_test.go @@ -0,0 +1,45 @@ +package epbs + +import ( + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" +) + +// testEntryPrivkey roots the key set used by the tests in this package. +const testEntryPrivkey = "0000000000000000000000000000000000000000000000000000000000000001" + +// newTestKeyRegistry builds a key registry whose keys are primed as active +// builders at the given on-chain builder indices (key i -> builderIndices[i]). +func newTestKeyRegistry(t *testing.T, builderIndices ...uint64) *builder_keys.Registry { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + cfg := &config.Config{BuilderKeys: config.BuilderKeysConfig{ + TargetCount: uint64(len(builderIndices)), + DiscoveryGap: 1, + MaxIndex: 32, + }} + + registry, err := builder_keys.NewRegistry(cfg, testEntryPrivkey, log) + require.NoError(t, err) + + for keyIndex, builderIndex := range builderIndices { + _, err := registry.PrimeKeyState(uint64(keyIndex), func(state *builder_keys.State) { + state.Status = builder_keys.StatusActive + state.BuilderIndex = builderIndex + state.HasBuilderIndex = true + state.Balance = 1_000_000_000_000 + state.EffectiveBalance = 1_000_000_000_000 + }) + require.NoError(t, err) + } + + return registry +} diff --git a/pkg/builderapi/epbs/payload_bid.go b/pkg/builderapi/epbs/payload_bid.go index 328dfc1d..0cc9367d 100644 --- a/pkg/builderapi/epbs/payload_bid.go +++ b/pkg/builderapi/epbs/payload_bid.go @@ -18,7 +18,6 @@ import ( "github.com/gorilla/mux" "github.com/sirupsen/logrus" - "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/payload_builder" @@ -51,14 +50,14 @@ type GetExecutionPayloadBidResponse struct { func (h *Handler) HandleGetExecutionPayloadBid(w http.ResponseWriter, r *http.Request) { log := h.log.WithField("path", "/eth/v1/builder/execution_payload_bid/...") - if h.payloadCache == nil || h.blsSigner == nil { - log.Warn("getExecutionPayloadBid: returning 204 — signer or payload cache unavailable") + if h.payloadCache == nil || h.registry == nil { + log.Warn("getExecutionPayloadBid: returning 204 — key registry or payload cache unavailable") w.WriteHeader(http.StatusNoContent) return } - if !chain.IsBuilderActive(h.chainSvc.GetBuilderByPubkey(h.blsSigner.PublicKey()), uint64(h.chainSvc.GetFinalizedEpoch())) { - log.Warn("getExecutionPayloadBid: returning 204 — builder not active on chain") + if !h.registry.AnyActive() { + log.Warn("getExecutionPayloadBid: returning 204 — no builder key active on chain") w.WriteHeader(http.StatusNoContent) return } @@ -302,13 +301,44 @@ func (h *Handler) HandleGetExecutionPayloadBid(w http.ResponseWriter, r *http.Re tctx, cancel := context.WithTimeout(r.Context(), transformTimeout) defer cancel() + bidKey := h.bidKey(bidKeyTarget{ + slot: slot, + parentHash: event.Attributes.ParentBlockHash, + parentRoot: event.Attributes.ParentBlockRoot, + }, frozenSettings.KeyStrategy, uint64(valueAfterSubsidy)) + if bidKey == nil { + log.WithField("value_gwei", uint64(valueAfterSubsidy)). + Warn("getExecutionPayloadBid: returning 204 — no builder key ready to cover the bid") + h.recordBid(slot, fork.String(), "", nil, uint64(valueAfterSubsidy), uint64(executionPayment), + bidStatusFailed, "no builder key ready to cover the bid") + w.WriteHeader(http.StatusNoContent) + + return + } + + // Key selection only ever offers active keys, which by definition carry an + // on-chain index — but signing with the zero value here would be silent: + // the bid is well-formed and served, its signature just belongs to whoever + // holds builder index 0. Refuse instead of serving an unverifiable bid. + builderIndex, registered := bidKey.BuilderIndex() + if !registered { + log.WithField("key", bidKey.String()). + Warn("getExecutionPayloadBid: returning 204 — selected builder key has no on-chain index") + h.recordBid(slot, fork.String(), "", nil, uint64(valueAfterSubsidy), uint64(executionPayment), + bidStatusFailed, "selected builder key has no on-chain index") + w.WriteHeader(http.StatusNoContent) + + return + } + signedBid, err := payload_bidder.BuildSignedBid(tctx, event, payload_bidder.BidParams{ - BuilderIndex: h.builderIndex.Load(), + BuilderIndex: builderIndex, FeeRecipient: prefs.FeeRecipient, Value: value, ExecutionPayment: executionPayment, Transform: bidTransform, - }, h.bidderSigner, bidForkVersion, h.chainSvc.GetGenesis().GenesisValidatorsRoot) + }, payload_bidder.NewSigner(bidKey.BLSSigner()), bidForkVersion, + h.chainSvc.GetGenesis().GenesisValidatorsRoot) if err != nil { log.WithError(err).Warn("getExecutionPayloadBid: failed to build signed bid") h.recordBid(slot, fork.String(), "", nil, uint64(valueAfterSubsidy), uint64(executionPayment), diff --git a/pkg/builderapi/epbs/payload_bid_test.go b/pkg/builderapi/epbs/payload_bid_test.go index 8ef1701c..2cb42be4 100644 --- a/pkg/builderapi/epbs/payload_bid_test.go +++ b/pkg/builderapi/epbs/payload_bid_test.go @@ -26,7 +26,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/memstore" "github.com/ethpandaops/buildoor/pkg/payload_builder" - "github.com/ethpandaops/buildoor/pkg/signer" ) // recordedBidCall captures one RecordBuilderAPIBid invocation. @@ -129,8 +128,7 @@ func newPayloadBidTestEnv(t *testing.T, enabled bool) *payloadBidTestEnv { log := logrus.New() log.SetLevel(logrus.PanicLevel) - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) cfg := &config.Config{ APIPort: 8080, @@ -157,7 +155,7 @@ func newPayloadBidTestEnv(t *testing.T, enabled bool) *payloadBidTestEnv { planSvc := action_plan.NewPlanService(cfg, chainSvc, log) h := NewHandler(&cfg.BuilderAPI, log, chainSvc, planSvc, - payload_builder.NewPayloadCache(10), blsSigner) + payload_builder.NewPayloadCache(10), registry) recorder := &stubSlotResultRecorder{} h.SetResultRecorder(recorder) @@ -477,7 +475,7 @@ func TestHandleSubmitBeaconBlock_RecordsSubmissions(t *testing.T) { blockHash := phase0.Hash32{0xab} seedGloasPayload(env.handler, slot, blockHash) - rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash)) + rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex)) require.Equal(t, http.StatusAccepted, rec.Code) calls := recorder.submissionCalls() @@ -502,7 +500,7 @@ func TestHandleSubmitBeaconBlock_RecordsSubmissions(t *testing.T) { blockHash := phase0.Hash32{0xab} seedGloasPayload(env.handler, slot, blockHash) - rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash)) + rec := postBeaconBlock(env.handler, signedBeaconBlockJSON(t, slot, blockHash, testBuilderIndex)) require.Equal(t, http.StatusInternalServerError, rec.Code) calls := recorder.submissionCalls() diff --git a/pkg/builderapi/keyset_test.go b/pkg/builderapi/keyset_test.go new file mode 100644 index 00000000..f3929a81 --- /dev/null +++ b/pkg/builderapi/keyset_test.go @@ -0,0 +1,45 @@ +package builderapi + +import ( + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" +) + +// testEntryPrivkey roots the key set used by the tests in this package. +const testEntryPrivkey = "0000000000000000000000000000000000000000000000000000000000000001" + +// newTestKeyRegistry builds a key registry whose keys are primed as active +// builders at the given on-chain builder indices (key i -> builderIndices[i]). +func newTestKeyRegistry(t *testing.T, builderIndices ...uint64) *builder_keys.Registry { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + cfg := &config.Config{BuilderKeys: config.BuilderKeysConfig{ + TargetCount: uint64(len(builderIndices)), + DiscoveryGap: 1, + MaxIndex: 32, + }} + + registry, err := builder_keys.NewRegistry(cfg, testEntryPrivkey, log) + require.NoError(t, err) + + for keyIndex, builderIndex := range builderIndices { + _, err := registry.PrimeKeyState(uint64(keyIndex), func(state *builder_keys.State) { + state.Status = builder_keys.StatusActive + state.BuilderIndex = builderIndex + state.HasBuilderIndex = true + state.Balance = 1_000_000_000_000 + state.EffectiveBalance = 1_000_000_000_000 + }) + require.NoError(t, err) + } + + return registry +} diff --git a/pkg/builderapi/legacy/get_header.go b/pkg/builderapi/legacy/get_header.go index d80d48aa..cae5e73c 100644 --- a/pkg/builderapi/legacy/get_header.go +++ b/pkg/builderapi/legacy/get_header.go @@ -36,7 +36,7 @@ type GetHeaderResponse struct { func (h *Handler) HandleGetHeader(w http.ResponseWriter, r *http.Request) { log := h.log.WithField("path", "/eth/v1/builder/header/...") - if h.payloadCache == nil || h.blsSigner == nil { + if h.payloadCache == nil || h.registry == nil { log.Warn("getHeader: returning 204 — payload cache or BLS signer not available") w.WriteHeader(http.StatusNoContent) return @@ -164,7 +164,12 @@ func (h *Handler) HandleGetHeader(w http.ResponseWriter, r *http.Request) { if chainSpec := h.chainSvc.GetChainSpec(); chainSpec != nil { maxWithdrawalsPerPayload = chainSpec.MaxWithdrawalsPerPayload } - signedBid, err := BuildSignedBuilderBid(event, fork, h.blsSigner.PublicKey(), h.blsSigner, + // Pre-Gloas bids are signed by the primary key: the proposer's relay verifies + // the signature against the pubkey it saw in this very response, so rotating + // keys per request would only churn what the relay sees. + bidKey := h.registry.Primary() + + signedBid, err := BuildSignedBuilderBid(event, fork, bidKey.Pubkey(), bidKey.BLSSigner(), subsidyGwei, totalValueGwei, h.chainSvc.GetGenesis().GenesisForkVersion, maxWithdrawalsPerPayload) if err != nil { log.WithError(err).Warn("getHeader: failed to build SignedBuilderBid") diff --git a/pkg/builderapi/legacy/get_header_test.go b/pkg/builderapi/legacy/get_header_test.go index af0a53b7..e37ae39e 100644 --- a/pkg/builderapi/legacy/get_header_test.go +++ b/pkg/builderapi/legacy/get_header_test.go @@ -27,7 +27,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/memstore" "github.com/ethpandaops/buildoor/pkg/payload_builder" - "github.com/ethpandaops/buildoor/pkg/signer" ) // recordedBidCall captures one RecordBuilderAPIBid invocation. @@ -122,8 +121,8 @@ func newGetHeaderTestEnv(t *testing.T, enabled bool, blockValueWei *big.Int) *ge log := logrus.New() log.SetLevel(logrus.PanicLevel) - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) + blsSigner := registry.Primary().BLSSigner() cfg := &config.Config{ APIPort: 8080, @@ -142,7 +141,7 @@ func newGetHeaderTestEnv(t *testing.T, enabled bool, blockValueWei *big.Int) *ge store := memstore.New[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration]() h := NewHandler(&cfg.BuilderAPI, log, chainSvc, planSvc, payload_builder.NewPayloadCache(10), - store, blsSigner) + store, registry) recorder := &stubSlotResultRecorder{} h.SetResultRecorder(recorder) diff --git a/pkg/builderapi/legacy/handler.go b/pkg/builderapi/legacy/handler.go index 9e12ac87..c9afaf53 100644 --- a/pkg/builderapi/legacy/handler.go +++ b/pkg/builderapi/legacy/handler.go @@ -14,11 +14,11 @@ import ( "github.com/sirupsen/logrus" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/memstore" "github.com/ethpandaops/buildoor/pkg/payload_builder" - "github.com/ethpandaops/buildoor/pkg/signer" ) // EventBroadcaster is the narrow WebUI event surface the legacy dialect needs @@ -81,7 +81,7 @@ type Handler struct { chainSvc chain.Service payloadCache *payload_builder.PayloadCache validatorsStore *memstore.Store[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration] - blsSigner *signer.BLSSigner + registry *builder_keys.Registry // planSvc is the mandatory per-slot scheduling/settings authority: bid // serving is decided exclusively by the slot's frozen plan. @@ -106,7 +106,7 @@ type Handler struct { func NewHandler(cfg *config.BuilderAPIConfig, log logrus.FieldLogger, chainSvc chain.Service, planSvc *action_plan.PlanService, payloadCache *payload_builder.PayloadCache, validatorsStore *memstore.Store[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration], - blsSigner *signer.BLSSigner) *Handler { + registry *builder_keys.Registry) *Handler { return &Handler{ cfg: cfg, log: log.WithField("component", "builderapi-legacy"), @@ -114,7 +114,7 @@ func NewHandler(cfg *config.BuilderAPIConfig, log logrus.FieldLogger, chainSvc c planSvc: planSvc, payloadCache: payloadCache, validatorsStore: validatorsStore, - blsSigner: blsSigner, + registry: registry, lastBids: make(map[phase0.Slot]recordedBid, maxRecordedBidSlots), } } diff --git a/pkg/builderapi/legacy/handler_test.go b/pkg/builderapi/legacy/handler_test.go index a09da898..62ac9227 100644 --- a/pkg/builderapi/legacy/handler_test.go +++ b/pkg/builderapi/legacy/handler_test.go @@ -25,13 +25,13 @@ import ( "github.com/stretchr/testify/require" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" legacytypes "github.com/ethpandaops/buildoor/pkg/builderapi/legacy/types" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/memstore" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/utils" ) @@ -107,19 +107,19 @@ func newServingPlanService(chainSvc chain.Service) *action_plan.PlanService { &config.Config{APIPort: 8080, BuilderAPIEnabled: true}, chainSvc, logrus.New()) } -func newTestHandler(chainSvc chain.Service, blsSigner *signer.BLSSigner) *Handler { +func newTestHandler(chainSvc chain.Service, registry *builder_keys.Registry) *Handler { return NewHandler(&config.BuilderAPIConfig{}, logrus.New(), chainSvc, newServingPlanService(chainSvc), payload_builder.NewPayloadCache(10), - memstore.New[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration](), blsSigner) + memstore.New[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration](), registry) } // TestHandleGetHeader_PostGloasForkGuard returns 204 once the chain has // activated Gloas, even when the handler is enabled and fully configured. func TestHandleGetHeader_PostGloasForkGuard(t *testing.T) { - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) + blsSigner := registry.Primary().BLSSigner() - h := newTestHandler(&stubChainService{currentFork: version.DataVersionGloas}, blsSigner) + h := newTestHandler(&stubChainService{currentFork: version.DataVersionGloas}, registry) h.SetEnabled(true) pk := blsSigner.PublicKey() @@ -142,13 +142,13 @@ func TestHandleGetHeader_PostGloasForkGuard(t *testing.T) { // when the Accept header prefers application/octet-stream, with identical // bid contents in both representations. func TestHandleGetHeader_Success(t *testing.T) { - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) + blsSigner := registry.Primary().BLSSigner() store := memstore.New[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration]() chainSvc := &stubChainService{currentFork: version.DataVersionFulu} h := NewHandler(&config.BuilderAPIConfig{}, logrus.New(), chainSvc, - newServingPlanService(chainSvc), payload_builder.NewPayloadCache(10), store, blsSigner) + newServingPlanService(chainSvc), payload_builder.NewPayloadCache(10), store, registry) pk := blsSigner.PublicKey() store.Put(pk, &apiv1.SignedValidatorRegistration{}) diff --git a/pkg/builderapi/legacy/keyset_test.go b/pkg/builderapi/legacy/keyset_test.go new file mode 100644 index 00000000..e1dac5b5 --- /dev/null +++ b/pkg/builderapi/legacy/keyset_test.go @@ -0,0 +1,45 @@ +package legacy + +import ( + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" +) + +// testEntryPrivkey roots the key set used by the tests in this package. +const testEntryPrivkey = "0000000000000000000000000000000000000000000000000000000000000001" + +// newTestKeyRegistry builds a key registry whose keys are primed as active +// builders at the given on-chain builder indices (key i -> builderIndices[i]). +func newTestKeyRegistry(t *testing.T, builderIndices ...uint64) *builder_keys.Registry { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + cfg := &config.Config{BuilderKeys: config.BuilderKeysConfig{ + TargetCount: uint64(len(builderIndices)), + DiscoveryGap: 1, + MaxIndex: 32, + }} + + registry, err := builder_keys.NewRegistry(cfg, testEntryPrivkey, log) + require.NoError(t, err) + + for keyIndex, builderIndex := range builderIndices { + _, err := registry.PrimeKeyState(uint64(keyIndex), func(state *builder_keys.State) { + state.Status = builder_keys.StatusActive + state.BuilderIndex = builderIndex + state.HasBuilderIndex = true + state.Balance = 1_000_000_000_000 + state.EffectiveBalance = 1_000_000_000_000 + }) + require.NoError(t, err) + } + + return registry +} diff --git a/pkg/builderapi/legacy/registrations_test.go b/pkg/builderapi/legacy/registrations_test.go index d1b79292..6557d9fd 100644 --- a/pkg/builderapi/legacy/registrations_test.go +++ b/pkg/builderapi/legacy/registrations_test.go @@ -150,8 +150,8 @@ func (f *fakePersistence) get(pubkey phase0.BLSPubKey) *apiv1.SignedValidatorReg // overwrites the stored entry (replace policy) and that the latest value // reaches the attached persistence on flush. func TestHandleRegisterValidators_OverwriteAndFlush(t *testing.T) { - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) + blsSigner := registry.Primary().BLSSigner() store := memstore.New[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration]() persistence := &fakePersistence{} @@ -160,7 +160,7 @@ func TestHandleRegisterValidators_OverwriteAndFlush(t *testing.T) { defer store.Stop() h := NewHandler(&config.BuilderAPIConfig{}, logrus.New(), &stubChainService{}, - newServingPlanService(&stubChainService{}), payload_builder.NewPayloadCache(10), store, blsSigner) + newServingPlanService(&stubChainService{}), payload_builder.NewPayloadCache(10), store, registry) h.SetEnabled(true) register := func(gasLimit uint64) { diff --git a/pkg/builderapi/server.go b/pkg/builderapi/server.go index aae5c0f1..c106ab4e 100644 --- a/pkg/builderapi/server.go +++ b/pkg/builderapi/server.go @@ -25,6 +25,7 @@ import ( "github.com/sirupsen/logrus" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" epbsapi "github.com/ethpandaops/buildoor/pkg/builderapi/epbs" "github.com/ethpandaops/buildoor/pkg/builderapi/legacy" "github.com/ethpandaops/buildoor/pkg/chain" @@ -33,7 +34,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" ) // EventBroadcaster provides methods for broadcasting Builder API events to the @@ -98,13 +98,14 @@ type Server struct { // planSvc is the mandatory per-slot scheduling/settings authority: the bid // handlers freeze the slot's plan at request time and serve per the frozen // effective settings. payloadCache may be nil; endpoints needing it degrade -// gracefully. blsSigner may be nil; if set, getHeader signs builder bids. +// gracefully. registry is the managed builder key set; getHeader signs +// pre-Gloas builder bids with its primary key. // validatorStore is optional (an in-memory store is created when nil); when // provided it is the shared instance also read by the legacy registration // settings resolver. func NewServer(cfg *config.BuilderAPIConfig, log *logrus.Logger, chainSvc chain.Service, planSvc *action_plan.PlanService, payloadCache *payload_builder.PayloadCache, - blsSigner *signer.BLSSigner, + registry *builder_keys.Registry, validatorStore *memstore.Store[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration]) *Server { store := validatorStore if store == nil { @@ -117,8 +118,8 @@ func NewServer(cfg *config.BuilderAPIConfig, log *logrus.Logger, chainSvc chain. chainSvc: chainSvc, payloadCache: payloadCache, validatorsStore: store, - legacy: legacy.NewHandler(cfg, log, chainSvc, planSvc, payloadCache, store, blsSigner), - epbs: epbsapi.NewHandler(cfg, log, chainSvc, planSvc, payloadCache, blsSigner), + legacy: legacy.NewHandler(cfg, log, chainSvc, planSvc, payloadCache, store, registry), + epbs: epbsapi.NewHandler(cfg, log, chainSvc, planSvc, payloadCache, registry), } } @@ -177,12 +178,6 @@ func (s *Server) SetResultRecorder(rec SlotResultRecorder) { s.epbs.SetResultRecorder(rec) } -// SetBuilderIndex sets the on-chain builder index inserted into Gloas bids. -// Called from the lifecycle manager once registration is observed. -func (s *Server) SetBuilderIndex(index uint64) { - s.epbs.SetBuilderIndex(index) -} - // GetBuilderPreferencesStore returns the store of latest per-validator builder // preferences submitted via the submitBuilderPreferences API. func (s *Server) GetBuilderPreferencesStore() *epbsapi.BuilderPreferencesStore { diff --git a/pkg/builderapi/server_test.go b/pkg/builderapi/server_test.go index 08547af1..857b7704 100644 --- a/pkg/builderapi/server_test.go +++ b/pkg/builderapi/server_test.go @@ -157,9 +157,9 @@ func TestRegisterValidators_MissingContentType(t *testing.T) { func TestGetHeader_NoPayload(t *testing.T) { cfg := &config.BuilderAPIConfig{} log := logrus.New() - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) - srv := NewServer(cfg, log, &mockChainService{}, newServingPlanService(), nil, blsSigner, nil) + registry := newTestKeyRegistry(t, 1) + blsSigner := registry.Primary().BLSSigner() + srv := NewServer(cfg, log, &mockChainService{}, newServingPlanService(), nil, registry, nil) pk := blsSigner.PublicKey() url := "/eth/v1/builder/header/1/0x0000000000000000000000000000000000000000000000000000000000000000/0x" + hex.EncodeToString(pk[:]) @@ -174,11 +174,12 @@ func TestGetHeader_NoPayload(t *testing.T) { func TestGetHeader_InvalidSlot(t *testing.T) { cfg := &config.BuilderAPIConfig{} log := logrus.New() - blsSigner, _ := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - srv := NewServer(cfg, log, &mockChainService{}, newServingPlanService(), payload_builder.NewPayloadCache(10), blsSigner, nil) + registry := newTestKeyRegistry(t, 1) + srv := NewServer(cfg, log, &mockChainService{}, newServingPlanService(), + payload_builder.NewPayloadCache(10), registry, nil) srv.SetEnabled(true) - pk := blsSigner.PublicKey() + pk := registry.Primary().Pubkey() url := "/eth/v1/builder/header/not_a_number/0x0000000000000000000000000000000000000000000000000000000000000000/0x" + hex.EncodeToString(pk[:]) req := httptest.NewRequest(http.MethodGet, url, nil) rec := httptest.NewRecorder() @@ -189,8 +190,8 @@ func TestGetHeader_InvalidSlot(t *testing.T) { // TestGetHeader_SubsidyInBidValue returns 200 with bid value = block_value + BlockValueSubsidyGwei. func TestGetHeader_SubsidyInBidValue(t *testing.T) { - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) + blsSigner := registry.Primary().BLSSigner() pk := blsSigner.PublicKey() // Create valid registration so getHeader does not return 204 for unregistered proposer @@ -241,7 +242,7 @@ func TestGetHeader_SubsidyInBidValue(t *testing.T) { } cache.Store(event) planSvc := action_plan.NewPlanService(fullCfg, &mockChainService{}, log) - srv := NewServer(cfg, log, &mockChainService{currentFork: version.DataVersionDeneb}, planSvc, cache, blsSigner, nil) + srv := NewServer(cfg, log, &mockChainService{currentFork: version.DataVersionDeneb}, planSvc, cache, registry, nil) srv.SetEnabled(true) req := httptest.NewRequest(http.MethodPost, "/eth/v1/builder/validators", bytes.NewReader(regs)) diff --git a/pkg/config/default.go b/pkg/config/default.go index 8a8ff281..f1e2ba2c 100644 --- a/pkg/config/default.go +++ b/pkg/config/default.go @@ -14,6 +14,13 @@ func DefaultConfig() *Config { BlockValueSubsidyGwei: 100000, // 100k Gwei ServeCandidates: "all", }, + BuilderKeys: BuilderKeysConfig{ + TargetCount: 1, // single key by default: index 0 is the entry key + MaxIndex: 1000, + DiscoveryGap: 100, + AutoDeposit: true, + AutoExit: true, + }, DepositAmount: 50000000000, // 50 ETH in Gwei TopupThreshold: 10000000000, // 10 ETH in Gwei TopupAmount: 50000000000, // 50 ETH in Gwei @@ -35,6 +42,8 @@ func DefaultConfig() *Config { BidSubsidy: 100000000, // 100M gwei = 0.1 ETH; clears validator local-EL threshold HeadVoteThresholdPct: 60, // Gloas builder payment quorum (6/10) BidCandidate: "auto", + KeyStrategy: "round_robin", + BidKeysPerStep: 1, }, Build: BuildConfig{ CandidateParentFull: CandidateModeAlways, diff --git a/pkg/config/settings_fields.go b/pkg/config/settings_fields.go index b2a27725..21d16f36 100644 --- a/pkg/config/settings_fields.go +++ b/pkg/config/settings_fields.go @@ -95,6 +95,9 @@ func Fields() []Field { newField(KeyEPBSHeadVoteThreshold, "epbs-vote-threshold", func(c *Config) *uint64 { return &c.EPBS.HeadVoteThresholdPct }), newField(KeyEPBSBidCandidate, "epbs-bid-candidate", func(c *Config) *string { return &c.EPBS.BidCandidate }), newField(KeyEPBSBidCandidateSwitch, "epbs-bid-candidate-switch", func(c *Config) *bool { return &c.EPBS.BidCandidateSwitch }), + newField(KeyEPBSKeyStrategy, "epbs-key-strategy", func(c *Config) *string { return &c.EPBS.KeyStrategy }), + newField(KeyEPBSBidKeysPerSlot, "epbs-bid-keys-per-slot", func(c *Config) *uint64 { return &c.EPBS.BidKeysPerSlot }), + newField(KeyEPBSBidKeysPerStep, "epbs-bid-keys-per-step", func(c *Config) *uint64 { return &c.EPBS.BidKeysPerStep }), newField(KeyRevealEnabled, "reveal-enabled", func(c *Config) *bool { return &c.Reveal.Enabled }), newField(KeyRevealGateMode, "reveal-gate-mode", func(c *Config) *string { return &c.Reveal.GateMode }), @@ -120,11 +123,17 @@ func Fields() []Field { newField(KeyBuilderAPIValueOverride, "builder-api-value-override", func(c *Config) *uint64 { return &c.BuilderAPI.ValueOverrideGwei }), newField(KeyBuilderAPIServeCandidates, "builder-api-serve-candidates", func(c *Config) *string { return &c.BuilderAPI.ServeCandidates }), newField(KeyBuilderAPIOnDemandBuild, "builder-api-on-demand-build", func(c *Config) *bool { return &c.BuilderAPI.OnDemandBuild }), + newField(KeyBuilderAPIKeyStrategy, "builder-api-key-strategy", func(c *Config) *string { return &c.BuilderAPI.KeyStrategy }), newField(KeySlotResultRetentionEpochs, "slot-result-retention-epochs", func(c *Config) *uint64 { return &c.SlotResultRetentionEpochs }), newField(KeySlotArtifactRetentionEpochs, "slot-artifact-retention-epochs", func(c *Config) *uint64 { return &c.SlotArtifactRetentionEpochs }), newField(KeySlotArtifactCaptureEnabled, "slot-artifact-capture-enabled", func(c *Config) *bool { return &c.SlotArtifactCaptureEnabled }), + newField(KeyBuilderKeysTargetCount, "builder-keys-target", func(c *Config) *uint64 { return &c.BuilderKeys.TargetCount }), + newField(KeyBuilderKeysMaxIndex, "builder-keys-max-index", func(c *Config) *uint64 { return &c.BuilderKeys.MaxIndex }), + newField(KeyBuilderKeysAutoDeposit, "builder-keys-auto-deposit", func(c *Config) *bool { return &c.BuilderKeys.AutoDeposit }), + newField(KeyBuilderKeysAutoExit, "builder-keys-auto-exit", func(c *Config) *bool { return &c.BuilderKeys.AutoExit }), + newField(KeyDepositAmount, "deposit-amount", func(c *Config) *uint64 { return &c.DepositAmount }), newField(KeyTopupThreshold, "topup-threshold", func(c *Config) *uint64 { return &c.TopupThreshold }), newField(KeyTopupAmount, "topup-amount", func(c *Config) *uint64 { return &c.TopupAmount }), diff --git a/pkg/config/settings_keys.go b/pkg/config/settings_keys.go index 37eead7e..fc1c18a6 100644 --- a/pkg/config/settings_keys.go +++ b/pkg/config/settings_keys.go @@ -20,6 +20,9 @@ const ( KeyEPBSHeadVoteThreshold = "epbs.head_vote_threshold_pct" KeyEPBSBidCandidate = "epbs.bid_candidate" KeyEPBSBidCandidateSwitch = "epbs.bid_candidate_switch" + KeyEPBSKeyStrategy = "epbs.key_strategy" + KeyEPBSBidKeysPerSlot = "epbs.bid_keys_per_slot" + KeyEPBSBidKeysPerStep = "epbs.bid_keys_per_step" KeyRevealEnabled = "reveal.enabled" KeyRevealGateMode = "reveal.gate_mode" @@ -45,6 +48,7 @@ const ( KeyBuilderAPIValueOverride = "builder_api.value_override_gwei" KeyBuilderAPIServeCandidates = "builder_api.serve_candidates" KeyBuilderAPIOnDemandBuild = "builder_api.on_demand_build" + KeyBuilderAPIKeyStrategy = "builder_api.key_strategy" KeySlotResultRetentionEpochs = "slot_result_retention_epochs" KeySlotArtifactRetentionEpochs = "slot_artifact_retention_epochs" @@ -54,6 +58,11 @@ const ( KeyTopupThreshold = "topup_threshold" KeyTopupAmount = "topup_amount" + KeyBuilderKeysTargetCount = "builder_keys.target_count" + KeyBuilderKeysMaxIndex = "builder_keys.max_index" + KeyBuilderKeysAutoDeposit = "builder_keys.auto_deposit" + KeyBuilderKeysAutoExit = "builder_keys.auto_exit" + KeyEPBSEnabled = "epbs_enabled" KeyBuilderAPIEnabled = "builder_api_enabled" KeyLifecycleEnabled = "lifecycle_enabled" diff --git a/pkg/config/settings_registry_test.go b/pkg/config/settings_registry_test.go new file mode 100644 index 00000000..7d87a40b --- /dev/null +++ b/pkg/config/settings_registry_test.go @@ -0,0 +1,80 @@ +package config + +import ( + "go/ast" + "go/parser" + "go/token" + "strconv" + "testing" + + "github.com/stretchr/testify/require" +) + +// Every canonical settings key must be registered in the field registry. +// Declaring the constant without wiring the field compiles fine and looks +// complete, but the setting is then simply missing: the API rejects it as +// unknown and neither the UI nor the state-db can ever touch it. +func TestEverySettingsKeyIsRegistered(t *testing.T) { + registered := make(map[string]struct{}, 64) + for _, field := range Fields() { + registered[field.Key] = struct{}{} + } + + for name, key := range declaredSettingsKeys(t) { + _, ok := registered[key] + require.True(t, ok, "settings key %s (%q) is declared but not registered in Fields()", name, key) + } +} + +// Flag keys must be unique too: two fields sharing one viper flag would make +// CLI-change detection resolve them against each other's last-seen value. +func TestSettingsFlagKeysAreUnique(t *testing.T) { + seen := make(map[string]string, 64) + + for _, field := range Fields() { + previous, duplicate := seen[field.FlagKey] + require.False(t, duplicate, "flag %q is used by both %s and %s", field.FlagKey, previous, field.Key) + + seen[field.FlagKey] = field.Key + } +} + +// declaredSettingsKeys parses settings_keys.go and returns every Key* constant +// as name -> value, so the check follows the source rather than a hand-kept +// list that would drift the same way the registry did. +func declaredSettingsKeys(t *testing.T) map[string]string { + t.Helper() + + file, err := parser.ParseFile(token.NewFileSet(), "settings_keys.go", nil, 0) + require.NoError(t, err) + + keys := make(map[string]string, 64) + + for _, decl := range file.Decls { + genDecl, ok := decl.(*ast.GenDecl) + if !ok || genDecl.Tok != token.CONST { + continue + } + + for _, spec := range genDecl.Specs { + valueSpec, ok := spec.(*ast.ValueSpec) + if !ok || len(valueSpec.Names) != 1 || len(valueSpec.Values) != 1 { + continue + } + + literal, ok := valueSpec.Values[0].(*ast.BasicLit) + if !ok || literal.Kind != token.STRING { + continue + } + + value, err := strconv.Unquote(literal.Value) + require.NoError(t, err) + + keys[valueSpec.Names[0].Name] = value + } + } + + require.NotEmpty(t, keys, "no settings keys found in settings_keys.go") + + return keys +} diff --git a/pkg/config/types.go b/pkg/config/types.go index 4c4decf1..87c8f6fd 100644 --- a/pkg/config/types.go +++ b/pkg/config/types.go @@ -24,32 +24,35 @@ type Config struct { // BuilderKeyIndex using the standard validator key path m/12381/3600/{index}/0/0. // Mutually exclusive with BuilderPrivkey. json:"-" keeps the secret out of every JSON // serialization path (WebUI REST + SSE); YAML config loading is unaffected. - BuilderMnemonic string `yaml:"builder_mnemonic" json:"-"` - BuilderKeyIndex uint64 `yaml:"builder_key_index" json:"builder_key_index"` - CLClient string `yaml:"cl_client" json:"cl_client,omitempty"` - ELEngineAPI string `yaml:"el_engine_api" json:"el_engine_api,omitempty"` // Engine API URL (required for payload building) - ELJWTSecret string `yaml:"el_jwt_secret" json:"el_jwt_secret,omitempty"` // Path to JWT secret file for engine API auth - ELRPC string `yaml:"el_rpc" json:"el_rpc,omitempty"` // Optional: EL JSON-RPC for transactions (lifecycle only) - WalletPrivkey string `yaml:"wallet_privkey" json:"wallet_privkey,omitempty"` // Optional: only if lifecycle enabled - APIPort int `yaml:"api_port" json:"api_port"` // Optional, 0 = disabled - AuthProviderURL string `yaml:"auth_provider_url" json:"auth_provider_url"` // Optional: authenticatoor URL; when set, API requests must carry a JWT verified against the authenticatoor's JWKS. When empty, the API is unauthenticated. - InjectHeadHTML string `yaml:"inject_head_html" json:"inject_head_html"` // Optional: raw HTML snippet (e.g. analytics tags) injected into of the served SPA. Falls back to BUILDOOR_INJECT_HEAD_HTML env var when empty. - OverviewURL string `yaml:"overview_url" json:"overview_url"` // Optional: URL of the multi-instance overview UI. When set, the dashboard renders an "Overview" entry in the top nav so operators get consistent navigation across instances. - LifecycleEnabled bool `yaml:"lifecycle_enabled" json:"lifecycle_enabled"` - EPBSEnabled bool `yaml:"epbs_enabled" json:"epbs_enabled"` // Initial enabled state for ePBS (service available if Gloas fork is scheduled) - BuilderAPIEnabled bool `yaml:"builder_api_enabled" json:"builder_api_enabled"` // Initial enabled state for Builder API - BuilderAPI BuilderAPIConfig `yaml:"builder_api" json:"builder_api"` // Builder API configuration - DepositAmount uint64 `yaml:"deposit_amount" json:"deposit_amount"` // Gwei, default 10 ETH - TopupThreshold uint64 `yaml:"topup_threshold" json:"topup_threshold"` // Gwei - TopupAmount uint64 `yaml:"topup_amount" json:"topup_amount"` // Gwei - DepositMaxFeeGwei uint64 `yaml:"deposit_max_fee" json:"deposit_max_fee"` - Schedule ScheduleConfig `yaml:"schedule" json:"schedule"` - EPBS EPBSConfig `yaml:"epbs" json:"epbs"` // Time-scheduled ePBS config - Reveal RevealConfig `yaml:"reveal" json:"reveal"` // Payload reveal config (shared by p2p bidder + Builder API) - Build BuildConfig `yaml:"build" json:"build"` // Payload build candidate policy - Debug bool `yaml:"debug" json:"debug"` - Pprof bool `yaml:"pprof" json:"pprof"` - PayloadBuildTime uint64 `yaml:"payload_build_time" json:"payload_build_time"` // The time given to the EL to build the payload after triggering the payload build via fcu (in ms) + BuilderMnemonic string `yaml:"builder_mnemonic" json:"-"` + BuilderKeyIndex uint64 `yaml:"builder_key_index" json:"builder_key_index"` + // BuilderKeys configures the managed set of internally derived builder keys + // that the entry key (BuilderPrivkey / BuilderMnemonic+BuilderKeyIndex) roots. + BuilderKeys BuilderKeysConfig `yaml:"builder_keys" json:"builder_keys"` + CLClient string `yaml:"cl_client" json:"cl_client,omitempty"` + ELEngineAPI string `yaml:"el_engine_api" json:"el_engine_api,omitempty"` // Engine API URL (required for payload building) + ELJWTSecret string `yaml:"el_jwt_secret" json:"el_jwt_secret,omitempty"` // Path to JWT secret file for engine API auth + ELRPC string `yaml:"el_rpc" json:"el_rpc,omitempty"` // Optional: EL JSON-RPC for transactions (lifecycle only) + WalletPrivkey string `yaml:"wallet_privkey" json:"wallet_privkey,omitempty"` // Optional: only if lifecycle enabled + APIPort int `yaml:"api_port" json:"api_port"` // Optional, 0 = disabled + AuthProviderURL string `yaml:"auth_provider_url" json:"auth_provider_url"` // Optional: authenticatoor URL; when set, API requests must carry a JWT verified against the authenticatoor's JWKS. When empty, the API is unauthenticated. + InjectHeadHTML string `yaml:"inject_head_html" json:"inject_head_html"` // Optional: raw HTML snippet (e.g. analytics tags) injected into of the served SPA. Falls back to BUILDOOR_INJECT_HEAD_HTML env var when empty. + OverviewURL string `yaml:"overview_url" json:"overview_url"` // Optional: URL of the multi-instance overview UI. When set, the dashboard renders an "Overview" entry in the top nav so operators get consistent navigation across instances. + LifecycleEnabled bool `yaml:"lifecycle_enabled" json:"lifecycle_enabled"` + EPBSEnabled bool `yaml:"epbs_enabled" json:"epbs_enabled"` // Initial enabled state for ePBS (service available if Gloas fork is scheduled) + BuilderAPIEnabled bool `yaml:"builder_api_enabled" json:"builder_api_enabled"` // Initial enabled state for Builder API + BuilderAPI BuilderAPIConfig `yaml:"builder_api" json:"builder_api"` // Builder API configuration + DepositAmount uint64 `yaml:"deposit_amount" json:"deposit_amount"` // Gwei, default 10 ETH + TopupThreshold uint64 `yaml:"topup_threshold" json:"topup_threshold"` // Gwei + TopupAmount uint64 `yaml:"topup_amount" json:"topup_amount"` // Gwei + DepositMaxFeeGwei uint64 `yaml:"deposit_max_fee" json:"deposit_max_fee"` + Schedule ScheduleConfig `yaml:"schedule" json:"schedule"` + EPBS EPBSConfig `yaml:"epbs" json:"epbs"` // Time-scheduled ePBS config + Reveal RevealConfig `yaml:"reveal" json:"reveal"` // Payload reveal config (shared by p2p bidder + Builder API) + Build BuildConfig `yaml:"build" json:"build"` // Payload build candidate policy + Debug bool `yaml:"debug" json:"debug"` + Pprof bool `yaml:"pprof" json:"pprof"` + PayloadBuildTime uint64 `yaml:"payload_build_time" json:"payload_build_time"` // The time given to the EL to build the payload after triggering the payload build via fcu (in ms) // ExtraData is the prefix injected into the built payload's extra-data field // (then padded with the EL's original extra data, truncated to 32 bytes). Used // to mark blocks built by this builder. Defaulted to "buildoor/" when empty. @@ -74,6 +77,48 @@ type Config struct { StateDBPath string `yaml:"state_db" json:"state_db,omitempty"` } +// BuilderKeysConfig defines the managed builder key set: how many keys buildoor +// keeps registered and funded, and how far the internal derivation may reach. +// +// Keys are derived from the entry key (see signer.DeriveInternalKey): internal +// index 0 is the entry key itself, so a TargetCount of 1 behaves exactly like a +// single-key deployment. +type BuilderKeysConfig struct { + // TargetCount is the number of builder keys kept registered and funded. + // Raising it deposits new keys; lowering it exits surplus keys when AutoExit + // is on. 0 is treated as 1. + TargetCount uint64 `yaml:"target_count" json:"target_count"` + + // MaxIndex caps internal derivation, bounding both the target and the + // startup discovery scan. + MaxIndex uint64 `yaml:"max_index" json:"max_index"` + + // DiscoveryGap is how many consecutive never-used indices end the startup + // scan for previously deposited keys. + DiscoveryGap uint64 `yaml:"discovery_gap" json:"discovery_gap"` + + // AutoDeposit deposits new keys to reach TargetCount. + AutoDeposit bool `yaml:"auto_deposit" json:"auto_deposit"` + + // AutoExit exits surplus keys when the managed count exceeds TargetCount. + // Irreversible: an exited builder key cannot be reactivated until its + // registry entry is reused by another builder's deposit. + AutoExit bool `yaml:"auto_exit" json:"auto_exit"` +} + +// EffectiveTargetCount returns the target key count clamped into the usable +// range: at least one key, and never more than the derivation cap allows +// (indices 0..MaxIndex, so MaxIndex+1 keys). +func (c *BuilderKeysConfig) EffectiveTargetCount() uint64 { + target := max(c.TargetCount, 1) + + if c.MaxIndex > 0 && target > c.MaxIndex+1 { + return c.MaxIndex + 1 + } + + return target +} + // ScheduleConfig defines when the builder should build blocks. type ScheduleConfig struct { Mode ScheduleMode `yaml:"mode" json:"mode"` // all, every_nth, next_n @@ -127,6 +172,12 @@ type BuilderAPIConfig struct { // legal parent tuple no candidate covers yet (bounded by the request's // response budget). OnDemandBuild bool `yaml:"on_demand_build" json:"on_demand_build"` + + // KeyStrategy selects which managed builder key signs served bids. Unlike + // p2p gossip there is no first-seen rule here, but the choice is sticky per + // (slot, parent tuple) so a polling proposer keeps seeing the same builder. + // Empty falls back to the ePBS key strategy. + KeyStrategy string `yaml:"key_strategy" json:"key_strategy"` } // ServeCandidateAllowed reports whether the given serve policy allows @@ -203,6 +254,25 @@ type EPBSConfig struct { // most nodes propagate only a builder's first bid per slot). BidCandidate string `yaml:"bid_candidate" json:"bid_candidate"` + // KeyStrategy selects which managed builder key signs each of a slot's + // bids: round_robin (default), single, random or least_used. Each key bids + // at most once per slot — the gossip rules ignore a builder's later bids — + // so with several built candidates the strategy decides which keys cover + // them. + KeyStrategy string `yaml:"key_strategy" json:"key_strategy"` + + // BidKeysPerSlot caps how many distinct builder keys bid a slot. 0 means + // no cap beyond the fleet itself; 1 reproduces single-key behaviour (one + // gossiped bid per slot) regardless of how many candidates were built. + BidKeysPerSlot uint64 `yaml:"bid_keys_per_slot" json:"bid_keys_per_slot"` + + // BidKeysPerStep is how many keys bid a payload per interval step, each + // one value-increment higher than the last. 1 (default) walks the fleet up + // the interval ladder one key at a time; 0 spends every remaining key at + // once, so a whole slot can be bid from all active keys in parallel as + // soon as the bid window opens. + BidKeysPerStep uint64 `yaml:"bid_keys_per_step" json:"bid_keys_per_step"` + // BidCandidateSwitch allows the auto selection to switch to a different // candidate mid-slot when the chain view changes. Default off: the first // gossiped candidate sticks (the gossip first-seen rule makes a switched diff --git a/pkg/lifecycle/balance.go b/pkg/lifecycle/balance.go index 0bfd01f6..87dbfc7d 100644 --- a/pkg/lifecycle/balance.go +++ b/pkg/lifecycle/balance.go @@ -3,128 +3,87 @@ package lifecycle import ( "context" "fmt" - "time" "github.com/ethpandaops/go-eth2-client/spec/phase0" "github.com/sirupsen/logrus" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" - "github.com/ethpandaops/buildoor/pkg/payload_bidder" - "github.com/ethpandaops/buildoor/pkg/rpc/beacon" ) -// topupCooldownEpochs suppresses further top-ups after one is submitted, giving -// a builder deposit time to land before the low balance triggers another (a -// builder deposit takes several epochs through the queue + fee-limit delay). +// topupCooldownEpochs suppresses further top-ups for a key after one is +// submitted, giving a builder deposit time to land before the low balance +// triggers another (a builder deposit takes several epochs through the queue + +// fee-limit delay). const topupCooldownEpochs phase0.Epoch = 8 -// BalanceService handles balance monitoring and automatic top-ups. +// BalanceService monitors builder key balances and performs automatic top-ups. +// Balances come from the key registry, which resolves them from the same epoch +// snapshot for the whole key set, so monitoring hundreds of keys costs one pass +// rather than one beacon query per key. type BalanceService struct { cfg *config.Config - clClient *beacon.Client + chainSvc chain.Service + registry *builder_keys.Registry depositSvc *DepositService - payments *payload_bidder.PaymentTracker - lastCheck time.Time - // lastTopupEpoch is the epoch of the most recent submitted top-up; 0 means - // none. Guards the cooldown so an in-flight deposit is not duplicated. - lastTopupEpoch phase0.Epoch - log logrus.FieldLogger + log logrus.FieldLogger } // NewBalanceService creates a new balance service. func NewBalanceService( cfg *config.Config, - clClient *beacon.Client, + chainSvc chain.Service, + registry *builder_keys.Registry, depositSvc *DepositService, - payments *payload_bidder.PaymentTracker, log logrus.FieldLogger, ) *BalanceService { return &BalanceService{ cfg: cfg, - clClient: clClient, + chainSvc: chainSvc, + registry: registry, depositSvc: depositSvc, - payments: payments, log: log.WithField("component", "balance-service"), } } -// GetCurrentBalance returns the builder's current balance from the beacon state. -func (s *BalanceService) GetCurrentBalance(ctx context.Context) (uint64, error) { - isRegistered, state, err := s.depositSvc.IsBuilderRegistered(ctx) - if err != nil { - return 0, fmt.Errorf("failed to check registration: %w", err) - } - - if !isRegistered { - return 0, fmt.Errorf("builder not registered") +// GetEffectiveBalance returns the key's spendable balance: the chain balance +// plus local adjustments (top-ups credited, revealed payments debited) minus +// the pending payments the beacon state still owes out of it. +func (s *BalanceService) GetEffectiveBalance(key *builder_keys.Key) (uint64, error) { + state := key.State() + if !state.HasBuilderIndex { + return 0, fmt.Errorf("builder key %s not registered", key) } - return state.Balance, nil + return s.registry.EffectiveBalance(key.KeyIndex()), nil } -// GetEffectiveBalance returns the live balance minus pending payments. -// Live balance = chain state balance + local adjustments (topups, revealed bid deductions). -// Pending payments = from chain state's BuilderPendingPayments (ground truth, survives restarts). -func (s *BalanceService) GetEffectiveBalance(ctx context.Context) (uint64, error) { - isRegistered, state, err := s.depositSvc.IsBuilderRegistered(ctx) - if err != nil { - return 0, fmt.Errorf("failed to check registration: %w", err) - } - - if !isRegistered { - return 0, fmt.Errorf("builder not registered") - } - - liveBalance := int64(state.Balance) - - // Apply local adjustments (topups add, revealed bids subtract since last state refresh) - if s.payments != nil { - liveBalance += s.payments.GetBalanceAdjustment() - } - - if liveBalance < 0 { - liveBalance = 0 - } - - // Get pending payments from chain state (ground truth from beacon state) - builderInfo := s.depositSvc.chainSvc.GetBuilderByPubkey(s.depositSvc.signer.PublicKey()) - if builderInfo != nil && builderInfo.PendingPayments > 0 { - effective := uint64(liveBalance) - if builderInfo.PendingPayments >= effective { - return 0, nil - } - - return effective - builderInfo.PendingPayments, nil - } +// NeedsTopup checks whether a key needs a top-up and returns the required amount. +// It returns ErrBuilderExited for a key whose exit has been initiated: after the +// sweep zeroes the balance a top-up would otherwise trigger every cooldown, +// cycling funds wallet -> exited entry -> (64 epochs locked) -> wallet forever. +func (s *BalanceService) NeedsTopup(key *builder_keys.Key) (bool, uint64, error) { + state := key.State() - return uint64(liveBalance), nil -} - -// NeedsTopup checks if a top-up is needed and returns the required amount. -// It returns ErrBuilderExited for a builder whose exit has been initiated: after the -// sweep zeroes the balance a top-up would otherwise trigger every cooldown, cycling -// funds wallet -> exited entry -> (64 epochs locked) -> wallet forever. -func (s *BalanceService) NeedsTopup(ctx context.Context) (bool, uint64, error) { - if chain.HasBuilderExited(s.depositSvc.chainSvc.GetBuilderByPubkey(s.depositSvc.signer.PublicKey())) { + switch state.Status { + case builder_keys.StatusExiting, builder_keys.StatusExited: return false, 0, ErrBuilderExited - } - - effectiveBalance, err := s.GetEffectiveBalance(ctx) - if err != nil { - return false, 0, err + case builder_keys.StatusPending, builder_keys.StatusActive: + // Registered: a top-up can land. + default: + return false, 0, nil } threshold := s.cfg.TopupThreshold - if effectiveBalance >= threshold { + if s.registry.EffectiveBalance(key.KeyIndex()) >= threshold { return false, 0, nil } // Hold off while a recent top-up is still expected to land, so the low // balance does not trigger duplicate deposits before the queued one arrives. - if s.lastTopupEpoch != 0 { - currentEpoch := s.depositSvc.chainSvc.GetCurrentEpoch() - if currentEpoch < s.lastTopupEpoch+topupCooldownEpochs { + if state.LastTopupEpoch != 0 { + if s.chainSvc.GetCurrentEpoch() < state.LastTopupEpoch+topupCooldownEpochs { return false, 0, nil } } @@ -137,44 +96,29 @@ func (s *BalanceService) NeedsTopup(ctx context.Context) (bool, uint64, error) { return true, topupAmount, nil } -// CheckAndTopup checks the balance and performs a top-up if needed. -func (s *BalanceService) CheckAndTopup(ctx context.Context) error { - needsTopup, amount, err := s.NeedsTopup(ctx) +// CheckAndTopup tops the key up when its balance is below the threshold. It +// returns the amount deposited in gwei, or 0 when no top-up was needed — the +// caller credits exactly that to the live balance rather than guessing it. +func (s *BalanceService) CheckAndTopup(ctx context.Context, key *builder_keys.Key) (uint64, error) { + needsTopup, amount, err := s.NeedsTopup(key) if err != nil { - return fmt.Errorf("failed to check if topup needed: %w", err) + return 0, fmt.Errorf("failed to check if topup needed: %w", err) } if !needsTopup { - return nil + return 0, nil } s.log.WithFields(logrus.Fields{ + "key": key.String(), "amount_gwei": amount, }).Info("Balance below threshold, topping up") - if err := s.depositSvc.CreateTopup(ctx, amount); err != nil { - return fmt.Errorf("failed to create topup: %w", err) + if err := s.depositSvc.CreateTopup(ctx, key, amount); err != nil { + return 0, fmt.Errorf("failed to create topup: %w", err) } - s.lastCheck = time.Now() - s.lastTopupEpoch = s.depositSvc.chainSvc.GetCurrentEpoch() - - return nil -} + s.registry.MarkToppedUp(key.KeyIndex(), s.chainSvc.GetCurrentEpoch()) -// RunBalanceMonitor runs a periodic balance check loop. -func (s *BalanceService) RunBalanceMonitor(ctx context.Context) { - ticker := time.NewTicker(1 * time.Minute) - defer ticker.Stop() - - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - if err := s.CheckAndTopup(ctx); err != nil { - s.log.WithError(err).Warn("Balance check failed") - } - } - } + return amount, nil } diff --git a/pkg/lifecycle/deposit.go b/pkg/lifecycle/deposit.go index dd1b7008..a2e07058 100644 --- a/pkg/lifecycle/deposit.go +++ b/pkg/lifecycle/deposit.go @@ -9,6 +9,7 @@ import ( "github.com/sirupsen/logrus" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/signer" @@ -48,12 +49,15 @@ func isDepositDeferred(err error) bool { // depositGasLimit is the gas limit for builder deposit transactions. const depositGasLimit = 1000000 +// depositConfirmTimeout bounds how long a deposit transaction may take to confirm. +const depositConfirmTimeout = 5 * time.Minute + // DepositService handles builder deposits and top-ups via the EIP-8282 builder -// deposit system contract. +// deposit system contract. It is key-agnostic: every operation names the builder +// key it acts on, so one service serves the whole managed key set. type DepositService struct { cfg *config.Config chainSvc chain.Service - signer *signer.BLSSigner wallet *wallet.Wallet log logrus.FieldLogger } @@ -62,7 +66,6 @@ type DepositService struct { func NewDepositService( cfg *config.Config, chainSvc chain.Service, - blsSigner *signer.BLSSigner, w *wallet.Wallet, log logrus.FieldLogger, ) (*DepositService, error) { @@ -79,15 +82,15 @@ func NewDepositService( return &DepositService{ cfg: cfg, chainSvc: chainSvc, - signer: blsSigner, wallet: w, log: depositLog, }, nil } -// IsBuilderRegistered checks if the builder is registered on the beacon chain. -func (s *DepositService) IsBuilderRegistered(_ context.Context) (bool, *BuilderState, error) { - pubkey := s.signer.PublicKey() +// IsBuilderRegistered checks if the given builder key is registered on the +// beacon chain. +func (s *DepositService) IsBuilderRegistered(key *builder_keys.Key) (bool, *BuilderState, error) { + pubkey := key.Pubkey() info := s.chainSvc.GetBuilderByPubkey(pubkey) if info == nil { @@ -107,26 +110,102 @@ func (s *DepositService) IsBuilderRegistered(_ context.Context) (bool, *BuilderS }, nil } -// CreateDeposit creates and sends an EIP-8282 builder deposit transaction. It is -// also used for top-ups (which are simply additional deposits for the same pubkey). +// CreateDeposit creates and sends an EIP-8282 builder deposit transaction for the +// given key. It is also used for top-ups (which are simply additional deposits +// for the same pubkey). // // Before submitting it reads the contract's current per-request queue fee and, when // DepositMaxFeeGwei is set, returns ErrDepositFeeTooHigh if the fee exceeds the limit // so the caller can delay and retry. The transaction value is stake + queue fee. -func (s *DepositService) CreateDeposit(ctx context.Context, amountGwei uint64) error { - pubkey := s.signer.PublicKey() +func (s *DepositService) CreateDeposit( + ctx context.Context, key *builder_keys.Key, amountGwei uint64, +) error { + fee, err := s.resolveDepositFee(ctx) + if err != nil { + return err + } + + request, err := s.depositRequest(key, amountGwei, fee) + if err != nil { + return err + } + + return s.sendDepositTransaction(ctx, request) +} + +// CreateDeposits submits deposits for several keys as one batch and returns the +// per-key errors (nil for the keys that landed). Batching matters because every +// deposit is serialized on the same funding key: bringing a fleet up one +// confirmed transaction at a time costs a block per key. +// +// The queue fee is read once for the whole batch — it is a property of the +// contract, not of a key — so a fee over the operator's limit defers the whole +// batch with a single error rather than per key. +func (s *DepositService) CreateDeposits( + ctx context.Context, keys []*builder_keys.Key, amountGwei uint64, +) ([]error, error) { + if len(keys) == 0 { + return nil, nil + } + + fee, err := s.resolveDepositFee(ctx) + if err != nil { + return nil, err + } + + errs := make([]error, len(keys)) + requests := make([]wallet.TxRequest, 0, len(keys)) + // requestKeys maps each built request back to its key, since keys whose + // request failed to build are not submitted. + requestKeys := make([]int, 0, len(keys)) + + for i, key := range keys { + request, err := s.depositRequest(key, amountGwei, fee) + if err != nil { + errs[i] = err + continue + } + + requests = append(requests, request) + requestKeys = append(requestKeys, i) + } + + for _, result := range s.wallet.SendBatchAndConfirm(ctx, requests, depositConfirmTimeout) { + keyIndex := requestKeys[result.Index] + + if result.Err != nil { + errs[keyIndex] = fmt.Errorf("deposit transaction failed: %w", result.Err) + continue + } + + s.log.WithFields(logrus.Fields{ + "key": keys[keyIndex].String(), + "tx_hash": result.Receipt.TxHash.Hex(), + "block_number": result.Receipt.BlockNumber.Uint64(), + }).Info("Deposit transaction confirmed") + } + + return errs, nil +} + +// depositRequest builds the signed deposit calldata and transaction parameters +// for one key. It performs no I/O beyond the chain-spec reads, so a batch can +// build every request before sending any of them. +func (s *DepositService) depositRequest( + key *builder_keys.Key, amountGwei uint64, fee *big.Int, +) (wallet.TxRequest, error) { + pubkey := key.Pubkey() // Refuse deposits for an exited builder entry: they cannot reactivate it and // are withdrawn back to the wallet, minus gas and the queue fee. A fresh // registration (pubkey absent from the registry) passes this check. if chain.HasBuilderExited(s.chainSvc.GetBuilderByPubkey(pubkey)) { - return ErrBuilderExited + return wallet.TxRequest{}, ErrBuilderExited } - s.log.WithField("amount_gwei", amountGwei).Info("Creating builder deposit") withdrawalCredentials := BuilderWithdrawalCredentials(s.wallet.Address()) - // Step 1: Compute the builder-deposit signing root (DOMAIN_BUILDER_DEPOSIT, + // Compute the builder-deposit signing root (DOMAIN_BUILDER_DEPOSIT, // GENESIS_FORK_VERSION) and sign it as a proof-of-possession. signingRoot, err := signer.ComputeBuilderDepositSigningRoot( pubkey, @@ -135,30 +214,24 @@ func (s *DepositService) CreateDeposit(ctx context.Context, amountGwei uint64) e s.chainSvc.GetGenesis().GenesisForkVersion, ) if err != nil { - return fmt.Errorf("failed to compute signing root: %w", err) + return wallet.TxRequest{}, fmt.Errorf("failed to compute signing root: %w", err) } - signature, err := s.signer.Sign(signingRoot[:]) + signature, err := key.BLSSigner().Sign(signingRoot[:]) if err != nil { - return fmt.Errorf("failed to sign deposit: %w", err) + return wallet.TxRequest{}, fmt.Errorf("failed to sign deposit: %w", err) } - // Step 2: Build the raw 184-byte request calldata. calldata, err := BuildBuilderDepositCalldata(pubkey[:], withdrawalCredentials[:], amountGwei, signature[:]) if err != nil { - return fmt.Errorf("failed to build deposit calldata: %w", err) + return wallet.TxRequest{}, fmt.Errorf("failed to build deposit calldata: %w", err) } - // Step 3: Resolve the queue fee and enforce the operator's fee limit. - fee, err := s.resolveDepositFee(ctx) - if err != nil { - return err - } - - // Step 4: msg.value = stake (wei) + queue fee (wei). + // msg.value = stake (wei) + queue fee (wei). value := new(big.Int).Add(GweiToWei(amountGwei), fee) s.log.WithFields(logrus.Fields{ + "key": key.String(), "pubkey": fmt.Sprintf("0x%x", pubkey[:]), "withdrawal_creds": fmt.Sprintf("0x%x", withdrawalCredentials[:]), "amount_gwei": amountGwei, @@ -166,14 +239,24 @@ func (s *DepositService) CreateDeposit(ctx context.Context, amountGwei uint64) e "value_wei": value.String(), }).Info("Builder deposit prepared") - return s.sendDepositTransaction(ctx, calldata, value) + return wallet.TxRequest{ + To: BuilderDepositContractAddress, + Value: value, + Data: calldata, + GasLimit: depositGasLimit, + }, nil } // CreateTopup creates and sends a top-up transaction (an additional deposit). -func (s *DepositService) CreateTopup(ctx context.Context, amountGwei uint64) error { - s.log.WithField("amount_gwei", amountGwei).Info("Creating builder top-up") +func (s *DepositService) CreateTopup( + ctx context.Context, key *builder_keys.Key, amountGwei uint64, +) error { + s.log.WithFields(logrus.Fields{ + "key": key.String(), + "amount_gwei": amountGwei, + }).Info("Creating builder top-up") - return s.CreateDeposit(ctx, amountGwei) + return s.CreateDeposit(ctx, key, amountGwei) } // resolveDepositFee reads the builder deposit contract's current queue fee and @@ -204,18 +287,19 @@ func (s *DepositService) resolveDepositFee(ctx context.Context) (*big.Int, error return fee, nil } -// sendDepositTransaction sends the deposit transaction to the builder deposit contract. +// sendDepositTransaction sends one deposit transaction to the builder deposit +// contract. // // SendAndConfirm sources a fresh nonce and resolves nonce conflicts/displacement, so // several instances can share this funding key safely. -func (s *DepositService) sendDepositTransaction(ctx context.Context, calldata []byte, value *big.Int) error { +func (s *DepositService) sendDepositTransaction(ctx context.Context, request wallet.TxRequest) error { receipt, err := s.wallet.SendAndConfirm( ctx, - BuilderDepositContractAddress, - value, - calldata, - depositGasLimit, - 5*time.Minute, + request.To, + request.Value, + request.Data, + request.GasLimit, + depositConfirmTimeout, ) if err != nil { return fmt.Errorf("deposit transaction failed: %w", err) diff --git a/pkg/lifecycle/early_deposit.go b/pkg/lifecycle/early_deposit.go index f5f854ef..a3faddbf 100644 --- a/pkg/lifecycle/early_deposit.go +++ b/pkg/lifecycle/early_deposit.go @@ -5,11 +5,12 @@ import ( "errors" "fmt" "strings" - "time" "github.com/ethereum/go-ethereum/accounts/abi" + "github.com/ethereum/go-ethereum/common" "github.com/sirupsen/logrus" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/signer" @@ -38,7 +39,6 @@ const depositContractABI = `[{"name":"deposit","type":"function","stateMutabilit type EarlyDepositService struct { cfg *config.Config chainSvc chain.Service - signer *signer.BLSSigner wallet *wallet.Wallet depositABI abi.ABI log logrus.FieldLogger @@ -48,7 +48,6 @@ type EarlyDepositService struct { func NewEarlyDepositService( cfg *config.Config, chainSvc chain.Service, - blsSigner *signer.BLSSigner, w *wallet.Wallet, log logrus.FieldLogger, ) (*EarlyDepositService, error) { @@ -60,23 +59,22 @@ func NewEarlyDepositService( return &EarlyDepositService{ cfg: cfg, chainSvc: chainSvc, - signer: blsSigner, wallet: w, depositABI: depositABI, log: log.WithField("component", "early-deposit-service"), }, nil } -// HasPendingDeposit reports whether this builder's pubkey is already present in the +// HasPendingDeposit reports whether the key's pubkey is already present in the // beacon state's pending_deposits queue. It is used after a restart to avoid submitting // a duplicate early deposit while a prior one is still waiting in the queue. -func (s *EarlyDepositService) HasPendingDeposit() bool { +func (s *EarlyDepositService) HasPendingDeposit(key *builder_keys.Key) bool { stats := s.chainSvc.GetCurrentEpochStats() if stats == nil { return false } - pubkey := s.signer.PublicKey() + pubkey := key.Pubkey() for i := range stats.PendingDeposits { if stats.PendingDeposits[i].Pubkey == pubkey { return true @@ -86,34 +84,84 @@ func (s *EarlyDepositService) HasPendingDeposit() bool { return false } -// CreateEarlyDeposit builds, signs and sends a validator deposit for this builder via -// the regular deposit contract. The deposit uses 0xB0 (BUILDER_WITHDRAWAL_PREFIX) withdrawal -// credentials pointing at the funding wallet and is signed with the validator deposit -// domain over GENESIS_FORK_VERSION. -func (s *EarlyDepositService) CreateEarlyDeposit(ctx context.Context, amountGwei uint64) error { +// CreateEarlyDeposits builds, signs and sends validator deposits for the given keys +// as one batch, returning the per-key errors (nil for the keys that landed). +// +// The deposits use 0xB0 (BUILDER_WITHDRAWAL_PREFIX) withdrawal credentials pointing at +// the funding wallet and are signed with the validator deposit domain over +// GENESIS_FORK_VERSION. They do not race each other: they sit in the pending-deposit +// queue together and the Gloas transition converts them all. +func (s *EarlyDepositService) CreateEarlyDeposits( + ctx context.Context, keys []*builder_keys.Key, amountGwei uint64, +) ([]error, error) { + if len(keys) == 0 { + return nil, nil + } + depositContract := s.chainSvc.GetChainSpec().DepositContractAddress if depositContract == nil { - return ErrNoDepositContract + return nil, ErrNoDepositContract + } + + errs := make([]error, len(keys)) + requests := make([]wallet.TxRequest, 0, len(keys)) + // requestKeys maps each built request back to its key, since keys whose + // request failed to build are not submitted. + requestKeys := make([]int, 0, len(keys)) + + for i, key := range keys { + request, err := s.earlyDepositRequest(key, amountGwei, *depositContract) + if err != nil { + errs[i] = err + continue + } + + requests = append(requests, request) + requestKeys = append(requestKeys, i) } - pubkey := s.signer.PublicKey() + for _, result := range s.wallet.SendBatchAndConfirm(ctx, requests, depositConfirmTimeout) { + keyIndex := requestKeys[result.Index] + + if result.Err != nil { + errs[keyIndex] = fmt.Errorf("early deposit transaction failed: %w", result.Err) + continue + } + + s.log.WithFields(logrus.Fields{ + "key": keys[keyIndex].String(), + "tx_hash": result.Receipt.TxHash.Hex(), + "block_number": result.Receipt.BlockNumber.Uint64(), + }).Info("Early deposit transaction confirmed") + } + + return errs, nil +} + +// earlyDepositRequest builds the signed deposit calldata and transaction +// parameters for one key. It performs no I/O, so a batch can build every request +// before sending any of them. +func (s *EarlyDepositService) earlyDepositRequest( + key *builder_keys.Key, amountGwei uint64, depositContract common.Address, +) (wallet.TxRequest, error) { + pubkey := key.Pubkey() withdrawalCredentials := ValidatorWithdrawalCredentials(s.wallet.Address()) genesisForkVersion := s.chainSvc.GetGenesis().GenesisForkVersion // Sign the deposit message with the validator deposit domain (DOMAIN_DEPOSIT). signingRoot, err := signer.ComputeDepositSigningRoot(pubkey, withdrawalCredentials, amountGwei, genesisForkVersion) if err != nil { - return fmt.Errorf("failed to compute deposit signing root: %w", err) + return wallet.TxRequest{}, fmt.Errorf("failed to compute deposit signing root: %w", err) } - signature, err := s.signer.Sign(signingRoot[:]) + signature, err := key.BLSSigner().Sign(signingRoot[:]) if err != nil { - return fmt.Errorf("failed to sign early deposit: %w", err) + return wallet.TxRequest{}, fmt.Errorf("failed to sign early deposit: %w", err) } depositDataRoot, err := signer.ComputeDepositDataRoot(pubkey, withdrawalCredentials, amountGwei, signature) if err != nil { - return fmt.Errorf("failed to compute deposit data root: %w", err) + return wallet.TxRequest{}, fmt.Errorf("failed to compute deposit data root: %w", err) } calldata, err := s.depositABI.Pack( @@ -124,12 +172,13 @@ func (s *EarlyDepositService) CreateEarlyDeposit(ctx context.Context, amountGwei [32]byte(depositDataRoot), ) if err != nil { - return fmt.Errorf("failed to encode deposit calldata: %w", err) + return wallet.TxRequest{}, fmt.Errorf("failed to encode deposit calldata: %w", err) } value := GweiToWei(amountGwei) s.log.WithFields(logrus.Fields{ + "key": key.String(), "pubkey": fmt.Sprintf("0x%x", pubkey[:]), "withdrawal_creds": fmt.Sprintf("0x%x", withdrawalCredentials[:]), "deposit_contract": depositContract.Hex(), @@ -137,15 +186,10 @@ func (s *EarlyDepositService) CreateEarlyDeposit(ctx context.Context, amountGwei "value_wei": value.String(), }).Info("Early builder deposit prepared (regular deposit contract)") - receipt, err := s.wallet.SendAndConfirm(ctx, *depositContract, value, calldata, depositGasLimit, 5*time.Minute) - if err != nil { - return fmt.Errorf("early deposit transaction failed: %w", err) - } - - s.log.WithFields(logrus.Fields{ - "tx_hash": receipt.TxHash.Hex(), - "block_number": receipt.BlockNumber.Uint64(), - }).Info("Early deposit transaction confirmed") - - return nil + return wallet.TxRequest{ + To: depositContract, + Value: value, + Data: calldata, + GasLimit: depositGasLimit, + }, nil } diff --git a/pkg/lifecycle/early_onboard.go b/pkg/lifecycle/early_onboard.go new file mode 100644 index 00000000..8fbae50e --- /dev/null +++ b/pkg/lifecycle/early_onboard.go @@ -0,0 +1,255 @@ +package lifecycle + +import ( + "context" + "fmt" + "time" + + "github.com/ethpandaops/go-eth2-client/spec/phase0" + "github.com/ethpandaops/go-eth2-client/spec/version" + "github.com/sirupsen/logrus" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" +) + +// earlyOnboard onboards the whole target key set before the Gloas fork via the +// regular validator deposit contract, so there is no coverage gap between the +// Fulu Builder-API range and Gloas. The deposits do not race each other: they +// sit in the pending-deposit queue together and the fork transition converts +// them all into builders. +// +// It returns immediately (no-op) when early onboarding does not apply: the early +// deposit service is unavailable, Gloas is not scheduled, no deposit contract is +// known, Gloas is already active, or every target key is already registered. +// +// When applicable it re-evaluates the deposit timing once per epoch until the +// batch is submitted (and it waited for registration), the fork is reached, or +// the manager stops. +func (m *Manager) earlyOnboard(ctx context.Context) { + if m.earlyDepositSvc == nil { + return + } + + spec := m.chainSvc.GetChainSpec() + if !spec.IsForkScheduled(version.DataVersionGloas) || spec.DepositContractAddress == nil { + return + } + + if spec.IsForkActive(version.DataVersionGloas, m.chainSvc.GetCurrentEpoch()) { + return + } + + m.registry.Refresh() + + if len(m.earlyOnboardTargets()) == 0 { + return + } + + forkEpoch := spec.GetForkEpoch(version.DataVersionGloas) + + m.log.WithFields(logrus.Fields{ + "gloas_fork_epoch": forkEpoch, + "target_keys": m.cfg.BuilderKeys.EffectiveTargetCount(), + }).Info("Gloas scheduled, evaluating early builder onboarding") + m.fireEvent("early_onboard", fmt.Sprintf( + "Gloas fork at epoch %d, preparing early onboarding of %d builder keys", + forkEpoch, m.cfg.BuilderKeys.EffectiveTargetCount()), "info") + + // Subscribe before the first evaluation so an epoch transition can't slip through + // between a "wait" decision and the subscription. + epochSub := m.chainSvc.SubscribeEpochStats() + defer epochSub.Unsubscribe() + + for { + if m.tryEarlyOnboardOnce(ctx, forkEpoch) { + return + } + + select { + case <-ctx.Done(): + return + case <-m.stopCh: + return + case _, ok := <-epochSub.Channel(): + if !ok { + return + } + } + } +} + +// earlyOnboardTargets returns the target keys that still need an early deposit: +// not yet in the builder registry and not already waiting in the pending-deposit +// queue (restart safety — a prior run's deposits must not be submitted twice). +func (m *Manager) earlyOnboardTargets() []*builder_keys.Key { + target := m.cfg.BuilderKeys.EffectiveTargetCount() + + pending := make([]*builder_keys.Key, 0, target) + + for keyIndex := uint64(0); keyIndex < target; keyIndex++ { + key, err := m.registry.Key(keyIndex) + if err != nil { + m.log.WithError(err).WithField("key_index", keyIndex). + Warn("Cannot derive builder key for early onboarding") + + break + } + + if m.chainSvc.GetBuilderByPubkey(key.Pubkey()) != nil { + continue + } + + if m.earlyDepositSvc.HasPendingDeposit(key) { + continue + } + + pending = append(pending, key) + } + + return pending +} + +// tryEarlyOnboardOnce performs one early-onboarding evaluation. It returns true when the +// early-onboarding phase is complete and the loop should stop: every target key is +// onboarded or queued (and we waited for registration), or the fork was reached without +// onboarding (handed off to the normal post-fork flow). +func (m *Manager) tryEarlyOnboardOnce(ctx context.Context, forkEpoch phase0.Epoch) bool { + if !m.enabled.Load() { + return false // paused; keep waiting until re-enabled + } + + m.registry.Refresh() + + currentEpoch := m.chainSvc.GetCurrentEpoch() + if currentEpoch >= forkEpoch { + // Fork reached without onboarding — let the normal post-fork flow take over. + return true + } + + remaining := m.earlyOnboardTargets() + if len(remaining) == 0 { + // Everything is registered or already queued; wait for the fork transition + // to convert the queued deposits into builders. + m.log.Info("All target builder keys onboarded or queued, waiting for registration") + m.fireEvent("early_onboard", "All target keys deposited, waiting for the Gloas transition", "info") + m.waitForEarlyRegistration(ctx, forkEpoch, currentEpoch) + + return true + } + + spec := m.chainSvc.GetChainSpec() + epochsUntilFork := uint64(forkEpoch - currentEpoch) + forkSlot := uint64(forkEpoch) * spec.SlotsPerEpoch + slotsUntilFork := forkSlot - uint64(m.chainSvc.GetCurrentSlot()) + + // Only onboard early if there is enough runway before the fork. Closer than the + // minimum, the early deposits may not land in (and finalize within) the pending + // queue in time, so abandon early onboarding and let the normal post-fork flow + // register the keys via the builder deposit contract instead. + if slotsUntilFork < minEarlyOnboardSlots { + m.log.WithFields(logrus.Fields{ + "fork_epoch": forkEpoch, + "slots_until_fork": slotsUntilFork, + "remaining_keys": len(remaining), + }).Info("Fewer than minimum slots until Gloas, skipping early onboarding (will deposit via builder contract after the fork)") + m.fireEvent("early_onboard", fmt.Sprintf( + "Only %d slots until Gloas, skipping early onboarding of %d keys; will deposit after the fork", + slotsUntilFork, len(remaining)), "info") + + return true + } + + amount := m.cfg.DepositAmount + + // Two deposit windows (per design): at least earlyOnboardFinalizationMargin epochs + // before the fork if the pending-deposit queue is long enough to shield the whole + // batch from being processed before the fork; otherwise at the latest epoch boundary + // that still leaves at least minEarlyOnboardSlots before the fork (a fresh + // back-of-queue batch is guaranteed to survive the remaining transitions). + lastSafeEpoch := (epochsUntilFork-1)*spec.SlotsPerEpoch < minEarlyOnboardSlots + shouldDeposit := lastSafeEpoch || + (epochsUntilFork >= earlyOnboardFinalizationMargin && + depositsSurviveUntilFork(m.chainSvc.GetCurrentEpochStats(), spec, + amount, uint64(len(remaining)), forkEpoch)) + + if !shouldDeposit { + m.log.WithFields(logrus.Fields{ + "current_epoch": currentEpoch, + "fork_epoch": forkEpoch, + "epochs_until_fork": epochsUntilFork, + "slots_until_fork": slotsUntilFork, + "remaining_keys": len(remaining), + }).Debug("Waiting for early onboarding deposit window") + + return false + } + + m.fireEvent("early_onboard", fmt.Sprintf( + "Submitting %d early onboarding deposits (%d gwei each, %d slots before fork)", + len(remaining), amount, slotsUntilFork), "info") + + if m.depositPendingCallback != nil { + m.depositPendingCallback() + } + + //nolint:gosec // the target key count is bounded by the derivation cap + if !m.walletCanFund(ctx, amount*uint64(len(remaining))) { + return false + } + + // The deposits go out as one batch: they do not race each other, they sit in + // the pending-deposit queue together and the fork transition converts them all. + errs, err := m.earlyDepositSvc.CreateEarlyDeposits(ctx, remaining, amount) + if err != nil { + m.log.WithError(err).Warn("Early onboarding deposits failed, retrying next epoch") + m.fireEvent("early_onboard", fmt.Sprintf("Early deposits failed: %v, retrying", err), "warning") + + return false + } + + submitted := 0 + + for i, key := range remaining { + if errs[i] != nil { + m.log.WithError(errs[i]).WithField("key", key.String()). + Warn("Early onboarding deposit failed, retrying next epoch") + m.fireEvent("early_onboard", fmt.Sprintf( + "Early deposit for key #%d failed: %v, retrying", key.KeyIndex(), errs[i]), "warning") + + continue + } + + m.registry.MarkDepositSubmitted(key.KeyIndex()) + + submitted++ + } + + if submitted == 0 { + return false // nothing landed; retry the whole batch next epoch + } + + m.fireEvent("early_onboard", fmt.Sprintf( + "%d early deposits confirmed, waiting for fork transition and registration", + submitted), "success") + m.waitForEarlyRegistration(ctx, forkEpoch, currentEpoch) + + return true +} + +// waitForEarlyRegistration waits for the primary key to be registered after the early +// deposits. The registration only happens once the Gloas fork converts the pending +// deposits into builders, which can be several epochs out, so the timeout spans until a +// few epochs past the fork. On timeout it logs and returns; the reconciler then retries +// via the builder deposit contract as a fallback. +func (m *Manager) waitForEarlyRegistration(ctx context.Context, forkEpoch, currentEpoch phase0.Epoch) { + spec := m.chainSvc.GetChainSpec() + + epochsToWait := uint64(forkEpoch-currentEpoch) + earlyOnboardFinalizationMargin + timeout := max(time.Duration(epochsToWait*spec.SlotsPerEpoch)*spec.SecondsPerSlot, 5*time.Minute) + + if err := m.WaitForRegistration(ctx, m.registry.Primary(), timeout); err != nil { + m.log.WithError(err).Warn("Builder not registered after early deposits; the reconciler will retry") + m.fireEvent("early_onboard", + "Builder not yet registered after the early deposits; the reconciler will retry", "warning") + } +} diff --git a/pkg/lifecycle/exit.go b/pkg/lifecycle/exit.go index d5149daa..21eb66b4 100644 --- a/pkg/lifecycle/exit.go +++ b/pkg/lifecycle/exit.go @@ -7,8 +7,8 @@ import ( "github.com/sirupsen/logrus" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/wallet" ) @@ -28,7 +28,6 @@ const exitGasLimit = 1000000 // as msg.value. type ExitService struct { chainSvc chain.Service - signer *signer.BLSSigner wallet *wallet.Wallet log logrus.FieldLogger } @@ -36,27 +35,28 @@ type ExitService struct { // NewExitService creates a new exit service. func NewExitService( chainSvc chain.Service, - blsSigner *signer.BLSSigner, w *wallet.Wallet, log logrus.FieldLogger, ) *ExitService { return &ExitService{ chainSvc: chainSvc, - signer: blsSigner, wallet: w, log: log.WithField("component", "exit-service"), } } -// CreateExit submits a builder exit request transaction for this builder. +// CreateExit submits a builder exit request transaction for the given key. // // Exits always proceed regardless of the configured deposit fee limit (unlike // deposits/top-ups, which are delayed when the fee is too high), so the operator can // always withdraw. The queue fee is read from the contract and paid as msg.value. -func (s *ExitService) CreateExit(ctx context.Context) error { - pubkey := s.signer.PublicKey() +func (s *ExitService) CreateExit(ctx context.Context, key *builder_keys.Key) error { + pubkey := key.Pubkey() - s.log.WithField("pubkey", fmt.Sprintf("0x%x", pubkey[:])).Info("Creating builder exit") + s.log.WithFields(logrus.Fields{ + "key": key.String(), + "pubkey": fmt.Sprintf("0x%x", pubkey[:]), + }).Info("Creating builder exit") calldata, err := BuildBuilderExitCalldata(pubkey[:]) if err != nil { diff --git a/pkg/lifecycle/manager.go b/pkg/lifecycle/manager.go index 970e94a2..8ec3142a 100644 --- a/pkg/lifecycle/manager.go +++ b/pkg/lifecycle/manager.go @@ -4,7 +4,6 @@ package lifecycle import ( "context" - "errors" "fmt" "sync" "sync/atomic" @@ -12,13 +11,12 @@ import ( "github.com/sirupsen/logrus" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/wallet" - "github.com/ethpandaops/go-eth2-client/spec/phase0" "github.com/ethpandaops/go-eth2-client/spec/version" ) @@ -44,7 +42,7 @@ type Manager struct { cfg *config.Config clClient *beacon.Client chainSvc chain.Service - signer *signer.BLSSigner + registry *builder_keys.Registry wallet *wallet.Wallet builderState *BuilderState stateMu sync.RWMutex @@ -63,8 +61,14 @@ type Manager struct { enabled atomic.Bool // exitNoticed dedupes the exited-builder warning event; re-armed when the // pubkey shows up unexited again (fresh registration after registry reuse). - exitNoticed atomic.Bool - eventCallback func(*LifecycleEvent) + exitNoticed atomic.Bool + // walletUnderfunded dedupes the "fund the wallet" event; re-armed once the + // wallet can cover a deposit again. + walletUnderfunded atomic.Bool + eventCallback func(*LifecycleEvent) + + // poke requests an immediate reconcile pass (cap 1). + poke chan struct{} } // NewManager creates a new lifecycle manager. @@ -72,7 +76,7 @@ func NewManager( cfg *config.Config, clClient *beacon.Client, chainSvc chain.Service, - blsSigner *signer.BLSSigner, + registry *builder_keys.Registry, w *wallet.Wallet, log logrus.FieldLogger, ) (*Manager, error) { @@ -82,15 +86,16 @@ func NewManager( cfg: cfg, clClient: clClient, chainSvc: chainSvc, - signer: blsSigner, + registry: registry, wallet: w, builderState: &BuilderState{}, log: managerLog, stopCh: make(chan struct{}), + poke: make(chan struct{}, 1), } // Initialize services - depositSvc, err := NewDepositService(cfg, chainSvc, blsSigner, w, managerLog) + depositSvc, err := NewDepositService(cfg, chainSvc, w, managerLog) if err != nil { return nil, fmt.Errorf("failed to create deposit service: %w", err) } @@ -99,7 +104,7 @@ func NewManager( // Early deposit service (regular validator deposit contract, used to onboard the // builder before the Gloas fork so there is no Builder-API-to-Gloas coverage gap). - earlyDepositSvc, err := NewEarlyDepositService(cfg, chainSvc, blsSigner, w, managerLog) + earlyDepositSvc, err := NewEarlyDepositService(cfg, chainSvc, w, managerLog) if err != nil { return nil, fmt.Errorf("failed to create early deposit service: %w", err) } @@ -107,7 +112,7 @@ func NewManager( m.earlyDepositSvc = earlyDepositSvc // Exit service (builder exit system contract, sent from the funding wallet) - m.exitSvc = NewExitService(chainSvc, blsSigner, w, managerLog) + m.exitSvc = NewExitService(chainSvc, w, managerLog) return m, nil } @@ -152,7 +157,7 @@ func (m *Manager) fireEvent(action, message, status string) { func (m *Manager) Start(ctx context.Context) error { m.wg.Add(1) - go m.runRegistrationAndMonitor(ctx) + go m.runReconciler(ctx) m.log.Info("Lifecycle manager started") @@ -177,15 +182,21 @@ func (m *Manager) GetBuilderState() *BuilderState { return &state } +// Registry returns the managed builder key set. +func (m *Manager) Registry() *builder_keys.Registry { + return m.registry +} + // GetWallet returns the wallet instance. func (m *Manager) GetWallet() *wallet.Wallet { return m.wallet } -// EnsureBuilderRegistered checks if builder is registered and deposits if needed. -// This is the synchronous version used by CLI commands (e.g. cmd/deposit.go). -func (m *Manager) EnsureBuilderRegistered(ctx context.Context) error { - isRegistered, state, err := m.depositSvc.IsBuilderRegistered(ctx) +// EnsureBuilderRegistered checks whether the given builder key is registered and +// deposits if needed. This is the synchronous version used by CLI commands +// (e.g. cmd/deposit.go) and the lifecycle API. +func (m *Manager) EnsureBuilderRegistered(ctx context.Context, key *builder_keys.Key) error { + isRegistered, state, err := m.depositSvc.IsBuilderRegistered(key) if err != nil { return fmt.Errorf("failed to check builder registration: %w", err) } @@ -217,7 +228,7 @@ func (m *Manager) EnsureBuilderRegistered(ctx context.Context) error { m.depositPendingCallback() } - if err := m.depositSvc.CreateDeposit(ctx, m.cfg.DepositAmount); err != nil { + if err := m.depositSvc.CreateDeposit(ctx, key, m.cfg.DepositAmount); err != nil { if isDepositDeferred(err) { // Fee too high or contract not active yet — delay, don't treat as failure. m.fireEvent("deposit", fmt.Sprintf("Deposit deferred: %v", err), "info") @@ -231,35 +242,64 @@ func (m *Manager) EnsureBuilderRegistered(ctx context.Context) error { m.fireEvent("deposit", "Deposit transaction confirmed, waiting for beacon chain inclusion", "success") // Wait for registration - return m.WaitForRegistration(ctx, 5*time.Minute) + return m.WaitForRegistration(ctx, key, 5*time.Minute) } -// CheckAndTopup checks balance and tops up if needed. -func (m *Manager) CheckAndTopup(ctx context.Context) error { +// CheckAndTopup tops the given key up when its balance is below the threshold. +func (m *Manager) CheckAndTopup(ctx context.Context, key *builder_keys.Key) error { if m.balanceSvc == nil { return nil } - return m.balanceSvc.CheckAndTopup(ctx) + amount, err := m.balanceSvc.CheckAndTopup(ctx, key) + if err != nil { + return err + } + + if amount > 0 { + if tracker := m.GetPaymentTracker(); tracker != nil { + tracker.AddDeposit(key.KeyIndex(), amount) + } + } + + return nil } -// InitiateExit submits a builder exit request via the builder exit system contract. -func (m *Manager) InitiateExit(ctx context.Context) error { - m.stateMu.RLock() - builderIndex := m.builderState.Index - isRegistered := m.builderState.IsRegistered - m.stateMu.RUnlock() - - // Index 0 is a valid builder index; only the registration flag tells us - // whether an exit can be submitted. - if !isRegistered { - return fmt.Errorf("builder not registered") +// TopupKey submits a top-up deposit for the key regardless of its current +// balance — the operator asked for it explicitly. amountGwei of 0 uses the +// configured top-up amount. +func (m *Manager) TopupKey(ctx context.Context, key *builder_keys.Key, amountGwei uint64) error { + if amountGwei == 0 { + amountGwei = m.cfg.TopupAmount } - // Check the live chain entry, not the cached state: an already-exited builder + if err := m.depositSvc.CreateTopup(ctx, key, amountGwei); err != nil { + return err + } + + m.registry.MarkToppedUp(key.KeyIndex(), m.chainSvc.GetCurrentEpoch()) + + if tracker := m.GetPaymentTracker(); tracker != nil { + tracker.AddDeposit(key.KeyIndex(), amountGwei) + } + + m.fireEvent("balance_topup", fmt.Sprintf( + "Key #%d topped up by %d gwei", key.KeyIndex(), amountGwei), "success") + + return nil +} + +// InitiateExit submits a builder exit request for the given key via the builder +// exit system contract. +func (m *Manager) InitiateExit(ctx context.Context, key *builder_keys.Key) error { + // Check the live chain entry, not a cached snapshot: an already-exited builder // fails is_active_builder on the beacon side, so a second request only wastes // the queue fee. - info := m.chainSvc.GetBuilderByPubkey(m.signer.PublicKey()) + info := m.chainSvc.GetBuilderByPubkey(key.Pubkey()) + if info == nil { + return fmt.Errorf("builder key #%d is not registered", key.KeyIndex()) + } + if chain.HasBuilderExited(info) { return fmt.Errorf("builder exit already initiated (withdrawable epoch %d)", info.WithdrawableEpoch) } @@ -267,32 +307,38 @@ func (m *Manager) InitiateExit(ctx context.Context) error { // The beacon chain silently ignores exit requests while the builder has pending // payments (get_pending_balance_to_withdraw_for_builder != 0) — the transaction // would confirm but the exit never happen. - if info != nil && info.PendingPayments > 0 { - return fmt.Errorf("builder has %d gwei in pending payments; the exit request would be ignored on chain — retry after they settle", info.PendingPayments) + if info.PendingPayments > 0 { + return fmt.Errorf("builder key #%d has %d gwei in pending payments; the exit request would be ignored on chain — retry after they settle", + key.KeyIndex(), info.PendingPayments) } - m.fireEvent("exit", fmt.Sprintf("Submitting builder exit for builder index %d", builderIndex), "info") + m.fireEvent("exit", fmt.Sprintf("Submitting exit for builder key #%d (builder index %d)", + key.KeyIndex(), info.Index), "info") - if err := m.exitSvc.CreateExit(ctx); err != nil { - m.fireEvent("exit", fmt.Sprintf("Exit failed: %v", err), "error") + if err := m.exitSvc.CreateExit(ctx, key); err != nil { + m.fireEvent("exit", fmt.Sprintf("Exit for key #%d failed: %v", key.KeyIndex(), err), "error") return err } - m.fireEvent("exit", fmt.Sprintf("Builder exit submitted for builder index %d", builderIndex), "success") + m.registry.MarkExitSubmitted(key.KeyIndex()) + m.fireEvent("exit", fmt.Sprintf("Builder key #%d exit submitted (builder index %d)", + key.KeyIndex(), info.Index), "success") return nil } -// WaitForRegistration waits for the builder to be registered. -func (m *Manager) WaitForRegistration(ctx context.Context, timeout time.Duration) error { +// WaitForRegistration waits for the given builder key to be registered. +func (m *Manager) WaitForRegistration( + ctx context.Context, key *builder_keys.Key, timeout time.Duration, +) error { timeoutCtx, cancel := context.WithTimeout(ctx, timeout) defer cancel() ticker := time.NewTicker(m.chainSvc.GetChainSpec().SecondsPerSlot) // Check every slot defer ticker.Stop() - pubkey := m.signer.PublicKey() + pubkey := key.Pubkey() for { select { @@ -336,7 +382,7 @@ func (m *Manager) WaitForRegistration(ctx context.Context, timeout time.Duration // stores it for direct access. func (m *Manager) SetPaymentTracker(payments *payload_bidder.PaymentTracker) { m.payments = payments - m.balanceSvc = NewBalanceService(m.cfg, m.clClient, m.depositSvc, payments, m.log) + m.balanceSvc = NewBalanceService(m.cfg, m.chainSvc, m.registry, m.depositSvc, m.log) } // GetPaymentTracker returns the shared payment tracker. @@ -353,38 +399,6 @@ func (m *Manager) onRegistered(index uint64) { } } -// runRegistrationAndMonitor handles async registration then balance monitoring. -// When disabled, it waits until re-enabled before proceeding. -func (m *Manager) runRegistrationAndMonitor(ctx context.Context) { - defer m.wg.Done() - - // Wait until enabled before doing anything - if !m.waitForEnabled(ctx) { - return - } - - // Step 0: Try to onboard the builder before the Gloas fork via the regular deposit - // contract (no-op when not applicable), so coverage is continuous across the fork. - m.maybeEarlyOnboard(ctx) - - // Step 1: Wait until the chain has loaded a Gloas (or later) beacon state. - // The on-chain builder set is available from the first Gloas EpochStats; the - // fork being active by epoch is not sufficient — the state must be fetched - // and cached before we can tell whether this builder is already registered - // (otherwise we'd read an empty set and deposit again unnecessarily). - if !m.waitForGloasState(ctx) { - return // stopped or context cancelled - } - - // Step 2: Ensure builder is registered (with retries) - if !m.registrationDone.Load() { - m.ensureRegisteredWithRetry(ctx) - } - - // Step 3: Run balance monitor (checks enabled flag each tick) - m.runBalanceMonitor(ctx) -} - // waitForEnabled waits until the manager is enabled or stopped. func (m *Manager) waitForEnabled(ctx context.Context) bool { if m.enabled.Load() { @@ -413,170 +427,6 @@ func (m *Manager) waitForEnabled(ctx context.Context) bool { } } -// maybeEarlyOnboard onboards the builder before the Gloas fork via the regular validator -// deposit contract, so there is no coverage gap between the Fulu Builder-API range and -// Gloas. It returns immediately (no-op) when early onboarding does not apply: the early -// deposit service is unavailable, Gloas is not scheduled, no deposit contract is known, -// Gloas is already active, or the builder is already registered. -// -// When applicable it re-evaluates the deposit timing once per epoch until it submits the -// deposit (and waits for registration), the builder gets registered, the fork is reached, -// or the manager stops. -func (m *Manager) maybeEarlyOnboard(ctx context.Context) { - if m.earlyDepositSvc == nil { - return - } - - spec := m.chainSvc.GetChainSpec() - if !spec.IsForkScheduled(version.DataVersionGloas) || spec.DepositContractAddress == nil { - return - } - - if spec.IsForkActive(version.DataVersionGloas, m.chainSvc.GetCurrentEpoch()) { - return - } - - if m.chainSvc.GetBuilderByPubkey(m.signer.PublicKey()) != nil { - return - } - - forkEpoch := spec.GetForkEpoch(version.DataVersionGloas) - - m.log.WithField("gloas_fork_epoch", forkEpoch).Info("Gloas scheduled, evaluating early builder onboarding") - m.fireEvent("early_onboard", fmt.Sprintf("Gloas fork at epoch %d, preparing early builder onboarding", forkEpoch), "info") - - // Subscribe before the first evaluation so an epoch transition can't slip through - // between a "wait" decision and the subscription. - epochSub := m.chainSvc.SubscribeEpochStats() - defer epochSub.Unsubscribe() - - for { - if m.tryEarlyOnboardOnce(ctx, forkEpoch) { - return - } - - select { - case <-ctx.Done(): - return - case <-m.stopCh: - return - case _, ok := <-epochSub.Channel(): - if !ok { - return - } - } - } -} - -// tryEarlyOnboardOnce performs one early-onboarding evaluation. It returns true when the -// early-onboarding phase is complete and the loop should stop: the builder is registered, -// the deposit was submitted (and we waited for registration), or the fork was reached -// without onboarding (handed off to the normal post-fork flow). -func (m *Manager) tryEarlyOnboardOnce(ctx context.Context, forkEpoch phase0.Epoch) bool { - if !m.enabled.Load() { - return false // paused; keep waiting until re-enabled - } - - if info := m.chainSvc.GetBuilderByPubkey(m.signer.PublicKey()); info != nil { - m.onRegistered(info.Index) - - return true - } - - currentEpoch := m.chainSvc.GetCurrentEpoch() - if currentEpoch >= forkEpoch { - // Fork reached without onboarding — let the normal post-fork flow take over. - return true - } - - // Restart safety: if our deposit is already in the pending queue, don't submit again; - // just wait for the fork transition to convert it into a builder. - if m.earlyDepositSvc.HasPendingDeposit() { - m.log.Info("Early builder deposit already pending, waiting for registration") - m.fireEvent("early_onboard", "Early deposit already pending, waiting for registration", "info") - m.waitForEarlyRegistration(ctx, forkEpoch, currentEpoch) - - return true - } - - spec := m.chainSvc.GetChainSpec() - epochsUntilFork := uint64(forkEpoch - currentEpoch) - forkSlot := uint64(forkEpoch) * spec.SlotsPerEpoch - slotsUntilFork := forkSlot - uint64(m.chainSvc.GetCurrentSlot()) - - // Only onboard early if there is enough runway before the fork. Closer than the - // minimum, the early deposit may not land in (and finalize within) the pending - // queue in time, so abandon early onboarding and let the normal post-fork flow - // register the builder via the builder deposit contract instead. - if slotsUntilFork < minEarlyOnboardSlots { - m.log.WithFields(logrus.Fields{ - "fork_epoch": forkEpoch, - "slots_until_fork": slotsUntilFork, - }).Info("Fewer than minimum slots until Gloas, skipping early onboarding (will deposit via builder contract after the fork)") - m.fireEvent("early_onboard", fmt.Sprintf("Only %d slots until Gloas, skipping early onboarding; will deposit after the fork", slotsUntilFork), "info") - - return true - } - - amount := m.cfg.DepositAmount - - // Two deposit windows (per design): at least earlyOnboardFinalizationMargin epochs - // before the fork if the pending-deposit queue is long enough to shield our deposit - // from being processed before the fork; otherwise at the latest epoch boundary that - // still leaves at least minEarlyOnboardSlots before the fork (a fresh back-of-queue - // deposit is guaranteed to survive the remaining transitions). - lastSafeEpoch := (epochsUntilFork-1)*spec.SlotsPerEpoch < minEarlyOnboardSlots - shouldDeposit := lastSafeEpoch || - (epochsUntilFork >= earlyOnboardFinalizationMargin && - depositSurvivesUntilFork(m.chainSvc.GetCurrentEpochStats(), spec, amount, forkEpoch)) - - if !shouldDeposit { - m.log.WithFields(logrus.Fields{ - "current_epoch": currentEpoch, - "fork_epoch": forkEpoch, - "epochs_until_fork": epochsUntilFork, - "slots_until_fork": slotsUntilFork, - }).Debug("Waiting for early onboarding deposit window") - - return false - } - - m.fireEvent("early_onboard", fmt.Sprintf("Submitting early onboarding deposit (%d gwei, %d slots before fork)", amount, slotsUntilFork), "info") - - if m.depositPendingCallback != nil { - m.depositPendingCallback() - } - - if err := m.earlyDepositSvc.CreateEarlyDeposit(ctx, amount); err != nil { - m.log.WithError(err).Warn("Early onboarding deposit failed, retrying next epoch") - m.fireEvent("early_onboard", fmt.Sprintf("Early deposit failed: %v, retrying", err), "warning") - - return false // retry on the next epoch - } - - m.fireEvent("early_onboard", "Early deposit confirmed, waiting for fork transition and registration", "success") - m.waitForEarlyRegistration(ctx, forkEpoch, currentEpoch) - - return true -} - -// waitForEarlyRegistration waits for the builder to be registered after an early deposit. -// The registration only happens once the Gloas fork converts the pending deposit into a -// builder, which can be several epochs out, so the timeout spans until a few epochs past -// the fork. On timeout it logs and returns; the normal post-fork flow then retries via the -// builder deposit contract as a fallback. -func (m *Manager) waitForEarlyRegistration(ctx context.Context, forkEpoch, currentEpoch phase0.Epoch) { - spec := m.chainSvc.GetChainSpec() - - epochsToWait := uint64(forkEpoch-currentEpoch) + earlyOnboardFinalizationMargin - timeout := max(time.Duration(epochsToWait*spec.SlotsPerEpoch)*spec.SecondsPerSlot, 5*time.Minute) - - if err := m.WaitForRegistration(ctx, timeout); err != nil { - m.log.WithError(err).Warn("Builder not registered after early deposit; normal post-fork flow will retry") - m.fireEvent("early_onboard", "Builder not yet registered after early deposit; post-fork flow will retry", "warning") - } -} - // waitForGloasState blocks until the chain service has loaded a Gloas (or later) // beacon state — the first EpochStats from which the on-chain builder set is // available. It logs/fires a waiting event only when it actually has to wait. @@ -615,99 +465,6 @@ func (m *Manager) waitForGloasState(ctx context.Context) bool { } } -// ensureRegisteredWithRetry attempts registration in a loop until success or stop. -func (m *Manager) ensureRegisteredWithRetry(ctx context.Context) { - for { - select { - case <-ctx.Done(): - return - case <-m.stopCh: - return - default: - } - - err := m.EnsureBuilderRegistered(ctx) - if err == nil { - return - } - - if isDepositDeferred(err) { - // Queue fee too high or contract not active yet — keep retrying quietly. - m.log.WithError(err).Info("Builder registration deferred, retrying in 30s") - } else { - m.log.WithError(err).Warn("Builder registration attempt failed, retrying in 30s") - m.fireEvent("deposit", fmt.Sprintf("Registration attempt failed: %v, retrying in 30s", err), "warning") - } - - select { - case <-ctx.Done(): - return - case <-m.stopCh: - return - case <-time.After(30 * time.Second): - } - } -} - -// runBalanceMonitor periodically refreshes builder state and tops up balance. -func (m *Manager) runBalanceMonitor(ctx context.Context) { - ticker := time.NewTicker(1 * time.Minute) - defer ticker.Stop() - - for { - select { - case <-ctx.Done(): - return - case <-m.stopCh: - return - case <-ticker.C: - // Always refresh builder state so the UI stays up to date - m.refreshBuilderState() - - // Skip active lifecycle operations when disabled - if !m.enabled.Load() || m.balanceSvc == nil { - continue - } - - needsTopup, amount, err := m.balanceSvc.NeedsTopup(ctx) - if err != nil { - if errors.Is(err, ErrBuilderExited) { - // Expected steady state after an exit; the one-time warning - // event already fired from refreshBuilderState. - m.log.Debug("Builder has exited, skipping top-up check") - } else { - m.log.WithError(err).Warn("Balance check failed") - } - - continue - } - - if needsTopup { - m.fireEvent("balance_topup", fmt.Sprintf("Balance below threshold, topping up %d gwei", amount), "info") - - if err := m.balanceSvc.CheckAndTopup(ctx); err != nil { - if isDepositDeferred(err) { - // Queue fee too high or contract not active — delay this top-up - // to the next monitor tick instead of failing. - m.log.WithError(err).Info("Balance top-up deferred") - m.fireEvent("balance_topup", fmt.Sprintf("Top-up deferred: %v", err), "info") - } else { - m.log.WithError(err).Warn("Balance topup failed") - m.fireEvent("balance_topup", fmt.Sprintf("Balance topup failed: %v", err), "error") - } - } else { - // Immediately reflect the topup in the live balance (no finalization delay) - if tracker := m.GetPaymentTracker(); tracker != nil { - tracker.AddDeposit(amount) - } - - m.fireEvent("balance_topup", fmt.Sprintf("Balance topped up by %d gwei", amount), "success") - } - } - } - } -} - // noticeExitOnce logs and fires the exited-builder warning the first time an // initiated exit is seen on chain: the key can never be reactivated, so top-ups // stay disabled until the pubkey leaves the builder registry. @@ -727,7 +484,7 @@ func (m *Manager) noticeExitOnce(withdrawableEpoch uint64) { // refreshBuilderState updates the cached builder state from the chain service. func (m *Manager) refreshBuilderState() { - pubkey := m.signer.PublicKey() + pubkey := m.registry.Primary().Pubkey() info := m.chainSvc.GetBuilderByPubkey(pubkey) if info == nil { diff --git a/pkg/lifecycle/pending_deposit_sim.go b/pkg/lifecycle/pending_deposit_sim.go index 2af60984..deb80537 100644 --- a/pkg/lifecycle/pending_deposit_sim.go +++ b/pkg/lifecycle/pending_deposit_sim.go @@ -48,18 +48,20 @@ func activationExitChurnLimit( return churn } -// simulateDepositSurvives appends our deposit (amount ourAmount, gwei) to the back of -// the current pending-deposit queue (amounts in gwei) and replays `transitions` epoch -// boundaries of Electra deposit processing. It returns true if our deposit is never -// drained — i.e. it is still queued at the fork boundary. +// simulateDepositSurvives appends our deposits (amounts in gwei) to the back of the +// current pending-deposit queue (amounts in gwei) and replays `transitions` epoch +// boundaries of Electra deposit processing. It returns true if the LAST of our +// deposits is never drained — i.e. the whole batch is still queued at the fork +// boundary. Onboarding several builder keys appends several entries back to back, +// so the batch survives exactly when its last entry does. // // Per transition the processable budget is depositBalanceToConsume (carried) + churnLimit; // entries drain FIFO while the per-epoch count stays under maxPerEpoch and the running // amount stays within the budget. The leftover budget is carried only when the churn // limit was hit, matching the spec. func simulateDepositSurvives( - queue []uint64, - ourAmount, depositBalanceToConsume, churnLimit, maxPerEpoch, transitions uint64, + queue, ourAmounts []uint64, + depositBalanceToConsume, churnLimit, maxPerEpoch, transitions uint64, ) bool { if maxPerEpoch == 0 { // No per-epoch cap means we cannot model draining; treat as undecidable and @@ -67,11 +69,16 @@ func simulateDepositSurvives( return false } - ourIdx := len(queue) + if len(ourAmounts) == 0 { + return true + } + + // ourIdx is the index of the last of our deposits: the one that must survive. + ourIdx := len(queue) + len(ourAmounts) - 1 - sim := make([]uint64, len(queue)+1) - copy(sim, queue) - sim[ourIdx] = ourAmount + sim := make([]uint64, 0, len(queue)+len(ourAmounts)) + sim = append(sim, queue...) + sim = append(sim, ourAmounts...) dbtc := depositBalanceToConsume head := 0 @@ -114,15 +121,16 @@ func simulateDepositSurvives( return head <= ourIdx } -// depositSurvivesUntilFork is the chain-typed wrapper around simulateDepositSurvives. It -// returns true if a deposit of ourAmountGwei submitted at stats.Epoch would still be in -// the pending_deposits queue at forkEpoch. It returns false (be conservative — prefer the -// deposit-just-before-fork path) when the fork is not in the future or the spec lacks the -// deposit-processing parameters needed to model draining. -func depositSurvivesUntilFork( +// depositsSurviveUntilFork is the chain-typed wrapper around simulateDepositSurvives. It +// returns true if a batch of depositCount deposits of ourAmountGwei each, submitted at +// stats.Epoch, would all still be in the pending_deposits queue at forkEpoch. It returns +// false (be conservative — prefer the deposit-just-before-fork path) when the fork is not +// in the future or the spec lacks the deposit-processing parameters needed to model +// draining. +func depositsSurviveUntilFork( stats *chain.EpochStats, spec *chain.ChainSpec, - ourAmountGwei uint64, + ourAmountGwei, depositCount uint64, forkEpoch phase0.Epoch, ) bool { if stats == nil || spec == nil || forkEpoch <= stats.Epoch { @@ -146,11 +154,16 @@ func depositSurvivesUntilFork( amounts[i] = deposit.Amount } + ourAmounts := make([]uint64, depositCount) + for i := range ourAmounts { + ourAmounts[i] = ourAmountGwei + } + transitions := uint64(forkEpoch - stats.Epoch) return simulateDepositSurvives( amounts, - ourAmountGwei, + ourAmounts, stats.DepositBalanceToConsume, churnLimit, spec.MaxPendingDepositsPerEpoch, diff --git a/pkg/lifecycle/pending_deposit_sim_test.go b/pkg/lifecycle/pending_deposit_sim_test.go index 5e05f8c7..5939ab48 100644 --- a/pkg/lifecycle/pending_deposit_sim_test.go +++ b/pkg/lifecycle/pending_deposit_sim_test.go @@ -159,7 +159,51 @@ func TestSimulateDepositSurvives(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - got := simulateDepositSurvives(tt.queue, tt.ourAmount, tt.dbtc, tt.churn, tt.maxPerEpoch, tt.transitions) + got := simulateDepositSurvives(tt.queue, []uint64{tt.ourAmount}, + tt.dbtc, tt.churn, tt.maxPerEpoch, tt.transitions) + assert.Equal(t, tt.want, got) + }) + } +} + +// Onboarding several keys appends several entries: the batch survives to the +// fork exactly when its LAST entry does, so a batch that fits the drain budget +// while a longer one does not must be judged separately. +func TestSimulateDepositSurvives_Batch(t *testing.T) { + const ( + deposit32 = 32 * gwei + maxPer = 16 + ) + + tests := []struct { + name string + ourAmounts []uint64 + churn uint64 + want bool + }{ + { + name: "empty batch trivially survives", + ourAmounts: nil, + churn: deposit32, + want: true, + }, + { + name: "batch smaller than the drain budget is consumed", + ourAmounts: repeatAmounts(2, deposit32), + churn: 4 * deposit32, + want: false, + }, + { + name: "batch larger than the drain budget survives at its tail", + ourAmounts: repeatAmounts(6, deposit32), + churn: 4 * deposit32, + want: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := simulateDepositSurvives(nil, tt.ourAmounts, 0, tt.churn, maxPer, 1) assert.Equal(t, tt.want, got) }) } diff --git a/pkg/lifecycle/reconcile.go b/pkg/lifecycle/reconcile.go new file mode 100644 index 00000000..a86ca28c --- /dev/null +++ b/pkg/lifecycle/reconcile.go @@ -0,0 +1,359 @@ +package lifecycle + +import ( + "context" + "errors" + "fmt" + "math/big" + "time" + + "github.com/sirupsen/logrus" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/wallet" +) + +const ( + // reconcileInterval is the idle cadence of the reconcile loop. Epoch + // transitions and target changes wake it sooner. + reconcileInterval = 30 * time.Second + + // reconcileWorkDelay is the pause between two reconcile passes that each + // performed work. Deposits, exits and top-ups all go through the single + // funding wallet, so the fleet ramps one transaction at a time; this spaces + // those out instead of hammering the deposit queue (whose fee grows with + // its length) in one burst. + reconcileWorkDelay = 2 * time.Second + + // walletFeeHeadroomGwei is the slack required on top of a deposit's stake + // before the funding wallet is considered able to cover it: the queue fee + // plus gas. A rough bound is enough — the transaction itself fails loudly. + walletFeeHeadroomGwei = 100_000_000 // 0.1 ETH +) + +// Reconcile requests an immediate reconcile pass. Called when the target key +// count changes so a UI edit acts within a second rather than at the next tick. +func (m *Manager) Reconcile() { + select { + case m.poke <- struct{}{}: + default: // a pass is already pending + } +} + +// runReconciler is the manager's main loop: it onboards the fleet before the +// Gloas fork, then keeps the managed key count at the configured target and +// every managed key funded. +func (m *Manager) runReconciler(ctx context.Context) { + defer m.wg.Done() + + if !m.waitForEnabled(ctx) { + return + } + + // Onboard the fleet before the Gloas fork via the regular deposit contract + // (no-op when not applicable), so coverage is continuous across the fork. + m.earlyOnboard(ctx) + + // The on-chain builder set is available from the first Gloas EpochStats; the + // fork being active by epoch is not sufficient — the state must be fetched + // and cached before we can tell which keys are already registered (otherwise + // we would read an empty set and deposit for all of them again). + if !m.waitForGloasState(ctx) { + return + } + + epochSub := m.chainSvc.SubscribeEpochStats() + defer epochSub.Unsubscribe() + + ticker := time.NewTicker(reconcileInterval) + defer ticker.Stop() + + for { + worked := m.reconcileOnce(ctx) + + if worked { + // More work may be pending; re-evaluate promptly rather than + // waiting out the idle interval. + select { + case <-ctx.Done(): + return + case <-m.stopCh: + return + case <-time.After(reconcileWorkDelay): + } + + continue + } + + select { + case <-ctx.Done(): + return + case <-m.stopCh: + return + case <-ticker.C: + case <-m.poke: + case _, ok := <-epochSub.Channel(): + if !ok { + return + } + } + } +} + +// reconcileOnce runs a single reconcile pass and reports whether it performed a +// lifecycle transaction. At most one transaction happens per pass: they all go +// through the same funding wallet, and each waits for its receipt. +func (m *Manager) reconcileOnce(ctx context.Context) bool { + m.registry.Refresh() + m.refreshBuilderState() + + if !m.enabled.Load() || m.balanceSvc == nil { + return false + } + + target := m.cfg.BuilderKeys.EffectiveTargetCount() + managed := m.registry.Aggregate().Managed + + switch { + case managed < target && m.cfg.BuilderKeys.AutoDeposit: + return m.depositKeysToTarget(ctx, target, managed) + + case managed > target && m.cfg.BuilderKeys.AutoExit: + return m.exitSurplusKey(ctx, target, managed) + } + + return m.topupNextKey(ctx) +} + +// depositKeysToTarget deposits for the lowest-index keys eligible for one, +// closing the whole gap to the target in a single batch. +func (m *Manager) depositKeysToTarget(ctx context.Context, target, managed uint64) bool { + shortfall := min(target-managed, uint64(wallet.MaxBatchSize)) + + keys := make([]*builder_keys.Key, 0, shortfall) + // The registry returns the lowest eligible key, so each pick must be marked + // in flight before asking for the next one. + for range shortfall { + key := m.registry.NextDepositCandidate() + if key == nil { + break + } + + keys = append(keys, key) + m.registry.MarkDepositPending(key.KeyIndex()) + } + + if len(keys) == 0 { + m.log.WithFields(logrus.Fields{ + "target": target, + "managed": managed, + }).Warn("No builder key available for deposit; raise builder_keys.max_index") + + return false + } + + amount := m.cfg.DepositAmount + + //nolint:gosec // the batch size is bounded by wallet.MaxBatchSize + if !m.walletCanFund(ctx, amount*uint64(len(keys))) { + m.releaseDepositPending(keys) + + return false + } + + m.log.WithFields(logrus.Fields{ + "keys": len(keys), + "target": target, + "managed": managed, + }).Info("Depositing builder keys to reach the target count") + m.fireEvent("deposit", fmt.Sprintf( + "Depositing %d builder key(s) (%d gwei each) — %d of %d keys managed", + len(keys), amount, managed, target), "info") + + if m.depositPendingCallback != nil { + m.depositPendingCallback() + } + + errs, err := m.depositSvc.CreateDeposits(ctx, keys, amount) + if err != nil { + m.releaseDepositPending(keys) + + if isDepositDeferred(err) { + // Queue fee too high or contract not active yet — retry later. This + // is also the automatic backoff when a large ramp pushes the fee up. + m.log.WithError(err).Info("Builder key deposits deferred") + m.fireEvent("deposit", fmt.Sprintf("Deposits deferred: %v", err), "info") + } else { + m.log.WithError(err).Warn("Builder key deposits failed") + m.fireEvent("deposit", fmt.Sprintf("Deposits failed: %v", err), "error") + } + + return false + } + + submitted := 0 + + for i, key := range keys { + if errs[i] != nil { + m.registry.ReleaseDepositPending(key.KeyIndex()) + m.log.WithError(errs[i]).WithField("key", key.String()).Warn("Builder key deposit failed") + m.fireEvent("deposit", fmt.Sprintf( + "Deposit for key #%d failed: %v", key.KeyIndex(), errs[i]), "error") + + continue + } + + m.registry.MarkDepositSubmitted(key.KeyIndex()) + + submitted++ + } + + if submitted == 0 { + return false + } + + m.fireEvent("deposit", fmt.Sprintf( + "%d deposit(s) confirmed, waiting for beacon chain inclusion", submitted), "success") + + return true +} + +// releaseDepositPending clears the in-flight marker of keys whose batch never +// reached the chain, so the next pass can pick them again. +func (m *Manager) releaseDepositPending(keys []*builder_keys.Key) { + for _, key := range keys { + m.registry.ReleaseDepositPending(key.KeyIndex()) + } +} + +// exitSurplusKey exits the highest-index key that can be exited, bringing the +// fleet down to the target. Keys with pending payments are skipped: the beacon +// chain silently ignores their exit requests. +func (m *Manager) exitSurplusKey(ctx context.Context, target, managed uint64) bool { + key := m.registry.NextExitCandidate() + if key == nil { + m.log.WithFields(logrus.Fields{ + "target": target, + "managed": managed, + }).Debug("Above the target key count but no key is exitable yet (pending payments)") + + return false + } + + m.log.WithFields(logrus.Fields{ + "key": key.String(), + "target": target, + "managed": managed, + }).Info("Exiting surplus builder key") + m.fireEvent("exit", fmt.Sprintf( + "Exiting surplus builder key #%d — %d of %d keys managed", + key.KeyIndex(), managed, target), "info") + + if err := m.exitSvc.CreateExit(ctx, key); err != nil { + m.log.WithError(err).WithField("key", key.String()).Warn("Builder key exit failed") + m.fireEvent("exit", fmt.Sprintf("Exit for key #%d failed: %v", key.KeyIndex(), err), "error") + + return false + } + + m.registry.MarkExitSubmitted(key.KeyIndex()) + m.fireEvent("exit", fmt.Sprintf("Builder key #%d exit submitted", key.KeyIndex()), "success") + + return true +} + +// topupNextKey tops up the first managed key whose balance fell below the +// threshold. One key per pass, so a fleet-wide dip ramps instead of queueing a +// transaction per key at once. +func (m *Manager) topupNextKey(ctx context.Context) bool { + for _, key := range m.registry.Keys() { + needsTopup, amount, err := m.balanceSvc.NeedsTopup(key) + if err != nil { + if errors.Is(err, ErrBuilderExited) { + // Expected steady state after an exit; the one-time warning + // already fired from refreshBuilderState. + continue + } + + m.log.WithError(err).WithField("key", key.String()).Warn("Balance check failed") + + continue + } + + if !needsTopup { + continue + } + + if !m.walletCanFund(ctx, amount) { + return false + } + + m.fireEvent("balance_topup", fmt.Sprintf( + "Key #%d below threshold, topping up %d gwei", key.KeyIndex(), amount), "info") + + deposited, err := m.balanceSvc.CheckAndTopup(ctx, key) + if err != nil { + if isDepositDeferred(err) { + // Queue fee too high or contract not active — delay this top-up + // to the next pass instead of failing. + m.log.WithError(err).Info("Balance top-up deferred") + m.fireEvent("balance_topup", fmt.Sprintf("Top-up deferred: %v", err), "info") + } else { + m.log.WithError(err).WithField("key", key.String()).Warn("Balance topup failed") + m.fireEvent("balance_topup", fmt.Sprintf( + "Top-up for key #%d failed: %v", key.KeyIndex(), err), "error") + } + + return false + } + + // Immediately reflect the topup in the live balance (no finalization delay) + if tracker := m.GetPaymentTracker(); tracker != nil { + tracker.AddDeposit(key.KeyIndex(), deposited) + } + + m.fireEvent("balance_topup", fmt.Sprintf( + "Key #%d topped up by %d gwei", key.KeyIndex(), deposited), "success") + + return true + } + + return false +} + +// walletCanFund reports whether the funding wallet still covers a stake of +// amountGwei plus headroom for the queue fee and gas. The whole fleet is funded +// from one wallet, so running it dry is a fleet-wide stall — worth one clear +// event rather than a failing transaction per key. +func (m *Manager) walletCanFund(ctx context.Context, amountGwei uint64) bool { + if m.wallet == nil { + return false + } + + balance, err := m.wallet.GetBalance(ctx) + if err != nil { + m.log.WithError(err).Warn("Failed to read funding wallet balance") + + return false + } + + required := new(big.Int).Add(GweiToWei(amountGwei), GweiToWei(walletFeeHeadroomGwei)) + if balance.Cmp(required) >= 0 { + m.walletUnderfunded.Store(false) + + return true + } + + if m.walletUnderfunded.CompareAndSwap(false, true) { + m.log.WithFields(logrus.Fields{ + "wallet": m.wallet.Address().Hex(), + "balance_wei": balance.String(), + "required_wei": required.String(), + }).Warn("Funding wallet cannot cover the next builder key deposit") + m.fireEvent("deposit", fmt.Sprintf( + "Funding wallet %s cannot cover the next %d gwei deposit; fund it to continue", + m.wallet.Address().Hex(), amountGwei), "error") + } + + return false +} diff --git a/pkg/p2p_bidder/bid_creator.go b/pkg/p2p_bidder/bid_creator.go index 61ccee50..dff51f5f 100644 --- a/pkg/p2p_bidder/bid_creator.go +++ b/pkg/p2p_bidder/bid_creator.go @@ -10,6 +10,7 @@ import ( "github.com/ethpandaops/go-eth2-client/spec/phase0" "github.com/sirupsen/logrus" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/payload_builder" @@ -23,43 +24,45 @@ type bidSubmitter interface { // BidCreator builds ePBS bids via the shared payload_bidder and gossips them // over p2p. It owns the p2p transport and the (caller-computed) bid economics; -// the bid construction and signing live in payload_bidder. +// the bid construction and signing live in payload_bidder. The builder identity +// comes in per bid: which of the managed keys signs is a scheduler decision. type BidCreator struct { - signer *payload_bidder.Signer - clClient bidSubmitter - chainSvc chain.Service - builderIndex uint64 - log logrus.FieldLogger + clClient bidSubmitter + chainSvc chain.Service + log logrus.FieldLogger } // NewBidCreator creates a new bid creator. func NewBidCreator( - signer *payload_bidder.Signer, clClient bidSubmitter, chainSvc chain.Service, - builderIndex uint64, log logrus.FieldLogger, ) *BidCreator { return &BidCreator{ - signer: signer, - clClient: clClient, - chainSvc: chainSvc, - builderIndex: builderIndex, - log: log.WithField("component", "bid-creator"), + clClient: clClient, + chainSvc: chainSvc, + log: log.WithField("component", "bid-creator"), } } // CreateAndSubmitBid builds, signs, and gossips a bid for the given payload at -// the supplied value. The competitive bid value is decided by the scheduler; -// the ePBS p2p path takes no execution payment. The constructed signed bid is -// returned even when the network submission fails so callers can record the -// exact object that was built; it is nil only when construction itself failed. +// the supplied value, from the given builder key. The competitive bid value is +// decided by the scheduler; the ePBS p2p path takes no execution payment. The +// constructed signed bid is returned even when the network submission fails so +// callers can record the exact object that was built; it is nil only when +// construction itself failed. func (c *BidCreator) CreateAndSubmitBid( ctx context.Context, + key *builder_keys.Key, payload *payload_builder.Payload, bidValue uint64, bidTransform string, ) (*eth2all.SignedExecutionPayloadBid, error) { + builderIndex, registered := key.BuilderIndex() + if !registered { + return nil, fmt.Errorf("builder key %s is not registered on chain", key) + } + var feeRecipient bellatrix.ExecutionAddress copy(feeRecipient[:], payload.FeeRecipient[:]) @@ -76,12 +79,13 @@ func (c *BidCreator) CreateAndSubmitBid( } signedBid, err := payload_bidder.BuildSignedBid(ctx, payload, payload_bidder.BidParams{ - BuilderIndex: c.builderIndex, + BuilderIndex: builderIndex, FeeRecipient: feeRecipient, Value: phase0.Gwei(bidValue), ExecutionPayment: 0, Transform: bidTransform, - }, c.signer, forkVersion, c.chainSvc.GetGenesis().GenesisValidatorsRoot) + }, payload_bidder.NewSigner(key.BLSSigner()), forkVersion, + c.chainSvc.GetGenesis().GenesisValidatorsRoot) if err != nil { return nil, fmt.Errorf("failed to build signed bid: %w", err) } @@ -90,7 +94,8 @@ func (c *BidCreator) CreateAndSubmitBid( "slot": payload.Attributes.ProposalSlot, "value": bidValue, "block_hash": fmt.Sprintf("%x", payload.BlockHash[:8]), - "builder_index": c.builderIndex, + "key": key.String(), + "builder_index": builderIndex, "fee_recipient": payload.FeeRecipient.Hex(), "gas_limit": payload.ExecutionPayload.GasLimit, "parent_block_hash": fmt.Sprintf("%x", payload.Attributes.ParentBlockHash[:8]), @@ -111,13 +116,3 @@ func (c *BidCreator) CreateAndSubmitBid( return signedBid, nil } - -// SetBuilderIndex updates the builder index. -func (c *BidCreator) SetBuilderIndex(index uint64) { - c.builderIndex = index -} - -// GetBuilderIndex returns the current builder index. -func (c *BidCreator) GetBuilderIndex() uint64 { - return c.builderIndex -} diff --git a/pkg/p2p_bidder/bid_tracker.go b/pkg/p2p_bidder/bid_tracker.go index 1be82021..c5aec68c 100644 --- a/pkg/p2p_bidder/bid_tracker.go +++ b/pkg/p2p_bidder/bid_tracker.go @@ -5,26 +5,36 @@ import ( "github.com/ethpandaops/go-eth2-client/spec/phase0" "github.com/sirupsen/logrus" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" ) // BidTracker tracks bids observed on the p2p network for competition analysis. type BidTracker struct { - slotBids map[phase0.Slot]*SlotBids - ourBuilderIdx uint64 - mu sync.RWMutex + slotBids map[phase0.Slot]*SlotBids + registry *builder_keys.Registry + mu sync.RWMutex log logrus.FieldLogger } -// NewBidTracker creates a new bid tracker. -func NewBidTracker(ourBuilderIdx uint64, log logrus.FieldLogger) *BidTracker { +// NewBidTracker creates a new bid tracker. The key registry identifies which +// gossiped bids are ours: with several managed keys bidding the same slot, any +// of their builder indices must be excluded from competitor comparisons. +func NewBidTracker(registry *builder_keys.Registry, log logrus.FieldLogger) *BidTracker { return &BidTracker{ - slotBids: make(map[phase0.Slot]*SlotBids, 64), - ourBuilderIdx: ourBuilderIdx, - log: log.WithField("component", "bid-tracker"), + slotBids: make(map[phase0.Slot]*SlotBids, 64), + registry: registry, + log: log.WithField("component", "bid-tracker"), } } +// IsOurs reports whether a gossiped bid's builder index belongs to the managed +// key set. +func (t *BidTracker) IsOurs(builderIndex uint64) bool { + return t.registry != nil && t.registry.ByBuilderIndex(builderIndex) != nil +} + // TrackBid adds a bid to the tracker. func (t *BidTracker) TrackBid(bid *ExecutionPayloadBid, isOurs bool) { t.mu.Lock() @@ -75,12 +85,13 @@ func (t *BidTracker) GetHighestBid(slot phase0.Slot) *TrackedBid { } // GetHighestCompetitorBid returns the highest tracked bid value (gwei) for -// the slot excluding our own builder index, and whether any competitor bid is -// known. Unlike GetHighestBid it can never report our own bid back to us. +// the slot excluding every key of ours, and whether any competitor bid is +// known. Unlike GetHighestBid it can never report one of our own bids back to +// us — which matters once several managed keys bid the same slot. // A non-zero parentHash restricts the comparison to bids committing to the // same execution parent (bids on other forks are not competing). func (t *BidTracker) GetHighestCompetitorBid( - slot phase0.Slot, ourBuilderIndex uint64, parentHash phase0.Hash32, + slot phase0.Slot, parentHash phase0.Hash32, ) (uint64, bool) { t.mu.RLock() defer t.mu.RUnlock() @@ -95,7 +106,7 @@ func (t *BidTracker) GetHighestCompetitorBid( found := false for builderIndex, tracked := range slotBids.Bids { - if builderIndex == ourBuilderIndex || tracked.IsOurs { + if tracked.IsOurs || t.IsOurs(builderIndex) { continue } @@ -144,11 +155,3 @@ func (t *BidTracker) Cleanup(olderThan phase0.Slot) { } } } - -// SetBuilderIndex updates the builder index. -func (t *BidTracker) SetBuilderIndex(index uint64) { - t.mu.Lock() - defer t.mu.Unlock() - - t.ourBuilderIdx = index -} diff --git a/pkg/p2p_bidder/bid_tracker_test.go b/pkg/p2p_bidder/bid_tracker_test.go index fb23f3a5..76287136 100644 --- a/pkg/p2p_bidder/bid_tracker_test.go +++ b/pkg/p2p_bidder/bid_tracker_test.go @@ -9,11 +9,13 @@ import ( "github.com/stretchr/testify/require" ) -func newTestBidTracker(ourBuilderIdx uint64) *BidTracker { +func newTestBidTracker(t *testing.T, ourBuilderIdx uint64) *BidTracker { + t.Helper() + log := logrus.New() log.SetLevel(logrus.PanicLevel) - return NewBidTracker(ourBuilderIdx, log) + return NewBidTracker(newTestKeyRegistry(t, ourBuilderIdx), log) } func newTestBid(slot phase0.Slot, builderIndex, value uint64) *ExecutionPayloadBid { @@ -92,7 +94,7 @@ func TestBidTracker_TrackBidAndGetHighestBid(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - tracker := newTestBidTracker(tt.ourBuilderIdx) + tracker := newTestBidTracker(t, tt.ourBuilderIdx) for _, bid := range tt.bids { tracker.TrackBid(bid, bid.BuilderIndex == tt.ourBuilderIdx) @@ -165,13 +167,13 @@ func TestBidTracker_GetHighestCompetitorBid(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - tracker := newTestBidTracker(tt.ourBuilderIdx) + tracker := newTestBidTracker(t, tt.ourBuilderIdx) for _, bid := range tt.bids { tracker.TrackBid(bid, bid.BuilderIndex == tt.ourBuilderIdx) } - value, ok := tracker.GetHighestCompetitorBid(tt.slot, tt.ourBuilderIdx, phase0.Hash32{}) + value, ok := tracker.GetHighestCompetitorBid(tt.slot, phase0.Hash32{}) assert.Equal(t, tt.wantOK, ok, "competitor bid known") assert.Equal(t, tt.wantValue, value, "highest competitor value") }) @@ -179,7 +181,7 @@ func TestBidTracker_GetHighestCompetitorBid(t *testing.T) { } func TestBidTracker_GetHighestBidUnknownSlot(t *testing.T) { - tracker := newTestBidTracker(1) + tracker := newTestBidTracker(t, 1) assert.Nil(t, tracker.GetHighestBid(42)) assert.Nil(t, tracker.GetOurBid(42)) @@ -187,7 +189,7 @@ func TestBidTracker_GetHighestBidUnknownSlot(t *testing.T) { } func TestBidTracker_GetSlotBids(t *testing.T) { - tracker := newTestBidTracker(1) + tracker := newTestBidTracker(t, 1) tracker.TrackBid(newTestBid(100, 1, 700), true) tracker.TrackBid(newTestBid(100, 2, 900), false) @@ -204,7 +206,7 @@ func TestBidTracker_GetSlotBids(t *testing.T) { } func TestBidTracker_Cleanup(t *testing.T) { - tracker := newTestBidTracker(1) + tracker := newTestBidTracker(t, 1) tracker.TrackBid(newTestBid(100, 2, 500), false) tracker.TrackBid(newTestBid(101, 2, 500), false) @@ -217,18 +219,24 @@ func TestBidTracker_Cleanup(t *testing.T) { assert.NotNil(t, tracker.GetSlotBids(102)) } -func TestBidTracker_SetBuilderIndex(t *testing.T) { - tracker := newTestBidTracker(0) +// With several managed keys bidding the same slot, none of our own bids may be +// reported back as the competition. +func TestBidTracker_ExcludesEveryManagedKey(t *testing.T) { + log := logrus.New() + log.SetLevel(logrus.PanicLevel) - // Simulate late registration: index becomes known after startup. - tracker.SetBuilderIndex(7) + tracker := NewBidTracker(newTestKeyRegistry(t, 7, 8), log) tracker.TrackBid(newTestBid(100, 7, 800), true) - - ourBid := tracker.GetOurBid(100) - require.NotNil(t, ourBid) - assert.Equal(t, uint64(7), ourBid.BuilderIndex) - assert.True(t, ourBid.IsOurs) + // Our second key's bid arrives over gossip, so it is not flagged as ours. + tracker.TrackBid(newTestBid(100, 8, 900), false) + tracker.TrackBid(newTestBid(100, 9, 600), false) + + value, ok := tracker.GetHighestCompetitorBid(100, phase0.Hash32{}) + require.True(t, ok) + assert.Equal(t, uint64(600), value, "our second key must not count as a competitor") + assert.True(t, tracker.IsOurs(8)) + assert.False(t, tracker.IsOurs(9)) } func uint64Ptr(v uint64) *uint64 { diff --git a/pkg/p2p_bidder/keyset_test.go b/pkg/p2p_bidder/keyset_test.go new file mode 100644 index 00000000..5d6f3994 --- /dev/null +++ b/pkg/p2p_bidder/keyset_test.go @@ -0,0 +1,42 @@ +package p2p_bidder + +import ( + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" +) + +// newTestKeyRegistry builds a key registry whose keys are primed as active +// builders at the given on-chain builder indices (key i -> builderIndices[i]). +func newTestKeyRegistry(t *testing.T, builderIndices ...uint64) *builder_keys.Registry { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + cfg := &config.Config{BuilderKeys: config.BuilderKeysConfig{ + TargetCount: uint64(len(builderIndices)), + DiscoveryGap: 1, + MaxIndex: 32, + }} + + registry, err := builder_keys.NewRegistry(cfg, testBuilderPrivkey, log) + require.NoError(t, err) + + for keyIndex, builderIndex := range builderIndices { + _, err := registry.PrimeKeyState(uint64(keyIndex), func(state *builder_keys.State) { + state.Status = builder_keys.StatusActive + state.BuilderIndex = builderIndex + state.HasBuilderIndex = true + state.Balance = 1_000_000_000_000 + state.EffectiveBalance = 1_000_000_000_000 + }) + require.NoError(t, err) + } + + return registry +} diff --git a/pkg/p2p_bidder/registration_test.go b/pkg/p2p_bidder/registration_test.go new file mode 100644 index 00000000..111f047b --- /dev/null +++ b/pkg/p2p_bidder/registration_test.go @@ -0,0 +1,55 @@ +package p2p_bidder + +import ( + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" +) + +// A fleet ramping toward its target deposits continuously, and the scheduler +// gates bidding on the reported registration state. Letting one key's in-flight +// deposit pull that state back to pending suppresses bidding from every +// already-active key for the whole ramp. +func TestSetRegistrationPendingKeepsFleetBidding(t *testing.T) { + newService := func(registry *builder_keys.Registry, state int32) *Service { + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + svc := &Service{registry: registry, log: log} + svc.registrationState.Store(state) + + return svc + } + + t.Run("active fleet keeps bidding while another key deposits", func(t *testing.T) { + registry := newTestKeyRegistry(t, 7) + require.True(t, registry.AnyActive()) + + svc := newService(registry, RegistrationStateRegistered) + svc.SetRegistrationPending() + + require.Equal(t, RegistrationStateRegistered, svc.GetRegistrationState()) + require.True(t, svc.IsRegistered()) + }) + + t.Run("no active key still reports pending", func(t *testing.T) { + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + registry, err := builder_keys.NewRegistry( + &config.Config{BuilderKeys: config.BuilderKeysConfig{ + TargetCount: 1, DiscoveryGap: 1, MaxIndex: 32, + }}, testBuilderPrivkey, log) + require.NoError(t, err) + require.False(t, registry.AnyActive()) + + svc := newService(registry, RegistrationStateUnregistered) + svc.SetRegistrationPending() + + require.Equal(t, RegistrationStatePending, svc.GetRegistrationState()) + }) +} diff --git a/pkg/p2p_bidder/scheduler.go b/pkg/p2p_bidder/scheduler.go index 4f42fe53..43699806 100644 --- a/pkg/p2p_bidder/scheduler.go +++ b/pkg/p2p_bidder/scheduler.go @@ -15,14 +15,26 @@ import ( "github.com/sirupsen/logrus" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/memstore" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" ) +// payloadBidState is one payload's own bidding progress within a slot. +type payloadBidState struct { + // lastBid gates the bid interval. + lastBid time.Time + // count is how often this payload has been bid. It also drives the value + // escalation: every bid lands one increase above the last, since gossip only + // forwards the highest bid seen for a (slot, parent) tuple and bids sharing + // a value cannot all propagate. It is per payload so several candidates do + // not inherit each other's escalation. + count int +} + // SlotState tracks the bidding state for a single slot. type SlotState struct { LastBidTime time.Time @@ -31,17 +43,25 @@ type SlotState struct { BidsClosed bool // Block received, no more bids possible ClosedByRoot phase0.Root // block root that closed bidding (reopens if orphaned) NoPrefsWarnedFor bool // Missing-preferences skip already reported for this slot - // BidPayloads tracks the last bid time per payload: interval throttling - // and single-bid dedup are PER PAYLOAD, so multi-candidate bidding - // ("all") does not starve the other candidates behind one payload's + NoKeyWarnedFor bool // No-ready-key skip already reported for this slot + + // PayloadBids tracks each payload's own bid progress, so multi-candidate + // bidding ("all") does not starve the other candidates behind one payload's // interval gate. - BidPayloads map[phase0.Hash32]time.Time + PayloadBids map[phase0.Hash32]*payloadBidState // BidCandidate is the candidate the auto selection committed to on the // first bid of the slot (sticky unless candidate switching is enabled). BidCandidate chain.CandidateKey BidCandidateSet bool + // UsedKeys holds the keys that already gossiped a bid for this slot. The + // gossip rules ignore every bid a builder makes after its first for a slot, + // so a key is SPENT once one of its bids reached the network — every later + // bid, including an escalated re-bid of the very same payload, has to come + // from a key that has not bid yet or it never propagates. + UsedKeys map[uint64]struct{} + // Frozen is the slot's immutable action-plan snapshot, resolved on the // first scheduler evaluation of the slot (nil until then). Frozen *action_plan.FrozenPlan @@ -55,7 +75,7 @@ type Scheduler struct { bidTracker *BidTracker payloadCache *payload_builder.PayloadCache service *Service // Reference to parent service for firing events - blsSigner *signer.BLSSigner + registry *builder_keys.Registry propPrefsStore *memstore.Store[phase0.Slot, *gloasspec.SignedProposerPreferences] planSvc *action_plan.PlanService // per-slot scheduling/settings authority cfg *config.Config // shared config; mutable settings read live @@ -64,6 +84,16 @@ type Scheduler struct { // Simple state tracking per slot slotStates map[phase0.Slot]*SlotState mu sync.Mutex + + // wg tracks in-flight bid submissions so shutdown can wait for them. They + // deliberately do NOT block the tick that started them. + wg sync.WaitGroup +} + +// Wait blocks until every in-flight bid submission has finished. Called during +// shutdown, after the context is cancelled. +func (s *Scheduler) Wait() { + s.wg.Wait() } // NewScheduler creates a new scheduler. planSvc is the mandatory per-slot @@ -77,7 +107,7 @@ func NewScheduler( bidTracker *BidTracker, payloadCache *payload_builder.PayloadCache, service *Service, - blsSigner *signer.BLSSigner, + registry *builder_keys.Registry, propPrefsStore *memstore.Store[phase0.Slot, *gloasspec.SignedProposerPreferences], planSvc *action_plan.PlanService, cfg *config.Config, @@ -89,7 +119,7 @@ func NewScheduler( bidTracker: bidTracker, payloadCache: payloadCache, service: service, - blsSigner: blsSigner, + registry: registry, propPrefsStore: propPrefsStore, planSvc: planSvc, cfg: cfg, @@ -177,8 +207,9 @@ func (s *Scheduler) ProcessTick(ctx context.Context) { return } - // Don't bid if the builder is not active on-chain. - if !chain.IsBuilderActive(s.chainSvc.GetBuilderByPubkey(s.blsSigner.PublicKey()), uint64(s.chainSvc.GetFinalizedEpoch())) { + // Don't bid unless at least one managed key is active on chain; which key + // each bid is signed with is decided per bid. + if !s.registry.AnyActive() { return } @@ -270,8 +301,23 @@ func (s *Scheduler) checkSlotForBidding(ctx context.Context, slot phase0.Slot, n return } + // Every planned bid is an independent key on an independent payload, so the + // submissions go out concurrently AND detached from this tick: a submission + // takes tens of milliseconds while the scheduler ticks every 10ms, so + // waiting on them here would stall the next step — and the next slot — for + // the duration of the slowest beacon call. All the state a later tick reads + // (the payload's interval claim, each key's claim) is already committed by + // planBidStep before any of these start. for _, payload := range payloads { - s.trySubmitBid(ctx, slot, now, msRelativeToSlot, bidSettings, payload, prefsBypassed) + for _, planned := range s.planBidStep(slot, now, bidSettings, payload) { + s.wg.Add(1) + + go func() { + defer s.wg.Done() + + s.submitBid(ctx, slot, msRelativeToSlot, bidSettings, planned, payload, prefsBypassed) + }() + } } } @@ -350,6 +396,101 @@ func (s *Scheduler) selectBidPayloads( return []*payload_builder.Payload{payload} } +// claimBidKey reserves a builder key for one bid on the given payload, or +// returns nil when the slot has no key left to spend. +// +// The gossip rules ignore every bid a builder makes after its first for a slot, +// so a key is spent as soon as one of its bids reaches the network. Every later +// bid therefore needs a key that has not bid yet — including an escalated re-bid +// of the very same payload, which is exactly what makes the escalation reach the +// network instead of being dropped as already known. +// +// The key is claimed here rather than after the submission because the +// submission is a network call taking tens of milliseconds while the scheduler +// ticks every 10ms: the ticks landing mid-flight would otherwise pick the same +// key again. releaseBidKey hands it back when the submission never made it out. +func (s *Scheduler) claimBidKey( + slot phase0.Slot, bidSettings *action_plan.ResolvedBidSettings, bidValue uint64, +) *builder_keys.Key { + s.mu.Lock() + state := s.getSlotState(slot) + + // Cap how many keys one slot may spend. Unset means every ready key. + if limit := bidSettings.BidKeysPerSlot; limit > 0 && uint64(len(state.UsedKeys)) >= limit { + s.mu.Unlock() + + return nil + } + + spent := make(map[uint64]struct{}, len(state.UsedKeys)) + for keyIndex := range state.UsedKeys { + spent[keyIndex] = struct{}{} + } + + s.mu.Unlock() + + selected := s.registry.SelectForBid(slot, builder_keys.SelectRequest{ + Strategy: bidSettings.KeyStrategy, + RequiredGwei: bidValue, + Count: 1, + Exclude: spent, + }) + + if len(selected) == 0 { + s.mu.Lock() + state = s.getSlotState(slot) + alreadyWarned := state.NoKeyWarnedFor + state.NoKeyWarnedFor = true + s.mu.Unlock() + + if !alreadyWarned { + s.log.WithFields(logrus.Fields{ + "slot": slot, + "spent_keys": len(spent), + "strategy": builder_keys.NormalizedStrategy(bidSettings.KeyStrategy), + }).Info("Every builder key has bid this slot — no further bid can propagate") + } + + return nil + } + + key := selected[0] + + s.mu.Lock() + state = s.getSlotState(slot) + + if state.UsedKeys == nil { + state.UsedKeys = make(map[uint64]struct{}, 4) + } + + // Re-check the cap: a concurrent tick may have claimed the last allowed key + // while this one was selecting. + if limit := bidSettings.BidKeysPerSlot; limit > 0 && uint64(len(state.UsedKeys)) >= limit { + s.mu.Unlock() + + return nil + } + + if _, taken := state.UsedKeys[key.KeyIndex()]; taken { + s.mu.Unlock() + + return nil + } + + state.UsedKeys[key.KeyIndex()] = struct{}{} + s.mu.Unlock() + + return key +} + +// releaseBidKey returns a claimed key to the slot's pool after a submission that +// never reached the network: nothing was gossiped, so the key is not spent. +func (s *Scheduler) releaseBidKey(slot phase0.Slot, key *builder_keys.Key) { + s.mu.Lock() + delete(s.getSlotState(slot).UsedKeys, key.KeyIndex()) + s.mu.Unlock() +} + // preferredPayload picks the built payload matching the chain view's current // head and its payload status, falling back to the cache's primary payload. func (s *Scheduler) preferredPayload(slot phase0.Slot) *payload_builder.Payload { @@ -371,104 +512,180 @@ func (s *Scheduler) preferredPayload(slot phase0.Slot) *payload_builder.Payload return s.payloadCache.Get(slot) } -// trySubmitBid runs the per-payload bid checks (window close, interval, -// single-bid dedup), computes the bid value and submits. -func (s *Scheduler) trySubmitBid( - ctx context.Context, +// plannedBid is one bid decided by a step: which key signs it, at what value. +type plannedBid struct { + key *builder_keys.Key + value uint64 + bidCount int +} + +// planBidStep decides the bids one evaluation of a payload should submit. +// +// A step spends up to BidKeysPerStep keys at once. That is what lets a slot be +// bid from several keys in parallel instead of one key per interval tick: the +// interval ladder is a single-key shape, and with a fleet the useful shapes run +// from "one key per step, escalating" all the way to "every key at once". +// +// All bookkeeping happens under one lock — the payload's interval claim, its +// escalation count and each key's claim — so the ticks that land while the +// submissions are in flight cannot re-decide the same bids. +func (s *Scheduler) planBidStep( slot phase0.Slot, now time.Time, - msRelativeToSlot int64, bidSettings *action_plan.ResolvedBidSettings, payload *payload_builder.Payload, - prefsBypassed bool, -) { +) []plannedBid { s.mu.Lock() + state := s.getSlotState(slot) - // Check if we should bid // - Not if bidding is closed (block already received) - // - Not if we bid too recently (respect interval) - // - Not if we already bid this payload (single bid mode) + // - Not if we bid this payload too recently (respect interval) + // - Not if we already bid it at all (single bid mode) if state.BidsClosed { s.mu.Unlock() - return + return nil } - // Check bid interval (per payload, so "all" mode candidates do not - // throttle each other) - lastBid, alreadyBid := state.BidPayloads[payload.BlockHash] + bidState, alreadyBid := state.PayloadBids[payload.BlockHash] if bidSettings.IntervalMs > 0 { - if alreadyBid && time.Since(lastBid) < time.Duration(bidSettings.IntervalMs)*time.Millisecond { + if alreadyBid && time.Since(bidState.lastBid) < time.Duration(bidSettings.IntervalMs)*time.Millisecond { s.mu.Unlock() - return + return nil } - } else { - // Single bid mode - only bid payloads we have not bid yet. - if alreadyBid { - s.mu.Unlock() - return + } else if alreadyBid { + s.mu.Unlock() + return nil + } + + if state.PayloadBids == nil { + state.PayloadBids = make(map[phase0.Hash32]*payloadBidState, 2) + } + + if bidState == nil { + bidState = &payloadBidState{} + state.PayloadBids[payload.BlockHash] = bidState + } + + bidState.lastBid = now + state.LastBidTime = now + state.LastBidHash = payload.BlockHash + + s.mu.Unlock() + + planned := make([]plannedBid, 0, 4) + + // Every bid gets its own value, one increase above the last. Gossip only + // forwards a bid that is the highest seen for the (slot, parent) tuple, so + // bids sharing a value cannot all propagate however many keys sign them — + // all but the first to arrive are dropped as too low. + for range bidSettings.EffectiveKeysPerStep() { + s.mu.Lock() + escalations := bidState.count + s.mu.Unlock() + + value := s.bidValueFor(slot, bidSettings, payload, escalations) + + key := s.claimBidKey(slot, bidSettings, value) + if key == nil { + break + } + + s.mu.Lock() + state = s.getSlotState(slot) + bidState.count++ + state.BidCount++ + bidCount := state.BidCount + s.mu.Unlock() + + planned = append(planned, plannedBid{key: key, value: value, bidCount: bidCount}) + } + + if len(planned) == 0 { + // Nothing was actually bid, so the payload's interval claim must not + // stand — the next tick has to re-evaluate it. + s.mu.Lock() + if bidState.count == 0 { + delete(s.getSlotState(slot).PayloadBids, payload.BlockHash) + } else { + bidState.lastBid = time.Time{} } + s.mu.Unlock() } - // Calculate bid value (all gwei, overflow-clamped). - // ValueGwei, when set, is an absolute base (per-slot custom value or the - // global bid value override, resolved at freeze time) replacing the - // max(blockValue, BidMinAmount) + BidSubsidy formula. The subsidy pads the - // formula bid so it clears the proposer BN's local-build threshold. - var bidBase uint64 + return planned +} + +// bidValueFor computes the bid value for the payload's next bid. +// +// ValueGwei, when set, is an absolute base (per-slot custom value or the global +// bid value override, resolved at freeze time) replacing the +// max(blockValue, BidMinAmount) + BidSubsidy formula. The subsidy pads the +// formula bid so it clears the proposer BN's local-build threshold. +// +// escalations is how many bids this payload already has, so every bid lands one +// increase above the last and each one can be the new highest for the tuple. +func (s *Scheduler) bidValueFor( + slot phase0.Slot, + bidSettings *action_plan.ResolvedBidSettings, + payload *payload_builder.Payload, + escalations int, +) uint64 { + var value uint64 if bidSettings.ValueGwei != nil { - bidBase = *bidSettings.ValueGwei + value = *bidSettings.ValueGwei } else { - bidBase = max(weiToGweiClamped(payload.BlockValue), bidSettings.MinGwei) - bidBase = s.addGweiClamped(slot, bidBase, bidSettings.SubsidyGwei) + value = max(weiToGweiClamped(payload.BlockValue), bidSettings.MinGwei) + value = s.addGweiClamped(slot, value, bidSettings.SubsidyGwei) } - // Re-bid increase applies in interval mode regardless of the base source. - bidValue := bidBase - if bidSettings.IntervalMs > 0 && state.BidCount > 0 { - increase := s.mulGweiClamped(slot, uint64(state.BidCount), bidSettings.IncreaseGwei) //nolint:gosec // BidCount >= 0 - bidValue = s.addGweiClamped(slot, bidValue, increase) + if escalations > 0 { + increase := s.mulGweiClamped(slot, uint64(escalations), bidSettings.IncreaseGwei) //nolint:gosec // escalations > 0 + value = s.addGweiClamped(slot, value, increase) } - s.mu.Unlock() + return value +} + +// submitBid gossips one planned bid and reports the outcome. +func (s *Scheduler) submitBid( + ctx context.Context, + slot phase0.Slot, + msRelativeToSlot int64, + bidSettings *action_plan.ResolvedBidSettings, + planned plannedBid, + payload *payload_builder.Payload, + prefsBypassed bool, +) { + builderIndex, _ := planned.key.BuilderIndex() s.log.WithFields(logrus.Fields{ "slot": slot, - "bid_value": bidValue, - "bid_count": state.BidCount, + "key": planned.key.String(), + "bid_value": planned.value, + "bid_count": planned.bidCount, "block_hash": fmt.Sprintf("%x", payload.BlockHash[:8]), "ms_into_slot": msRelativeToSlot, }).Info("Creating and submitting bid") // Submit bid, applying the slot's frozen bid transform if any. var bidTransform string - if state.Frozen != nil && state.Frozen.Transforms != nil { - bidTransform = state.Frozen.Transforms.Bid - } - - signedBid, err := s.bidCreator.CreateAndSubmitBid(ctx, payload, bidValue, bidTransform) - // Update state regardless of success - we don't want to spam on failure s.mu.Lock() - state.LastBidTime = now - state.LastBidHash = payload.BlockHash - - if state.BidPayloads == nil { - state.BidPayloads = make(map[phase0.Hash32]time.Time, 2) + if frozen := s.getSlotState(slot).Frozen; frozen != nil && frozen.Transforms != nil { + bidTransform = frozen.Transforms.Bid } - - state.BidPayloads[payload.BlockHash] = now - state.BidCount++ - bidCount := state.BidCount s.mu.Unlock() + signedBid, err := s.bidCreator.CreateAndSubmitBid(ctx, planned.key, payload, planned.value, bidTransform) + event := &BidSubmissionEvent{ Slot: slot, BlockHash: payload.BlockHash, - Value: bidValue, - BidCount: bidCount, + Value: planned.value, + BidCount: planned.bidCount, SignedBid: signedBid, } @@ -476,13 +693,33 @@ func (s *Scheduler) trySubmitBid( event.Warning = "no proposer preferences for slot — bid sent anyway (ignore_missing_prefs)" } - if high, ok := s.bidTracker.GetHighestCompetitorBid(slot, s.bidCreator.GetBuilderIndex(), - payload.Attributes.ParentBlockHash); ok { + if high, ok := s.bidTracker.GetHighestCompetitorBid(slot, payload.Attributes.ParentBlockHash); ok { event.CompetitorHighGwei = &high } if err != nil { - s.log.WithError(err).WithField("slot", slot).Error("Failed to submit bid") + // A rejection means the beacon node saw the bid and turned it down, so + // the key stays spent: retrying it unchanged only repeats the rejection. + // Only a submission that never reached the node hands its key back. + rejected := beacon.BidRejected(err) + if !rejected { + s.releaseBidKey(slot, planned.key) + } + + // Rejections are the normal outcome of bidding a whole fleet: the node + // keeps only the best bid per parent, so every key beyond the first to + // arrive at a given value is turned down. Logging those at error level + // buries the transport failures that actually need attention — the + // per-attempt record on the slot result keeps them all inspectable. + logLevel := logrus.ErrorLevel + if rejected { + logLevel = logrus.DebugLevel + } + + s.log.WithError(err).WithFields(logrus.Fields{ + "slot": slot, + "key": planned.key.String(), + }).Log(logLevel, "Failed to submit bid") // Constructed-but-not-submitted bids carry the signed bid object so // consumers can record exactly what was built. @@ -500,11 +737,13 @@ func (s *Scheduler) trySubmitBid( return } + s.registry.RecordBid(planned.key.KeyIndex()) + // Track the bid s.bidTracker.TrackBid(&ExecutionPayloadBid{ Slot: slot, - BuilderIndex: s.bidCreator.builderIndex, - Value: bidValue, + BuilderIndex: builderIndex, + Value: planned.value, BlockHash: payload.BlockHash, ParentBlockHash: payload.Attributes.ParentBlockHash, ParentBlockRoot: payload.Attributes.ParentBlockRoot, @@ -525,8 +764,9 @@ func (s *Scheduler) trySubmitBid( s.log.WithFields(logrus.Fields{ "slot": slot, - "bid_value": bidValue, - "bid_count": bidCount, + "key": planned.key.String(), + "bid_value": planned.value, + "bid_count": planned.bidCount, "block_hash": payload.BlockHash[:8], }).Info("Bid submitted") } diff --git a/pkg/p2p_bidder/scheduler_test.go b/pkg/p2p_bidder/scheduler_test.go index 7af8cc32..b62df192 100644 --- a/pkg/p2p_bidder/scheduler_test.go +++ b/pkg/p2p_bidder/scheduler_test.go @@ -6,9 +6,12 @@ import ( "errors" "math" "math/big" + "slices" + "sync" "testing" "time" + "github.com/ethpandaops/go-eth2-client/api" eth2all "github.com/ethpandaops/go-eth2-client/spec/all" gloasspec "github.com/ethpandaops/go-eth2-client/spec/gloas" "github.com/ethpandaops/go-eth2-client/spec/phase0" @@ -21,10 +24,8 @@ import ( "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/memstore" - "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/utils" ) @@ -74,13 +75,33 @@ func (s *stubChainService) ActiveForkAtEpoch(phase0.Epoch) version.DataVersion { // mockBidSubmitter records submitted bids and can be told to fail. type mockBidSubmitter struct { + mu sync.Mutex submitted []*eth2all.SignedExecutionPayloadBid err error + inFlight int + + // beforeSubmit, when set, runs while a submission is in flight — it models + // the network call the scheduler's tick can race against. + beforeSubmit func() } func (m *mockBidSubmitter) SubmitExecutionPayloadBid( _ context.Context, bid *eth2all.SignedExecutionPayloadBid, ) error { + m.mu.Lock() + hook := m.beforeSubmit + m.inFlight++ + m.mu.Unlock() + + if hook != nil { + hook() + } + + m.mu.Lock() + defer m.mu.Unlock() + + m.inFlight-- + if m.err != nil { return m.err } @@ -90,6 +111,22 @@ func (m *mockBidSubmitter) SubmitExecutionPayloadBid( return nil } +// count returns how many bids were submitted successfully. +func (m *mockBidSubmitter) count() int { + m.mu.Lock() + defer m.mu.Unlock() + + return len(m.submitted) +} + +// pending returns how many submissions are currently in flight. +func (m *mockBidSubmitter) pending() int { + m.mu.Lock() + defer m.mu.Unlock() + + return m.inFlight +} + // newSchedulerTestPayload builds a minimal Gloas payload sufficient for bid // construction and signing. func newSchedulerTestPayload(slot phase0.Slot, blockValueWei *big.Int) *payload_builder.Payload { @@ -130,6 +167,28 @@ type schedulerHarness struct { events *utils.Subscription[*BidSubmissionEvent] } +// bidTick runs one scheduler evaluation of the test slot and waits for the bid +// submissions it started: they are dispatched detached from the tick so a slow +// beacon call cannot stall the scheduler. +func (h *schedulerHarness) bidTick(msRelativeToSlot int64) { + h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), msRelativeToSlot) + h.scheduler.Wait() +} + +// agePayloadBid moves a payload's last bid past any interval so the next +// evaluation re-bids it. +func (h *schedulerHarness) agePayloadBid(blockHash phase0.Hash32) { + h.scheduler.mu.Lock() + defer h.scheduler.mu.Unlock() + + state := h.scheduler.getSlotState(testSlot) + state.LastBidTime = time.Now().Add(-time.Second) + + if bidState, ok := state.PayloadBids[blockHash]; ok { + bidState.lastBid = time.Now().Add(-time.Second) + } +} + func newSchedulerHarness(t *testing.T, opts harnessOptions) *schedulerHarness { t.Helper() @@ -150,22 +209,24 @@ func newSchedulerHarness(t *testing.T, opts harnessOptions) *schedulerHarness { planSvc := action_plan.NewPlanService(cfg, chainSvc, log) - blsSigner, err := signer.NewBLSSigner(testBuilderPrivkey) - require.NoError(t, err) + // Several keys by default: a key is spent once it has bid a slot, so tests + // that re-bid or bid several candidates need more than one. + registry := newTestKeyRegistry(t, testBuilderIndex, testBuilderIndex+1, + testBuilderIndex+2, testBuilderIndex+3) prefs := memstore.New[phase0.Slot, *gloasspec.SignedProposerPreferences]() - svc, err := NewService(nil, chainSvc, blsSigner, prefs, planSvc, log) + svc, err := NewService(nil, chainSvc, registry, prefs, planSvc, log) require.NoError(t, err) svc.SetEnabled(opts.serviceEnabled) submitter := &mockBidSubmitter{} - bidCreator := NewBidCreator(payload_bidder.NewSigner(blsSigner), submitter, chainSvc, testBuilderIndex, log) - bidTracker := NewBidTracker(testBuilderIndex, log) + bidCreator := NewBidCreator(submitter, chainSvc, log) + bidTracker := NewBidTracker(registry, log) cache := payload_builder.NewPayloadCache(8) scheduler := NewScheduler(chainSvc, bidCreator, bidTracker, - cache, svc, blsSigner, prefs, planSvc, cfg, log) + cache, svc, registry, prefs, planSvc, cfg, log) events := svc.SubscribeBidSubmissions(16, false) t.Cleanup(events.Unsubscribe) @@ -244,7 +305,7 @@ func TestSchedulerSuppressedSlotSkips(t *testing.T) { } h.preparePayload(testSlot, 100, false) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) assert.Empty(t, h.submitter.submitted, "no bid must be submitted for a suppressed slot") assert.Nil(t, h.nextEvent(), "no event must fire for a suppressed slot") @@ -263,7 +324,7 @@ func TestSchedulerForcedSlotBidsWhileDisabled(t *testing.T) { h.applyBidPlan(t, testSlot, `{"mode":"custom"}`) h.preparePayload(testSlot, 100, false) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) require.Len(t, h.submitter.submitted, 1, "forced slot must bid despite global disable") @@ -329,7 +390,7 @@ func TestSchedulerAbsoluteBidValue(t *testing.T) { h.applyBidPlan(t, testSlot, tt.bidPlan) h.preparePayload(testSlot, tt.blockValueGwei, false) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) require.Len(t, h.submitter.submitted, 1) assert.Equal(t, phase0.Gwei(tt.wantValue), h.submitter.submitted[0].Message.Value) @@ -362,7 +423,7 @@ func TestSchedulerSignedNegativeWindow(t *testing.T) { h.applyBidPlan(t, testSlot, bidPlan) h.preparePayload(testSlot, 100, false) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), tt.msRelativeToSlot) + h.bidTick(tt.msRelativeToSlot) if tt.wantBid { assert.Len(t, h.submitter.submitted, 1) @@ -380,7 +441,7 @@ func TestSchedulerPrefsGate(t *testing.T) { }) h.preparePayload(testSlot, 100, true) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) assert.Empty(t, h.submitter.submitted) @@ -392,7 +453,7 @@ func TestSchedulerPrefsGate(t *testing.T) { assert.Empty(t, event.Status, "pre-construction skips carry no submission status") // The skip is reported once per slot, not on every tick. - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1010) + h.bidTick(1010) assert.Nil(t, h.nextEvent()) }) @@ -403,7 +464,7 @@ func TestSchedulerPrefsGate(t *testing.T) { h.applyBidPlan(t, testSlot, `{"mode":"custom","ignore_missing_prefs":true}`) h.preparePayload(testSlot, 100, true) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) require.Len(t, h.submitter.submitted, 1, "bypass must bid without preferences") @@ -424,7 +485,7 @@ func TestSchedulerConstructedEventOnSubmitFailure(t *testing.T) { h.applyBidPlan(t, testSlot, `{"mode":"custom"}`) h.preparePayload(testSlot, 100, false) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) event := h.nextEvent() require.NotNil(t, event) @@ -447,7 +508,7 @@ func TestSchedulerGlobalDefaultsWithoutPlan(t *testing.T) { h.cfg.EPBS.BidSubsidy = 7 h.preparePayload(testSlot, 100, false) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) require.Len(t, h.submitter.submitted, 1) assert.Equal(t, phase0.Gwei(107), h.submitter.submitted[0].Message.Value, @@ -472,7 +533,7 @@ func TestSchedulerIntervalIncreaseAndCompetitorHigh(t *testing.T) { h.scheduler.bidTracker.TrackBid(newTestBid(testSlot, 99, 500), false) h.scheduler.bidTracker.TrackBid(newTestBid(testSlot, testBuilderIndex, 1000), true) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) event := h.nextEvent() require.NotNil(t, event) @@ -481,16 +542,13 @@ func TestSchedulerIntervalIncreaseAndCompetitorHigh(t *testing.T) { assert.Equal(t, uint64(500), *event.CompetitorHighGwei, "our own 1000 gwei bid must be excluded") // Age the last bid past the interval, then re-bid with the increase. - h.scheduler.mu.Lock() - h.scheduler.slotStates[testSlot].LastBidTime = time.Now().Add(-time.Second) - h.scheduler.slotStates[testSlot].BidPayloads[phase0.Hash32{0xbb}] = time.Now().Add(-time.Second) - h.scheduler.mu.Unlock() + h.agePayloadBid(phase0.Hash32{0xbb}) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1100) + h.bidTick(1100) event = h.nextEvent() require.NotNil(t, event) - assert.Equal(t, uint64(110), event.Value, "re-bid adds BidCount * increase") + assert.Equal(t, uint64(110), event.Value, "re-bid adds the payload's bid count * increase") assert.Equal(t, 2, event.BidCount) } @@ -503,18 +561,15 @@ func TestSchedulerOverflowClampsInsteadOfWrapping(t *testing.T) { `{"mode":"custom","bid_value_gwei":18446744073709551615,"bid_interval":10,"bid_increase":10}`) h.preparePayload(testSlot, 100, false) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) event := h.nextEvent() require.NotNil(t, event) assert.Equal(t, uint64(math.MaxUint64), event.Value) // Re-bid: MaxUint64 + 1*10 must clamp, not wrap to 9. - h.scheduler.mu.Lock() - h.scheduler.slotStates[testSlot].LastBidTime = time.Now().Add(-time.Second) - h.scheduler.slotStates[testSlot].BidPayloads[phase0.Hash32{0xbb}] = time.Now().Add(-time.Second) - h.scheduler.mu.Unlock() + h.agePayloadBid(phase0.Hash32{0xbb}) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1100) + h.bidTick(1100) event = h.nextEvent() require.NotNil(t, event) assert.Equal(t, uint64(math.MaxUint64), event.Value, "overflowing re-bid must clamp to MaxUint64") @@ -541,7 +596,9 @@ func TestBidCreatorReturnsBidOnSubmitFailure(t *testing.T) { chainSvc := newStubChainService() - blsSigner, err := signer.NewBLSSigner(testBuilderPrivkey) + registry := newTestKeyRegistry(t, testBuilderIndex) + + key, err := registry.Key(0) require.NoError(t, err) tests := []struct { @@ -562,12 +619,11 @@ func TestBidCreatorReturnsBidOnSubmitFailure(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { submitter := &mockBidSubmitter{err: tt.submitErr} - creator := NewBidCreator(payload_bidder.NewSigner(blsSigner), submitter, - chainSvc, testBuilderIndex, log) + creator := NewBidCreator(submitter, chainSvc, log) payload := newSchedulerTestPayload(testSlot, gweiToWei(100)) - signedBid, err := creator.CreateAndSubmitBid(context.Background(), payload, 42, "") + signedBid, err := creator.CreateAndSubmitBid(context.Background(), key, payload, 42, "") if tt.wantErr { require.Error(t, err) } else { @@ -614,7 +670,7 @@ func TestSchedulerBidCandidateSelection(t *testing.T) { // A forced candidate key bids exactly that payload. h.cfg.EPBS.BidCandidate = "parent_empty" - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) event := h.nextEvent() require.NotNil(t, event) @@ -630,14 +686,14 @@ func TestSchedulerBidCandidateAll(t *testing.T) { h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) h.cfg.EPBS.BidCandidate = "all" - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) require.NotNil(t, h.nextEvent(), "first candidate bid expected") require.NotNil(t, h.nextEvent(), "second candidate bid expected") require.Nil(t, h.nextEvent()) // The single-bid dedup is per payload: a second tick bids nothing new. - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1100) + h.bidTick(1100) require.Nil(t, h.nextEvent()) } @@ -650,7 +706,7 @@ func TestSchedulerAutoCandidateSticky(t *testing.T) { // Auto (no head tracker in the stub): primary payload wins and the // choice sticks on the slot state. - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) event := h.nextEvent() require.NotNil(t, event) @@ -673,7 +729,7 @@ func TestSchedulerBidAllIntervalPerPayload(t *testing.T) { h.cache.Store(newCandidatePayload(testSlot, chain.CandidateParentEmpty, 0x02)) h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + h.bidTick(1000) require.NotNil(t, h.nextEvent(), "first candidate bid expected") require.NotNil(t, h.nextEvent(), @@ -681,6 +737,344 @@ func TestSchedulerBidAllIntervalPerPayload(t *testing.T) { require.Nil(t, h.nextEvent()) // Within the interval neither payload re-bids. - h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1100) + h.bidTick(1100) + require.Nil(t, h.nextEvent()) +} + +// With several managed keys, each built candidate is bid from a DIFFERENT key. +// That is what makes the bids propagate: the gossip rules ignore every bid after +// a builder's first for a slot, so one key can only ever land one of them. +func TestSchedulerAssignsDistinctKeysPerCandidate(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13) + + full := newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01) + empty := newCandidatePayload(testSlot, chain.CandidateParentEmpty, 0x02) + h.cache.Store(full) + h.cache.Store(empty) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + h.cfg.EPBS.BidCandidate = "all" + h.bidTick(1000) + + first := h.nextEvent() + require.NotNil(t, first) + second := h.nextEvent() + require.NotNil(t, second) + require.Nil(t, h.nextEvent()) + + require.NotEqual(t, first.SignedBid.Message.BuilderIndex, second.SignedBid.Message.BuilderIndex, + "each candidate must be bid from a distinct builder key") + + // Both keys are now spent for the slot: the third key is what an escalated + // re-bid has to use, because a spent key's next bid is ignored by gossip. + h.scheduler.mu.Lock() + spent := len(h.scheduler.getSlotState(testSlot).UsedKeys) + h.scheduler.mu.Unlock() + + require.Equal(t, 2, spent) +} + +// An escalated re-bid of the SAME payload must come from a key that has not bid +// this slot yet: the gossip rules ignore a builder's later bids, so re-bidding +// from the same key means the higher value never reaches the network. +func TestSchedulerEscalatesOntoUnusedKeys(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13) + + h.applyBidPlan(t, testSlot, `{"mode":"custom","bid_interval":50,"bid_increase":10}`) + + payload := newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01) + h.cache.Store(payload) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + seen := make(map[uint64]struct{}, 3) + + for range 3 { + h.bidTick(1000) + + event := h.nextEvent() + require.NotNil(t, event) + require.NotNil(t, event.SignedBid) + + builderIndex := uint64(event.SignedBid.Message.BuilderIndex) + _, repeated := seen[builderIndex] + require.False(t, repeated, "builder %d bid the slot twice", builderIndex) + + seen[builderIndex] = struct{}{} + + h.agePayloadBid(payload.BlockHash) + } + + // The fleet is exhausted: a fourth attempt has no key left that could + // propagate, so it bids nothing rather than repeating a spent key. + h.bidTick(1000) + require.Nil(t, h.nextEvent()) +} + +// bid_keys_per_slot caps how many distinct keys bid a slot, so an operator can +// keep the single-bid behaviour even with several candidates built. +func TestSchedulerBidKeysPerSlotCap(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13) + + h.applyBidPlan(t, testSlot, + `{"mode":"custom","bid_candidate":"all","bid_keys_per_slot":1}`) + + h.cache.Store(newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01)) + h.cache.Store(newCandidatePayload(testSlot, chain.CandidateParentEmpty, 0x02)) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + h.bidTick(1000) + + require.NotNil(t, h.nextEvent(), "first candidate bid expected") + require.Nil(t, h.nextEvent(), "the key cap must stop the second candidate") +} + +// The scheduler ticks every 10ms while a bid submission is a network call taking +// tens of milliseconds. A payload's bid slot must therefore be claimed before the +// submission, not after it: otherwise the ticks that land mid-flight pass the +// interval check and gossip the same bid again — which the beacon node rejects as +// already known and which burns the key's one bid for the slot. +func TestSchedulerDoesNotReBidDuringSubmission(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + + // Interval mode, so only the in-flight claim can stop a second submission. + h.applyBidPlan(t, testSlot, `{"mode":"custom","bid_interval":500}`) + + payload := newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01) + h.cache.Store(payload) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + // Block the submission so a concurrent tick runs while it is in flight. + release := make(chan struct{}) + h.submitter.beforeSubmit = func() { <-release } + + var wg sync.WaitGroup + + wg.Add(1) + + go func() { + defer wg.Done() + + h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + }() + + // Give the first submission time to reach the blocked submitter, then tick + // again exactly as the 10ms scheduler would. + require.Eventually(t, func() bool { return h.submitter.pending() > 0 }, time.Second, 5*time.Millisecond) + + h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1010) + h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1020) + + close(release) + wg.Wait() + h.scheduler.Wait() + + assert.Equal(t, 1, h.submitter.count(), "only one bid may be submitted while one is in flight") +} + +// A tick that finds nothing due must not consume a key. The scheduler ticks +// every 10ms while the bid interval is far longer, so claiming a key before the +// interval gate spends the whole fleet within a few ticks and leaves the slot +// unable to bid at all. +func TestSchedulerDoesNotSpendKeysOnIdleTicks(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13) + + h.applyBidPlan(t, testSlot, `{"mode":"custom","bid_interval":5000}`) + + h.cache.Store(newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01)) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + // One bid, then many ticks inside the interval. + h.bidTick(1000) + require.NotNil(t, h.nextEvent()) + + for range 10 { + h.bidTick(1010) + } + + require.Nil(t, h.nextEvent(), "ticks inside the interval must not bid") + + h.scheduler.mu.Lock() + spent := len(h.scheduler.getSlotState(testSlot).UsedKeys) + h.scheduler.mu.Unlock() + + assert.Equal(t, 1, spent, "only the tick that actually bid may spend a key") +} + +// With the per-step count unset a single evaluation spends the whole fleet at +// once — a slot bid from every active key in parallel instead of one key per +// interval tick. Each of those bids carries its own escalated value: gossip +// forwards only the highest bid seen for a (slot, parent) tuple, so bids sharing +// a value cannot all propagate no matter how many keys sign them. +func TestSchedulerBidsFromEveryKeyInOneStep(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13) + + h.applyBidPlan(t, testSlot, + `{"mode":"custom","bid_value_gwei":100,"bid_increase":10,"bid_keys_per_step":0}`) + + h.cache.Store(newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01)) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + h.bidTick(1000) + + values := make([]uint64, 0, 3) + builders := make(map[uint64]struct{}, 3) + + for range 3 { + event := h.nextEvent() + require.NotNil(t, event) + require.NotNil(t, event.SignedBid) + + values = append(values, event.Value) + builders[uint64(event.SignedBid.Message.BuilderIndex)] = struct{}{} + } + + require.Nil(t, h.nextEvent(), "the fleet is spent after one step") + require.Len(t, builders, 3, "each bid must come from a distinct key") + + // Dispatched concurrently, so the report order is arbitrary. + slices.Sort(values) + + assert.Equal(t, []uint64{100, 110, 120}, values, + "every key bids one increment above the previous bid") +} + +// The escalation runs BETWEEN steps: the next step's keys all bid one increment +// above the last step's. +func TestSchedulerEscalatesEveryBid(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13, 14) + + h.applyBidPlan(t, testSlot, + `{"mode":"custom","bid_value_gwei":100,"bid_increase":10,"bid_interval":50,"bid_keys_per_step":2}`) + + payload := newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01) + h.cache.Store(payload) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + h.bidTick(1000) + first := []uint64{h.nextEvent().Value, h.nextEvent().Value} + require.Nil(t, h.nextEvent()) + + h.agePayloadBid(payload.BlockHash) + h.bidTick(1100) + second := []uint64{h.nextEvent().Value, h.nextEvent().Value} + require.Nil(t, h.nextEvent()) + + // The bids of one step are dispatched concurrently, so they can be reported + // in either order; what matters is the set of values. + slices.Sort(first) + slices.Sort(second) + + assert.Equal(t, []uint64{100, 110}, first, "each bid escalates over the previous one") + assert.Equal(t, []uint64{120, 130}, second, "escalation continues across steps") +} + +// The per-step count bounds one evaluation; the rest of the fleet waits for the +// next interval step. +func TestSchedulerBidKeysPerStepBoundsOneStep(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13) + + h.applyBidPlan(t, testSlot, + `{"mode":"custom","bid_interval":50,"bid_keys_per_step":2}`) + + payload := newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01) + h.cache.Store(payload) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + h.bidTick(1000) + + require.NotNil(t, h.nextEvent()) + require.NotNil(t, h.nextEvent()) + require.Nil(t, h.nextEvent(), "the step spends at most two keys") + + h.agePayloadBid(payload.BlockHash) + h.bidTick(1100) + + require.NotNil(t, h.nextEvent(), "the next step spends the last key") require.Nil(t, h.nextEvent()) } + +// A submission in flight must not hold up the scheduler: the tick dispatches it +// and returns, so the next step — and the next slot — is evaluated on time even +// when a beacon call is slow. +func TestSchedulerTickDoesNotWaitForSubmissions(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + + h.cache.Store(newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01)) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + release := make(chan struct{}) + h.submitter.beforeSubmit = func() { <-release } + + done := make(chan struct{}) + + go func() { + defer close(done) + + h.scheduler.checkSlotForBidding(context.Background(), testSlot, time.Now(), 1000) + }() + + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("the tick blocked on an in-flight submission") + } + + // The submission is still in flight; releasing it lets shutdown drain. + require.Eventually(t, func() bool { return h.submitter.pending() > 0 }, + time.Second, 5*time.Millisecond, "the submission must run detached from the tick") + + close(release) + h.scheduler.Wait() +} + +// A bid the beacon node rejected was seen and turned down on merit, so its key +// stays spent — retrying it unchanged only repeats the rejection, and at fleet +// scale that spins through every key. A submission that never reached the node +// still hands its key back. +func TestSchedulerKeepsRejectedKeysSpent(t *testing.T) { + tests := []struct { + name string + err error + wantSpent int + wantResubs bool + }{ + { + name: "rejected by the beacon node", + err: api.Error{Method: "POST", StatusCode: 400, Data: []byte("BID_TOO_LOW")}, + wantSpent: 1, + }, + { + name: "never reached the beacon node", + err: errors.New("connection refused"), + wantSpent: 0, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + h := newSchedulerHarness(t, harnessOptions{epbsEnabled: true, serviceEnabled: true}) + h.scheduler.registry = newTestKeyRegistry(t, 11, 12, 13) + h.submitter.err = test.err + + h.applyBidPlan(t, testSlot, `{"mode":"custom","bid_keys_per_step":1}`) + + h.cache.Store(newCandidatePayload(testSlot, chain.CandidateParentFull, 0x01)) + h.prefs.Put(testSlot, &gloasspec.SignedProposerPreferences{}) + + h.bidTick(1000) + + h.scheduler.mu.Lock() + spent := len(h.scheduler.getSlotState(testSlot).UsedKeys) + h.scheduler.mu.Unlock() + + assert.Equal(t, test.wantSpent, spent) + }) + } +} diff --git a/pkg/p2p_bidder/service.go b/pkg/p2p_bidder/service.go index cf716cb9..7c6c94b3 100644 --- a/pkg/p2p_bidder/service.go +++ b/pkg/p2p_bidder/service.go @@ -16,12 +16,11 @@ import ( "github.com/sirupsen/logrus" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/memstore" - "github.com/ethpandaops/buildoor/pkg/payload_bidder" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/utils" ) @@ -100,8 +99,7 @@ type BidSubmissionEvent struct { // reveals, inclusion tracking, and payment accounting live in the shared // payload_bidder services. type Service struct { - signer *payload_bidder.Signer - blsSigner *signer.BLSSigner + registry *builder_keys.Registry scheduler *Scheduler bidCreator *BidCreator bidTracker *BidTracker @@ -109,8 +107,6 @@ type Service struct { chainSvc chain.Service propPrefsStore *memstore.Store[phase0.Slot, *gloasspec.SignedProposerPreferences] planSvc *action_plan.PlanService - builderIndex uint64 - builderPubkey phase0.BLSPubKey bidSubmissionDispatch *utils.Dispatcher[*BidSubmissionEvent] builderSvc *payload_builder.Service @@ -134,24 +130,19 @@ type Service struct { func NewService( clClient *beacon.Client, chainSvc chain.Service, - blsSigner *signer.BLSSigner, + registry *builder_keys.Registry, propPrefsStore *memstore.Store[phase0.Slot, *gloasspec.SignedProposerPreferences], planSvc *action_plan.PlanService, log logrus.FieldLogger, ) (*Service, error) { serviceLog := log.WithField("component", "p2p-bidder") - // Create the shared payload bidder signer - epbsSigner := payload_bidder.NewSigner(blsSigner) - s := &Service{ - signer: epbsSigner, - blsSigner: blsSigner, + registry: registry, clClient: clClient, chainSvc: chainSvc, propPrefsStore: propPrefsStore, planSvc: planSvc, - builderPubkey: blsSigner.PublicKey(), bidSubmissionDispatch: &utils.Dispatcher[*BidSubmissionEvent]{}, log: serviceLog, } @@ -193,32 +184,29 @@ func (s *Service) Start(ctx context.Context, builderSvc *payload_builder.Service s.ctx, s.cancel = context.WithCancel(ctx) s.builderSvc = builderSvc - // Load builder index from chain service and determine initial registration state + // Determine the initial registration state from the primary key's on-chain entry + pubkey := s.GetBuilderPubkey() + if s.chainSvc.GetCurrentFork() < version.DataVersionGloas { s.log.Info("No builders in beacon state (pre-Gloas), waiting for registration") - s.builderIndex = 0 s.registrationState.Store(RegistrationStateWaitingGloas) - } else if builderInfo := s.chainSvc.GetBuilderByPubkey(s.builderPubkey); builderInfo == nil { + } else if builderInfo := s.chainSvc.GetBuilderByPubkey(pubkey); builderInfo == nil { s.log.Info("Builder not found in beacon state") - s.builderIndex = 0 s.registrationState.Store(RegistrationStateUnregistered) } else { - s.builderIndex = builderInfo.Index s.registrationState.Store(s.computeRegistrationState(builderInfo)) s.log.WithFields(logrus.Fields{ - "builder_index": s.builderIndex, - "builder_pubkey": fmt.Sprintf("%x", s.builderPubkey[:8]), + "builder_index": builderInfo.Index, + "builder_pubkey": fmt.Sprintf("%x", pubkey[:8]), "state": RegistrationStateName(s.registrationState.Load()), }).Info("Builder found in beacon state") } // Initialize components - s.bidTracker = NewBidTracker(s.builderIndex, s.log) + s.bidTracker = NewBidTracker(s.registry, s.log) s.bidCreator = NewBidCreator( - s.signer, s.clClient, s.chainSvc, - s.builderIndex, s.log, ) // The scheduler skips bidding for slots without cached proposer preferences: @@ -229,7 +217,7 @@ func (s *Service) Start(ctx context.Context, builderSvc *payload_builder.Service s.bidTracker, builderSvc.GetPayloadCache(), s, - s.blsSigner, + s.registry, s.propPrefsStore, s.planSvc, builderSvc.GetConfig(), @@ -256,6 +244,12 @@ func (s *Service) Stop() { s.wg.Wait() + // Bid submissions run detached from the scheduler's tick, so they are + // awaited separately. + if s.scheduler != nil { + s.scheduler.Wait() + } + s.log.Info("p2p bidder service stopped") } @@ -334,7 +328,7 @@ func (s *Service) handleHeadEvent(event *beacon.HeadEvent) { // handleBidEvent processes a bid event from the event stream. func (s *Service) handleBidEvent(event *beacon.BidEvent) { - isOurs := event.BuilderIndex == s.builderIndex + isOurs := s.bidTracker.IsOurs(event.BuilderIndex) bid := &ExecutionPayloadBid{ Slot: event.Slot, @@ -376,27 +370,27 @@ func (s *Service) IsActive() bool { // SetRegistrationPending marks the builder as having a deposit in flight. // Called by the lifecycle manager when a deposit is submitted. +// +// A deposit only says something about the key it funds. Once any key of the +// fleet is active the builder can bid, so a deposit for some other key must not +// pull the reported state back to pending — the scheduler gates bidding on it, +// and a fleet ramping toward its target deposits continuously, which would +// otherwise suppress bidding for the whole ramp. func (s *Service) SetRegistrationPending() { + if s.registry != nil && s.registry.AnyActive() { + return + } + s.registrationState.Store(RegistrationStatePending) s.log.Info("Builder deposit submitted, waiting for beacon chain inclusion") } -// SetBuilderRegistered updates the builder index when the lifecycle manager detects registration. -// It sets the appropriate state based on finalization status. -// Called by the lifecycle manager's registration callback. +// SetBuilderRegistered re-evaluates the reported registration state when the +// lifecycle manager detects a registration. The builder index itself lives on +// the key registry; this only drives the status reporting. func (s *Service) SetBuilderRegistered(index uint64) { - s.builderIndex = index - - if s.bidCreator != nil { - s.bidCreator.SetBuilderIndex(index) - } - - if s.bidTracker != nil { - s.bidTracker.SetBuilderIndex(index) - } - // Determine the correct state based on finalization - info := s.chainSvc.GetBuilderByPubkey(s.builderPubkey) + info := s.chainSvc.GetBuilderByPubkey(s.GetBuilderPubkey()) if info != nil { s.registrationState.Store(s.computeRegistrationState(info)) } else { @@ -440,7 +434,7 @@ func (s *Service) RefreshRegistrationState() { return } - info := s.chainSvc.GetBuilderByPubkey(s.builderPubkey) + info := s.chainSvc.GetBuilderByPubkey(s.GetBuilderPubkey()) if info == nil { // Builder not in state — keep current state if pending (deposit submitted), // otherwise mark as unregistered @@ -467,12 +461,21 @@ func (s *Service) GetBidTracker() *BidTracker { return s.bidTracker } -// GetBuilderIndex returns the builder index. +// GetBuilderIndex returns the primary key's on-chain builder index (0 when it is +// not registered). Status reporting only — bids carry the index of the key that +// signed them. func (s *Service) GetBuilderIndex() uint64 { - return s.builderIndex + index, _ := s.registry.Primary().BuilderIndex() + + return index } -// GetBuilderPubkey returns the builder public key. +// GetBuilderPubkey returns the primary key's public key. func (s *Service) GetBuilderPubkey() phase0.BLSPubKey { - return s.builderPubkey + return s.registry.Primary().Pubkey() +} + +// Registry returns the managed builder key set. +func (s *Service) Registry() *builder_keys.Registry { + return s.registry } diff --git a/pkg/payload_bidder/inclusion_tracker.go b/pkg/payload_bidder/inclusion_tracker.go index a1e9daaf..459bd348 100644 --- a/pkg/payload_bidder/inclusion_tracker.go +++ b/pkg/payload_bidder/inclusion_tracker.go @@ -11,6 +11,7 @@ import ( "github.com/ethpandaops/go-eth2-client/spec/version" "github.com/sirupsen/logrus" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" @@ -66,6 +67,7 @@ const ( type wonTracking struct { blockRoot phase0.Root // beacon block that committed to our payload execHash phase0.Hash32 // our payload's execution block hash + keyIndex uint64 // builder key whose bid won the slot (owes the payment) verdict PayloadVerdict // last fired verdict; empty until first resolution } @@ -79,6 +81,7 @@ type InclusionTracker struct { clClient *beacon.Client chainSvc chain.Service builderSvc *payload_builder.Service // payload cache + inclusion stats + registry *builder_keys.Registry // resolves a block's builder index to our key revealSvc *RevealService // optional; nil pre-Gloas payments *PaymentTracker // optional; nil pre-Gloas @@ -104,6 +107,7 @@ func NewInclusionTracker( clClient *beacon.Client, chainSvc chain.Service, builderSvc *payload_builder.Service, + registry *builder_keys.Registry, revealSvc *RevealService, payments *PaymentTracker, log logrus.FieldLogger, @@ -112,6 +116,7 @@ func NewInclusionTracker( clClient: clClient, chainSvc: chainSvc, builderSvc: builderSvc, + registry: registry, revealSvc: revealSvc, payments: payments, trackedWins: make(map[phase0.Slot]*wonTracking, 4), @@ -268,7 +273,7 @@ func (t *InclusionTracker) applyVerdictSideEffects( } if t.payments != nil { - t.payments.SetPaymentDisputed(slot, orphaned) + t.payments.SetPaymentDisputed(win.keyIndex, slot, orphaned) } } @@ -381,25 +386,45 @@ func (t *InclusionTracker) checkForOurPayload(blockInfo *beacon.BlockInfo) { bidValueGwei = new(big.Int).Div(payload.BlockValue, big.NewInt(1_000_000_000)).Uint64() } + // Resolve which of our keys actually won. The block's bid builder index is + // the only ground truth: several managed keys may have bid the very same + // payload, and the winning key both owes the payment and must sign the + // reveal. + winner := t.winningKey(blockInfo) + if winner == nil { + t.log.WithFields(logrus.Fields{ + "slot": blockInfo.Slot, + "builder_index": blockInfo.BuilderIndex, + "block_hash": fmt.Sprintf("%x", blockInfo.ExecutionBlockHash[:8]), + }).Error("Our payload was included under a builder index we do not own — " + + "cannot bind the payment or the reveal to a key") + + return + } + t.log.WithFields(logrus.Fields{ "slot": blockInfo.Slot, + "key": winner.String(), "block_hash": fmt.Sprintf("%x", blockInfo.ExecutionBlockHash[:8]), "bid_value": bidValueGwei, }).Info("Our payload was included in a beacon block!") + t.registry.RecordWin(winner.KeyIndex()) + // Builder payments and reveals only exist post-Gloas; before that the // payload is part of the block itself and nothing is owed or revealed. if t.chainSvc.GetCurrentFork() >= version.DataVersionGloas && t.revealSvc != nil && t.payments != nil { // Record as pending payment (moved to a balance deduction if revealed, // or pending for 2 epochs if not). if bidValueGwei > 0 { - t.payments.RecordWonBid(payload.Attributes.ProposalSlot, bidValueGwei) + t.payments.RecordWonBid(winner.KeyIndex(), payload.Attributes.ProposalSlot, bidValueGwei) } // Request the reveal; the per-slot dedup makes this a no-op for // Builder-API-won slots whose reveal was requested at delivery time. t.revealSvc.RequestReveal(&RevealRequest{ Payload: payload, + Key: winner, BlockInfo: blockInfo, Transport: payload_builder.BidTransportP2P, }) @@ -428,10 +453,23 @@ func (t *InclusionTracker) checkForOurPayload(blockInfo *beacon.BlockInfo) { t.trackedWins[slot] = &wonTracking{ blockRoot: blockInfo.Root, execHash: blockInfo.ExecutionBlockHash, + keyIndex: winner.KeyIndex(), } } } +// winningKey resolves the managed key behind a block's committed bid. From +// Gloas on the block names the builder index, which maps to exactly one of our +// keys; pre-Gloas no builder is named and the primary key stands in (nothing is +// signed or owed on that path anyway). +func (t *InclusionTracker) winningKey(blockInfo *beacon.BlockInfo) *builder_keys.Key { + if !blockInfo.BuilderIndexKnown { + return t.registry.Primary() + } + + return t.registry.ByBuilderIndex(blockInfo.BuilderIndex) +} + // buildWonBlock derives the won-block summary for an included payload (no // storage side effects). The source is derived from the payload's bid // records: any Builder-API bid marks the win as a Builder API delivery, diff --git a/pkg/payload_bidder/inclusion_tracker_test.go b/pkg/payload_bidder/inclusion_tracker_test.go index 67c74d18..af50c6ea 100644 --- a/pkg/payload_bidder/inclusion_tracker_test.go +++ b/pkg/payload_bidder/inclusion_tracker_test.go @@ -18,7 +18,6 @@ import ( "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" ) // newHookedLogger returns a logger capturing entries in a test hook while @@ -107,7 +106,7 @@ func TestInclusionTracker_PayloadVerdicts(t *testing.T) { // head events would). chainSvc.primeHeadTracker(logger, winBlock, competing5) builderSvc := newTestBuilderSvc(chainSvc) - tracker := NewInclusionTracker(nil, chainSvc, builderSvc, nil, nil, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, newTestKeyRegistry(t, 1), nil, nil, logger) statusSub := tracker.SubscribePayloadStatus(4, false) defer statusSub.Unsubscribe() @@ -150,7 +149,7 @@ func TestInclusionTracker_ReorgRevisesVerdict(t *testing.T) { chainSvc := &stubChainService{currentFork: version.DataVersionGloas} chainSvc.primeHeadTracker(logger, winBlock, onChain6, competing5) builderSvc := newTestBuilderSvc(chainSvc) - tracker := NewInclusionTracker(nil, chainSvc, builderSvc, nil, nil, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, newTestKeyRegistry(t, 1), nil, nil, logger) statusSub := tracker.SubscribePayloadStatus(8, false) defer statusSub.Unsubscribe() @@ -206,7 +205,7 @@ func TestInclusionTracker_PaymentStateLogging(t *testing.T) { chainSvc := &stubChainService{currentFork: version.DataVersionGloas} chainSvc.primeHeadTracker(logger, winBlock) builderSvc := newTestBuilderSvc(chainSvc) - tracker := NewInclusionTracker(nil, chainSvc, builderSvc, nil, nil, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, newTestKeyRegistry(t, 1), nil, nil, logger) payload := newTestPayload(5, ourHash, big.NewInt(1_000_000_000_000)) builderSvc.GetPayloadCache().Store(payload) @@ -250,15 +249,14 @@ func TestInclusionTracker_GloasGatingAndInclusion(t *testing.T) { builderSvc := newTestBuilderSvc(chainSvc) payments := NewPaymentTracker(chainSvc, logger) - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) // Not started: RequestReveal just queues on the buffered channel. cfg := &config.Config{} - revealSvc := NewRevealService(cfg, NewSigner(blsSigner), &mockEnvelopePublisher{}, + revealSvc := NewRevealService(cfg, registry, &mockEnvelopePublisher{}, chainSvc, builderSvc, payments, action_plan.NewPlanService(cfg, chainSvc, logger), nil, logger) - tracker := NewInclusionTracker(nil, chainSvc, builderSvc, revealSvc, payments, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, registry, revealSvc, payments, logger) includedSub := tracker.SubscribeIncluded(4, false) defer includedSub.Unsubscribe() @@ -335,7 +333,7 @@ func TestInclusionTracker_BuildWonBlockSource(t *testing.T) { logger, _ := newHookedLogger() chainSvc := &stubChainService{currentFork: version.DataVersionGloas} builderSvc := newTestBuilderSvc(chainSvc) - tracker := NewInclusionTracker(nil, chainSvc, builderSvc, nil, nil, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, newTestKeyRegistry(t, 1), nil, nil, logger) blockHash := phase0.Hash32{0xaa} payload := newTestPayload(5, blockHash, big.NewInt(1_000_000_000_000)) @@ -412,14 +410,14 @@ func TestInclusionTracker_OrphanDisputesPaymentAndUnmarksWin(t *testing.T) { chainSvc.primeHeadTracker(logger, winBlock, competing5, onChain6) builderSvc := newTestBuilderSvc(chainSvc) payments := NewPaymentTracker(chainSvc, logger) - tracker := NewInclusionTracker(nil, chainSvc, builderSvc, nil, payments, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, newTestKeyRegistry(t, 1), nil, payments, logger) payload := newTestPayload(5, ourHash, big.NewInt(1_000_000_000_000)) builderSvc.GetPayloadCache().Store(payload) // Win at slot 5: pending payment recorded... (payments require revealSvc // too in checkForOurPayload, so record directly). - payments.RecordWonBid(5, 1000) + payments.RecordWonBid(0, 5, 1000) tracker.processBlockInfo(winBlock) require.Equal(t, uint64(1000), payments.GetTotalPendingPayments()) @@ -443,3 +441,71 @@ func TestInclusionTracker_OrphanDisputesPaymentAndUnmarksWin(t *testing.T) { assert.Equal(t, uint64(1000), payments.GetTotalPendingPayments(), "re-canonical win must restore the pending payment") } + +// The block's bid builder index — not the payload — decides which key owes the +// payment and signs the reveal: several managed keys can bid the very same +// payload, so the block hash alone cannot identify the winner. +func TestInclusionTracker_BindsWinToTheBiddingKey(t *testing.T) { + logger, _ := newHookedLogger() + chainSvc := &stubChainService{currentFork: version.DataVersionGloas} + builderSvc := newTestBuilderSvc(chainSvc) + payments := NewPaymentTracker(chainSvc, logger) + + // Two managed keys, registered under builder indices 4 and 9. + registry := newTestKeyRegistry(t, 4, 9) + + cfg := &config.Config{} + revealSvc := NewRevealService(cfg, registry, &mockEnvelopePublisher{}, + chainSvc, builderSvc, payments, action_plan.NewPlanService(cfg, chainSvc, logger), nil, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, registry, revealSvc, payments, logger) + + blockHash := phase0.Hash32{0xab} + payload := newTestPayload(7, blockHash, big.NewInt(3_000_000_000_000)) // 3000 gwei + builderSvc.GetPayloadCache().Store(payload) + + tracker.processBlockInfo(&beacon.BlockInfo{ + Slot: 7, + ExecutionBlockHash: blockHash, + BuilderIndex: 9, + BuilderIndexKnown: true, + }) + + // The payment lands on the second key, and the reveal is bound to it. + assert.Equal(t, uint64(0), payments.GetPendingPayments(0)) + assert.Equal(t, uint64(3000), payments.GetPendingPayments(1)) + + require.Len(t, revealSvc.requests, 1) + req := <-revealSvc.requests + require.NotNil(t, req.Key) + assert.Equal(t, uint64(1), req.Key.KeyIndex()) + +} + +// A block naming a builder index we do not own is refused outright rather than +// bound to some other key of ours: signing that reveal would be worthless and +// the payment would be misattributed. +func TestInclusionTracker_RejectsForeignBuilderIndex(t *testing.T) { + logger, _ := newHookedLogger() + chainSvc := &stubChainService{currentFork: version.DataVersionGloas} + builderSvc := newTestBuilderSvc(chainSvc) + payments := NewPaymentTracker(chainSvc, logger) + registry := newTestKeyRegistry(t, 4, 9) + + cfg := &config.Config{} + revealSvc := NewRevealService(cfg, registry, &mockEnvelopePublisher{}, + chainSvc, builderSvc, payments, action_plan.NewPlanService(cfg, chainSvc, logger), nil, logger) + tracker := NewInclusionTracker(nil, chainSvc, builderSvc, registry, revealSvc, payments, logger) + + blockHash := phase0.Hash32{0xcd} + builderSvc.GetPayloadCache().Store(newTestPayload(8, blockHash, big.NewInt(1_000_000_000_000))) + + tracker.processBlockInfo(&beacon.BlockInfo{ + Slot: 8, + ExecutionBlockHash: blockHash, + BuilderIndex: 42, + BuilderIndexKnown: true, + }) + + assert.Empty(t, revealSvc.requests, "a foreign builder index must not schedule a reveal") + assert.Equal(t, uint64(0), payments.GetTotalPendingPayments()) +} diff --git a/pkg/payload_bidder/keyset_test.go b/pkg/payload_bidder/keyset_test.go new file mode 100644 index 00000000..34420bcb --- /dev/null +++ b/pkg/payload_bidder/keyset_test.go @@ -0,0 +1,45 @@ +package payload_bidder + +import ( + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" +) + +// testEntryPrivkey roots the key set used by the tests in this package. +const testEntryPrivkey = "0000000000000000000000000000000000000000000000000000000000000001" + +// newTestKeyRegistry builds a key registry whose keys are primed as active +// builders at the given on-chain builder indices (key i -> builderIndices[i]). +func newTestKeyRegistry(t *testing.T, builderIndices ...uint64) *builder_keys.Registry { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + cfg := &config.Config{BuilderKeys: config.BuilderKeysConfig{ + TargetCount: uint64(len(builderIndices)), + DiscoveryGap: 1, + MaxIndex: 32, + }} + + registry, err := builder_keys.NewRegistry(cfg, testEntryPrivkey, log) + require.NoError(t, err) + + for keyIndex, builderIndex := range builderIndices { + _, err := registry.PrimeKeyState(uint64(keyIndex), func(state *builder_keys.State) { + state.Status = builder_keys.StatusActive + state.BuilderIndex = builderIndex + state.HasBuilderIndex = true + state.Balance = 1_000_000_000_000 + state.EffectiveBalance = 1_000_000_000_000 + }) + require.NoError(t, err) + } + + return registry +} diff --git a/pkg/payload_bidder/payment_tracker.go b/pkg/payload_bidder/payment_tracker.go index 8b3fcb6a..256828bf 100644 --- a/pkg/payload_bidder/payment_tracker.go +++ b/pkg/payload_bidder/payment_tracker.go @@ -11,20 +11,17 @@ import ( // PendingPayment records an unrevealed won bid that may be deducted later. type PendingPayment struct { - Slot phase0.Slot - Epoch phase0.Epoch - Value uint64 // Gwei + KeyIndex uint64 + Slot phase0.Slot + Epoch phase0.Epoch + Value uint64 // Gwei // Disputed marks a payment whose winning block was reorged out: it no // longer counts toward the pending total unless the block returns. Disputed bool } -// PaymentTracker tracks the builder's payment obligations and live balance -// adjustments across both bid flows (p2p and Builder API). Fed by the -// InclusionTracker (won bids) and RevealService (reveals); consumed by the -// lifecycle manager (top-ups) and the WebUI. Passive and thread-safe: it runs -// no goroutine of its own. -type PaymentTracker struct { +// keyPayments is one builder key's payment accounting. +type keyPayments struct { // balanceAdjustment bridges the gap between an operation and the epoch // snapshot reflecting it: positive = deposits/topups, negative = revealed // bid payments. It holds only deltas from the current snapshot epoch; @@ -33,12 +30,24 @@ type PaymentTracker struct { // was anchored to. balanceAdjustment int64 adjustmentEpoch phase0.Epoch - adjustmentMu sync.Mutex - // Pending payments: unrevealed won bids, pending for 2 epochs. - // Only these count as "pending" in the UI and for topup checks. - pendingPayments map[phase0.Slot]*PendingPayment - pendingMu sync.Mutex + // pending holds unrevealed won bids, kept for 2 epochs. Only these count as + // "pending" in the UI and for topup checks. + pending map[phase0.Slot]*PendingPayment +} + +// PaymentTracker tracks payment obligations and live balance adjustments per +// builder key, across both bid flows (p2p and Builder API). Fed by the +// InclusionTracker (won bids) and RevealService (reveals); consumed by the +// lifecycle manager (top-ups), the key registry (effective balances) and the +// WebUI. Passive and thread-safe: it runs no goroutine of its own. +// +// Accounting is per key because a payment is owed by the key whose bid won — +// with several managed keys bidding, charging the wrong one would let an +// underfunded key keep bidding while a funded one looks broke. +type PaymentTracker struct { + mu sync.Mutex + keys map[uint64]*keyPayments chainSvc chain.Service log logrus.FieldLogger @@ -47,121 +56,182 @@ type PaymentTracker struct { // NewPaymentTracker creates a new payment tracker. func NewPaymentTracker(chainSvc chain.Service, log logrus.FieldLogger) *PaymentTracker { return &PaymentTracker{ - pendingPayments: make(map[phase0.Slot]*PendingPayment, 16), - chainSvc: chainSvc, - log: log.WithField("component", "payment-tracker"), + keys: make(map[uint64]*keyPayments, 8), + chainSvc: chainSvc, + log: log.WithField("component", "payment-tracker"), } } -// RecordWonBid records a won bid as a pending payment (unrevealed). -// Called when our bid is included in a beacon block. -// If we later reveal, call MarkRevealed to move it from pending to a balance deduction. -// If we don't reveal, it stays pending for 2 epochs then expires. -func (t *PaymentTracker) RecordWonBid(slot phase0.Slot, value uint64) { - t.pendingMu.Lock() - defer t.pendingMu.Unlock() +// forKey returns the key's accounting, creating it on first use. Callers must +// hold mu. +func (t *PaymentTracker) forKey(keyIndex uint64) *keyPayments { + entry, ok := t.keys[keyIndex] + if !ok { + entry = &keyPayments{pending: make(map[phase0.Slot]*PendingPayment, 8)} + t.keys[keyIndex] = entry + } + + return entry +} +// RecordWonBid records a won bid as a pending payment (unrevealed) against the +// key whose bid was included. If we later reveal, MarkRevealed moves it from +// pending to a balance deduction; otherwise it stays pending for 2 epochs and +// then expires. +func (t *PaymentTracker) RecordWonBid(keyIndex uint64, slot phase0.Slot, value uint64) { epoch := t.chainSvc.GetEpochOfSlot(slot) - t.pendingPayments[slot] = &PendingPayment{ - Slot: slot, - Epoch: epoch, - Value: value, + t.mu.Lock() + + t.forKey(keyIndex).pending[slot] = &PendingPayment{ + KeyIndex: keyIndex, + Slot: slot, + Epoch: epoch, + Value: value, } + t.mu.Unlock() + t.log.WithFields(logrus.Fields{ - "slot": slot, - "epoch": epoch, - "value": value, + "key_index": keyIndex, + "slot": slot, + "epoch": epoch, + "value": value, }).Info("Recorded won bid as pending payment") } -// MarkRevealed moves a won bid from pending to an immediate balance deduction. -// The payment is removed from pending and subtracted from the balance adjustment. -func (t *PaymentTracker) MarkRevealed(slot phase0.Slot) { - t.pendingMu.Lock() - p, ok := t.pendingPayments[slot] +// MarkRevealed moves a won bid from pending to an immediate balance deduction on +// the key that owes it. +func (t *PaymentTracker) MarkRevealed(keyIndex uint64, slot phase0.Slot) { + slotEpoch := t.chainSvc.GetEpochOfSlot(slot) + + t.mu.Lock() + + entry := t.forKey(keyIndex) + + payment, ok := entry.pending[slot] if !ok { - t.pendingMu.Unlock() + t.mu.Unlock() return } - value := p.Value - delete(t.pendingPayments, slot) - t.pendingMu.Unlock() + value := payment.Value + delete(entry.pending, slot) // Deduct from live balance, anchored to this slot's epoch so the // reconciler keeps the delta until the snapshot advances past it. - t.adjustmentMu.Lock() - t.balanceAdjustment -= int64(value) - t.anchorEpochLocked(t.chainSvc.GetEpochOfSlot(slot)) - t.adjustmentMu.Unlock() + entry.balanceAdjustment -= int64(value) + anchorEpoch(entry, slotEpoch) + + t.mu.Unlock() t.log.WithFields(logrus.Fields{ - "slot": slot, - "value": value, + "key_index": keyIndex, + "slot": slot, + "value": value, }).Info("Revealed bid: deducted from live balance") } -// AddDeposit credits a deposit/topup to the live balance adjustment, anchored -// to the current epoch. The credit is reconciled away by ReconcileToEpoch once -// the authoritative snapshot advances past that epoch. -func (t *PaymentTracker) AddDeposit(amount uint64) { - t.adjustmentMu.Lock() - t.balanceAdjustment += int64(amount) - t.anchorEpochLocked(t.chainSvc.GetCurrentEpoch()) - t.adjustmentMu.Unlock() +// AddDeposit credits a deposit/topup to the key's live balance adjustment, +// anchored to the current epoch. The credit is reconciled away by +// ReconcileToEpoch once the authoritative snapshot advances past that epoch. +func (t *PaymentTracker) AddDeposit(keyIndex, amount uint64) { + currentEpoch := t.chainSvc.GetCurrentEpoch() + + t.mu.Lock() + + entry := t.forKey(keyIndex) + entry.balanceAdjustment += int64(amount) + anchorEpoch(entry, currentEpoch) - t.log.WithField("amount", amount).Info("Deposit added to live balance") + t.mu.Unlock() + + t.log.WithFields(logrus.Fields{ + "key_index": keyIndex, + "amount": amount, + }).Info("Deposit added to live balance") } -// anchorEpochLocked advances the adjustment's anchor epoch so the reconciler -// keeps the current delta through opEpoch. Callers must hold adjustmentMu. -func (t *PaymentTracker) anchorEpochLocked(opEpoch phase0.Epoch) { - if opEpoch > t.adjustmentEpoch { - t.adjustmentEpoch = opEpoch +// anchorEpoch advances the adjustment's anchor epoch so the reconciler keeps the +// current delta through opEpoch. +func anchorEpoch(entry *keyPayments, opEpoch phase0.Epoch) { + if opEpoch > entry.adjustmentEpoch { + entry.adjustmentEpoch = opEpoch } } -// GetBalanceAdjustment returns the cumulative balance adjustment since last state refresh. -func (t *PaymentTracker) GetBalanceAdjustment() int64 { - t.adjustmentMu.Lock() - defer t.adjustmentMu.Unlock() +// GetBalanceAdjustment returns the key's cumulative balance adjustment since the +// last state refresh. It implements builder_keys.BalanceAdjuster. +func (t *PaymentTracker) GetBalanceAdjustment(keyIndex uint64) int64 { + t.mu.Lock() + defer t.mu.Unlock() + + entry, ok := t.keys[keyIndex] + if !ok { + return 0 + } - return t.balanceAdjustment + return entry.balanceAdjustment } -// ReconcileToEpoch drops the local adjustment once the authoritative builder -// snapshot advances past the epoch the adjustment is anchored to: the newer -// snapshot already accounts for every reveal/top-up from earlier epochs. +// ReconcileToEpoch drops each key's local adjustment once the authoritative +// builder snapshot advances past the epoch the adjustment is anchored to: the +// newer snapshot already accounts for every reveal/top-up from earlier epochs. // Deltas anchored to the snapshot's own epoch are retained (not yet reflected). // Safe to call every refresh; a no-op until the epoch advances. func (t *PaymentTracker) ReconcileToEpoch(snapshotEpoch phase0.Epoch) { - t.adjustmentMu.Lock() - defer t.adjustmentMu.Unlock() + t.mu.Lock() + defer t.mu.Unlock() - if snapshotEpoch <= t.adjustmentEpoch { - return - } + for _, entry := range t.keys { + if snapshotEpoch <= entry.adjustmentEpoch { + continue + } - t.balanceAdjustment = 0 - t.adjustmentEpoch = snapshotEpoch + entry.balanceAdjustment = 0 + entry.adjustmentEpoch = snapshotEpoch + } } // GetTotalPendingPayments returns the sum of unrevealed won bid obligations -// (disputed payments — winning block reorged out — excluded). +// across all keys (disputed payments — winning block reorged out — excluded). func (t *PaymentTracker) GetTotalPendingPayments() uint64 { - t.pendingMu.Lock() - defer t.pendingMu.Unlock() + t.mu.Lock() + defer t.mu.Unlock() var total uint64 - for _, p := range t.pendingPayments { - if p.Disputed { + for keyIndex := range t.keys { + total += t.pendingForKeyLocked(keyIndex) + } + + return total +} + +// GetPendingPayments returns one key's unrevealed won bid obligations. +func (t *PaymentTracker) GetPendingPayments(keyIndex uint64) uint64 { + t.mu.Lock() + defer t.mu.Unlock() + + return t.pendingForKeyLocked(keyIndex) +} + +// pendingForKeyLocked sums a key's undisputed pending payments. Callers must +// hold mu. +func (t *PaymentTracker) pendingForKeyLocked(keyIndex uint64) uint64 { + entry, ok := t.keys[keyIndex] + if !ok { + return 0 + } + + var total uint64 + + for _, payment := range entry.pending { + if payment.Disputed { continue } - total += p.Value + total += payment.Value } return total @@ -171,22 +241,30 @@ func (t *PaymentTracker) GetTotalPendingPayments() uint64 { // was reorged out. An already settled payment (revealed and deducted) cannot // be rolled back locally — the on-chain payment quorum decides its fate — so // the dispute is only logged in that case. -func (t *PaymentTracker) SetPaymentDisputed(slot phase0.Slot, disputed bool) { - t.pendingMu.Lock() - payment, ok := t.pendingPayments[slot] +func (t *PaymentTracker) SetPaymentDisputed(keyIndex uint64, slot phase0.Slot, disputed bool) { + t.mu.Lock() - if ok { + entry, hasKey := t.keys[keyIndex] + + var payment *PendingPayment + if hasKey { + payment = entry.pending[slot] + } + + if payment != nil { payment.Disputed = disputed } - t.pendingMu.Unlock() + + t.mu.Unlock() logCtx := t.log.WithFields(logrus.Fields{ - "slot": slot, - "disputed": disputed, + "key_index": keyIndex, + "slot": slot, + "disputed": disputed, }) switch { - case ok: + case payment != nil: logCtx.Info("Updated pending payment dispute state (reorg)") case disputed: logCtx.Warn("Winning block reorged out after the payment settled locally — " + @@ -196,19 +274,24 @@ func (t *PaymentTracker) SetPaymentDisputed(slot phase0.Slot, disputed bool) { // PruneExpiredPayments removes pending payments older than 2 epochs. func (t *PaymentTracker) PruneExpiredPayments(currentEpoch phase0.Epoch) { - t.pendingMu.Lock() - defer t.pendingMu.Unlock() + t.mu.Lock() + defer t.mu.Unlock() + + for keyIndex, entry := range t.keys { + for slot, payment := range entry.pending { + if currentEpoch <= payment.Epoch+1 { + continue + } - for slot, p := range t.pendingPayments { - if currentEpoch > p.Epoch+1 { t.log.WithFields(logrus.Fields{ + "key_index": keyIndex, "slot": slot, - "payment_epoch": p.Epoch, + "payment_epoch": payment.Epoch, "current_epoch": currentEpoch, - "value": p.Value, + "value": payment.Value, }).Debug("Pruning expired pending payment") - delete(t.pendingPayments, slot) + delete(entry.pending, slot) } } } diff --git a/pkg/payload_bidder/payment_tracker_test.go b/pkg/payload_bidder/payment_tracker_test.go index 4e1abe75..86286ca6 100644 --- a/pkg/payload_bidder/payment_tracker_test.go +++ b/pkg/payload_bidder/payment_tracker_test.go @@ -18,35 +18,35 @@ func newTestPaymentTracker() *PaymentTracker { func TestPaymentTracker_RecordAndReveal(t *testing.T) { tracker := newTestPaymentTracker() - tracker.RecordWonBid(100, 1000) - tracker.RecordWonBid(101, 500) + tracker.RecordWonBid(0, 100, 1000) + tracker.RecordWonBid(0, 101, 500) assert.Equal(t, uint64(1500), tracker.GetTotalPendingPayments()) - assert.Equal(t, int64(0), tracker.GetBalanceAdjustment()) + assert.Equal(t, int64(0), tracker.GetBalanceAdjustment(0)) // Revealing moves the payment from pending to a balance deduction. - tracker.MarkRevealed(100) + tracker.MarkRevealed(0, 100) assert.Equal(t, uint64(500), tracker.GetTotalPendingPayments()) - assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment()) + assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment(0)) // Revealing an unknown slot is a no-op. - tracker.MarkRevealed(999) + tracker.MarkRevealed(0, 999) assert.Equal(t, uint64(500), tracker.GetTotalPendingPayments()) - assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment()) + assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment(0)) // Re-recording a slot overwrites the pending value. - tracker.RecordWonBid(101, 700) + tracker.RecordWonBid(0, 101, 700) assert.Equal(t, uint64(700), tracker.GetTotalPendingPayments()) } func TestPaymentTracker_DepositsAndDeductions(t *testing.T) { tracker := newTestPaymentTracker() - tracker.AddDeposit(3000) - assert.Equal(t, int64(3000), tracker.GetBalanceAdjustment()) + tracker.AddDeposit(0, 3000) + assert.Equal(t, int64(3000), tracker.GetBalanceAdjustment(0)) - tracker.RecordWonBid(10, 1000) - tracker.MarkRevealed(10) - assert.Equal(t, int64(2000), tracker.GetBalanceAdjustment()) + tracker.RecordWonBid(0, 10, 1000) + tracker.MarkRevealed(0, 10) + assert.Equal(t, int64(2000), tracker.GetBalanceAdjustment(0)) } func TestPaymentTracker_ReconcileToEpoch(t *testing.T) { @@ -58,42 +58,42 @@ func TestPaymentTracker_ReconcileToEpoch(t *testing.T) { tracker := NewPaymentTracker(chainSvc, log) // A top-up credit is anchored to the current epoch (5). - tracker.AddDeposit(50_000_000_000) - assert.Equal(t, int64(50_000_000_000), tracker.GetBalanceAdjustment()) + tracker.AddDeposit(0, 50_000_000_000) + assert.Equal(t, int64(50_000_000_000), tracker.GetBalanceAdjustment(0)) // Reconciling to the same epoch keeps it (the snapshot does not yet // reflect the in-epoch top-up). tracker.ReconcileToEpoch(5) - assert.Equal(t, int64(50_000_000_000), tracker.GetBalanceAdjustment(), + assert.Equal(t, int64(50_000_000_000), tracker.GetBalanceAdjustment(0), "same-epoch reconcile must retain the in-epoch delta") // Once the authoritative snapshot advances, the credit is dropped — it is // now reflected in (or superseded by) the snapshot balance. This is what // prevents an unlanded top-up from inflating the balance forever. tracker.ReconcileToEpoch(6) - assert.Equal(t, int64(0), tracker.GetBalanceAdjustment(), + assert.Equal(t, int64(0), tracker.GetBalanceAdjustment(0), "advancing the snapshot epoch must drop the stale credit") // A reveal deduction in the new epoch anchors to that epoch and survives // same-epoch reconciles. chainSvc.currentEpoch = 6 - tracker.RecordWonBid(6*32, 1000) - tracker.MarkRevealed(6 * 32) - assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment()) + tracker.RecordWonBid(0, 6*32, 1000) + tracker.MarkRevealed(0, 6*32) + assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment(0)) tracker.ReconcileToEpoch(6) - assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment()) + assert.Equal(t, int64(-1000), tracker.GetBalanceAdjustment(0)) tracker.ReconcileToEpoch(7) - assert.Equal(t, int64(0), tracker.GetBalanceAdjustment()) + assert.Equal(t, int64(0), tracker.GetBalanceAdjustment(0)) } func TestPaymentTracker_PruneExpiredPayments(t *testing.T) { tracker := newTestPaymentTracker() // stubChainService maps slot -> epoch via slot/32. - tracker.RecordWonBid(32, 100) // epoch 1 - tracker.RecordWonBid(64, 200) // epoch 2 + tracker.RecordWonBid(0, 32, 100) // epoch 1 + tracker.RecordWonBid(0, 64, 200) // epoch 2 // Payments stay pending through payment epoch + 1. tracker.PruneExpiredPayments(phase0.Epoch(2)) @@ -107,5 +107,34 @@ func TestPaymentTracker_PruneExpiredPayments(t *testing.T) { assert.Equal(t, uint64(0), tracker.GetTotalPendingPayments()) // Pruning never touches the balance adjustment. - assert.Equal(t, int64(0), tracker.GetBalanceAdjustment()) + assert.Equal(t, int64(0), tracker.GetBalanceAdjustment(0)) +} + +// Payments are owed per key: charging the wrong one would let an underfunded key +// keep bidding while a funded one looks broke. +func TestPaymentTracker_PerKeyAccounting(t *testing.T) { + tracker := newTestPaymentTracker() + + tracker.RecordWonBid(0, 100, 1000) + tracker.RecordWonBid(3, 101, 500) + + assert.Equal(t, uint64(1500), tracker.GetTotalPendingPayments()) + assert.Equal(t, uint64(1000), tracker.GetPendingPayments(0)) + assert.Equal(t, uint64(500), tracker.GetPendingPayments(3)) + assert.Equal(t, uint64(0), tracker.GetPendingPayments(7)) + + // Revealing key 3's win deducts from key 3 only. + tracker.MarkRevealed(3, 101) + assert.Equal(t, int64(0), tracker.GetBalanceAdjustment(0)) + assert.Equal(t, int64(-500), tracker.GetBalanceAdjustment(3)) + assert.Equal(t, uint64(1000), tracker.GetTotalPendingPayments()) + + // The same slot can be owed by two keys (both bid it; only one won on + // chain), and disputing one must not touch the other. + tracker.RecordWonBid(5, 200, 700) + tracker.RecordWonBid(6, 200, 800) + tracker.SetPaymentDisputed(5, 200, true) + + assert.Equal(t, uint64(0), tracker.GetPendingPayments(5)) + assert.Equal(t, uint64(800), tracker.GetPendingPayments(6)) } diff --git a/pkg/payload_bidder/reveal_service.go b/pkg/payload_bidder/reveal_service.go index c3faff1c..6fdc98bb 100644 --- a/pkg/payload_bidder/reveal_service.go +++ b/pkg/payload_bidder/reveal_service.go @@ -4,7 +4,6 @@ import ( "context" "fmt" "sync" - "sync/atomic" "time" eth2all "github.com/ethpandaops/go-eth2-client/spec/all" @@ -12,6 +11,7 @@ import ( "github.com/sirupsen/logrus" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/payload_builder" @@ -38,7 +38,11 @@ type headVoteSource interface { // RevealRequest asks the RevealService to publish a payload's envelope at the // configured reveal time. Both flows submit these; the service dedupes by slot. type RevealRequest struct { - Payload *payload_builder.Payload + Payload *payload_builder.Payload + // Key is the builder key whose bid the block committed to. The envelope + // must be signed by it and carry its builder index, or the reveal is + // rejected while the payment is still owed. + Key *builder_keys.Key BlockInfo *beacon.BlockInfo // root + parent root of the committing beacon block Transport payload_builder.BidTransport } @@ -103,15 +107,14 @@ const ( // broadcast validation, deadline bypass) — the plan service is the single // per-slot settings authority. type RevealService struct { - cfg *config.Config // shared live config (reveal settings resolve via planSvc.Freeze) - signer *Signer - publisher envelopePublisher - chainSvc chain.Service - builderSvc *payload_builder.Service // reveal success/failure stats - payments *PaymentTracker // optional; nil-guarded - planSvc *action_plan.PlanService // per-slot scheduling/settings authority; required - votes headVoteSource // optional; nil = vote gates can never open - builderIndex atomic.Uint64 + cfg *config.Config // shared live config (reveal settings resolve via planSvc.Freeze) + registry *builder_keys.Registry + publisher envelopePublisher + chainSvc chain.Service + builderSvc *payload_builder.Service // reveal success/failure stats + payments *PaymentTracker // optional; nil-guarded + planSvc *action_plan.PlanService // per-slot scheduling/settings authority; required + votes headVoteSource // optional; nil = vote gates can never open requests chan *RevealRequest results utils.Dispatcher[*RevealResult] @@ -172,7 +175,7 @@ func (st *revealState) gateSatisfied(now time.Time) bool { // it may be nil (vote gates then never open and expire at the slot end). func NewRevealService( cfg *config.Config, - signer *Signer, + registry *builder_keys.Registry, publisher envelopePublisher, chainSvc chain.Service, builderSvc *payload_builder.Service, @@ -183,7 +186,7 @@ func NewRevealService( ) *RevealService { return &RevealService{ cfg: cfg, - signer: signer, + registry: registry, publisher: publisher, chainSvc: chainSvc, builderSvc: builderSvc, @@ -220,11 +223,6 @@ func (s *RevealService) Stop() { s.log.Info("Reveal service stopped") } -// SetBuilderIndex updates the builder index used when signing envelopes. -func (s *RevealService) SetBuilderIndex(index uint64) { - s.builderIndex.Store(index) -} - // SubscribeResults subscribes to reveal results (consumed by the WebUI). func (s *RevealService) SubscribeResults(capacity int, blocking bool) *utils.Subscription[*RevealResult] { return s.results.Subscribe(capacity, blocking) @@ -589,7 +587,7 @@ func (s *RevealService) processDue(now time.Time) { }) if s.payments != nil { - s.payments.MarkRevealed(slot) + s.payments.MarkRevealed(state.req.Key.KeyIndex(), slot) } s.builderSvc.IncrementRevealsSuccess() @@ -724,12 +722,23 @@ func (s *RevealService) buildEnvelope(req *RevealRequest) ( ctx, cancel := context.WithTimeout(s.ctx, transformTimeout) defer cancel() + revealKey := req.Key + if revealKey == nil { + return nil, nil, nil, fmt.Errorf("reveal request for slot %d has no builder key", slot) + } + + builderIndex, registered := revealKey.BuilderIndex() + if !registered { + return nil, nil, nil, fmt.Errorf("builder key %s is not registered on chain", revealKey) + } + envelope, blobs, proofs, err = BuildSignedEnvelope(ctx, req.Payload, RevealContext{ - BuilderIndex: s.builderIndex.Load(), + BuilderIndex: builderIndex, BeaconBlockRoot: req.BlockInfo.Root, ParentBeaconBlockRoot: req.BlockInfo.ParentRoot, Transform: envelopeTransform, - }, s.signer, forkVersion, s.chainSvc.GetGenesis().GenesisValidatorsRoot) + }, NewSigner(revealKey.BLSSigner()), forkVersion, + s.chainSvc.GetGenesis().GenesisValidatorsRoot) if err != nil { return nil, nil, nil, fmt.Errorf("failed to build signed envelope: %w", err) } diff --git a/pkg/payload_bidder/reveal_service_test.go b/pkg/payload_bidder/reveal_service_test.go index df16036f..4b5c1674 100644 --- a/pkg/payload_bidder/reveal_service_test.go +++ b/pkg/payload_bidder/reveal_service_test.go @@ -17,11 +17,11 @@ import ( "github.com/stretchr/testify/require" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/payload_builder" "github.com/ethpandaops/buildoor/pkg/rpc/beacon" - "github.com/ethpandaops/buildoor/pkg/signer" "github.com/ethpandaops/buildoor/pkg/utils" ) @@ -119,6 +119,7 @@ func (s *stubVoteSource) fire(slot phase0.Slot, root phase0.Root, pct float64) { // revealTestEnv bundles the wiring shared by the reveal service tests. type revealTestEnv struct { + registry *builder_keys.Registry cfg *config.Config chainSvc *stubChainService builderSvc *payload_builder.Service @@ -139,8 +140,7 @@ func newRevealTestEnv(t *testing.T, slotDuration time.Duration, revealTimeMs int log := logrus.New() log.SetLevel(logrus.PanicLevel) - blsSigner, err := signer.NewBLSSigner("0x0000000000000000000000000000000000000000000000000000000000000001") - require.NoError(t, err) + registry := newTestKeyRegistry(t, 1) cfg := &config.Config{} cfg.Reveal = config.DefaultConfig().Reveal @@ -157,11 +157,12 @@ func newRevealTestEnv(t *testing.T, slotDuration time.Duration, revealTimeMs int planSvc := action_plan.NewPlanService(cfg, chainSvc, log) votes := newStubVoteSource() - svc := NewRevealService(cfg, NewSigner(blsSigner), publisher, chainSvc, builderSvc, + svc := NewRevealService(cfg, registry, publisher, chainSvc, builderSvc, payments, planSvc, votes, log) return &revealTestEnv{ cfg: cfg, + registry: registry, chainSvc: chainSvc, builderSvc: builderSvc, payments: payments, @@ -209,10 +210,11 @@ func TestRevealService_RevealsAtDueTime(t *testing.T) { blockRoot := phase0.Root{0x11} payload := newTestPayload(slot, phase0.Hash32{0xab}, big.NewInt(2_000_000_000_000)) // 2000 gwei - env.payments.RecordWonBid(slot, 2000) + env.payments.RecordWonBid(0, slot, 2000) env.svc.RequestReveal(&RevealRequest{ Payload: payload, + Key: env.key(t), BlockInfo: &beacon.BlockInfo{Slot: slot, Root: blockRoot, ParentRoot: phase0.Root{0x22}}, Transport: payload_builder.BidTransportBuilderAPI, }) @@ -242,7 +244,7 @@ func TestRevealService_RevealsAtDueTime(t *testing.T) { // The pending payment moved to a balance deduction. assert.Equal(t, uint64(0), env.payments.GetTotalPendingPayments()) - assert.Equal(t, int64(-2000), env.payments.GetBalanceAdjustment()) + assert.Equal(t, int64(-2000), env.payments.GetBalanceAdjustment(0)) assert.Equal(t, uint64(1), env.builderSvc.GetStats().RevealsSuccess) } @@ -262,11 +264,11 @@ func TestRevealService_DedupsBySlot(t *testing.T) { // Two requests for the same slot from different transports. env.svc.RequestReveal(&RevealRequest{ - Payload: payload, BlockInfo: blockInfo, + Payload: payload, Key: env.key(t), BlockInfo: blockInfo, Transport: payload_builder.BidTransportBuilderAPI, }) env.svc.RequestReveal(&RevealRequest{ - Payload: payload, BlockInfo: blockInfo, + Payload: payload, Key: env.key(t), BlockInfo: blockInfo, Transport: payload_builder.BidTransportP2P, }) @@ -307,10 +309,11 @@ func TestRevealService_RetriesThenGivesUp(t *testing.T) { slot := phase0.Slot(1) payload := newTestPayload(slot, phase0.Hash32{0xab}, big.NewInt(1)) - env.payments.RecordWonBid(slot, 42) + env.payments.RecordWonBid(0, slot, 42) env.svc.RequestReveal(&RevealRequest{ Payload: payload, + Key: env.key(t), BlockInfo: &beacon.BlockInfo{Slot: slot, Root: phase0.Root{0x11}, ParentRoot: phase0.Root{0x22}}, Transport: payload_builder.BidTransportP2P, }) @@ -358,6 +361,7 @@ func TestRevealService_SkipsStaleSlot(t *testing.T) { env.svc.RequestReveal(&RevealRequest{ Payload: payload, + Key: env.key(t), BlockInfo: &beacon.BlockInfo{Slot: slot, Root: phase0.Root{0x11}, ParentRoot: phase0.Root{0x22}}, Transport: payload_builder.BidTransportP2P, }) @@ -391,11 +395,11 @@ func TestRevealService_PlanSuppressedReveal(t *testing.T) { // Two requests for the suppressed slot — the second must be a no-op too. env.svc.RequestReveal(&RevealRequest{ - Payload: payload, BlockInfo: blockInfo, + Payload: payload, Key: env.key(t), BlockInfo: blockInfo, Transport: payload_builder.BidTransportBuilderAPI, }) env.svc.RequestReveal(&RevealRequest{ - Payload: payload, BlockInfo: blockInfo, + Payload: payload, Key: env.key(t), BlockInfo: blockInfo, Transport: payload_builder.BidTransportP2P, }) @@ -441,6 +445,7 @@ func TestRevealService_BypassDeadlinePublishesLate(t *testing.T) { env.svc.RequestReveal(&RevealRequest{ Payload: payload, + Key: env.key(t), BlockInfo: &beacon.BlockInfo{Slot: slot, Root: phase0.Root{0x11}, ParentRoot: phase0.Root{0x22}}, Transport: payload_builder.BidTransportP2P, }) @@ -457,10 +462,25 @@ func TestRevealService_BypassDeadlinePublishesLate(t *testing.T) { require.NotNil(t, payload.Reveal(), "payload must be marked revealed") } -// revealRequest builds a standard request for slot 1. -func revealRequest(slot phase0.Slot, root phase0.Root) *RevealRequest { +// key returns the key set's first key: every reveal must name the key whose bid +// the block committed to, since the envelope is signed with it. +func (e *revealTestEnv) key(t *testing.T) *builder_keys.Key { + t.Helper() + + key, err := e.registry.Key(0) + require.NoError(t, err) + + return key +} + +// revealRequest builds a standard request for slot 1, bound to the key set's +// first key (the envelope must be signed by whichever key's bid won). +func revealRequest(t *testing.T, env *revealTestEnv, slot phase0.Slot, root phase0.Root) *RevealRequest { + t.Helper() + return &RevealRequest{ Payload: newTestPayload(slot, phase0.Hash32{0xab}, big.NewInt(1)), + Key: env.key(t), BlockInfo: &beacon.BlockInfo{Slot: slot, Root: root, ParentRoot: phase0.Root{0x22}}, Transport: payload_builder.BidTransportP2P, } @@ -479,7 +499,7 @@ func TestRevealService_VoteGateRevealsOnThreshold(t *testing.T) { slot := phase0.Slot(1) root := phase0.Root{0x11} - env.svc.RequestReveal(revealRequest(slot, root)) + env.svc.RequestReveal(revealRequest(t, env, slot, root)) // Below the threshold: nothing may publish. time.Sleep(150 * time.Millisecond) @@ -511,7 +531,7 @@ func TestRevealService_VoteGateAlreadyMetAtSchedule(t *testing.T) { // Participation is already above the threshold when the request arrives. env.votes.fire(slot, root, 80) - env.svc.RequestReveal(revealRequest(slot, root)) + env.svc.RequestReveal(revealRequest(t, env, slot, root)) require.Eventually(t, func() bool { return env.publisher.callCount() == 1 @@ -527,7 +547,7 @@ func TestRevealService_VoteOrTimeFallsBackToTimeGate(t *testing.T) { defer env.svc.Stop() slot := phase0.Slot(1) - env.svc.RequestReveal(revealRequest(slot, phase0.Root{0x11})) + env.svc.RequestReveal(revealRequest(t, env, slot, phase0.Root{0x11})) // The threshold is never reached — the time gate publishes at 400ms. time.Sleep(150 * time.Millisecond) @@ -551,7 +571,7 @@ func TestRevealService_VoteAndTimeWaitsForBoth(t *testing.T) { // Vote gate opens right away; the time gate must still hold the reveal. env.votes.fire(slot, root, 90) - env.svc.RequestReveal(revealRequest(slot, root)) + env.svc.RequestReveal(revealRequest(t, env, slot, root)) time.Sleep(250 * time.Millisecond) require.Equal(t, 0, env.publisher.callCount(), "and-mode must wait for the time gate") @@ -572,7 +592,7 @@ func TestRevealService_VoteGateTimeoutWithholds(t *testing.T) { require.NoError(t, env.svc.Start(context.Background())) defer env.svc.Stop() - env.svc.RequestReveal(revealRequest(1, phase0.Root{0x11})) + env.svc.RequestReveal(revealRequest(t, env, 1, phase0.Root{0x11})) res := waitForResult(t, sub.Channel(), 3*time.Second) assert.True(t, res.Skipped) @@ -590,7 +610,7 @@ func TestRevealService_GloballyDisabled(t *testing.T) { require.NoError(t, env.svc.Start(context.Background())) defer env.svc.Stop() - env.svc.RequestReveal(revealRequest(1, phase0.Root{0x11})) + env.svc.RequestReveal(revealRequest(t, env, 1, phase0.Root{0x11})) res := waitForResult(t, sub.Channel(), 2*time.Second) assert.True(t, res.Skipped) @@ -607,7 +627,7 @@ func TestRevealService_PlanCustomForcesDespiteGlobalDisable(t *testing.T) { require.NoError(t, env.svc.Start(context.Background())) defer env.svc.Stop() - env.svc.RequestReveal(revealRequest(1, phase0.Root{0x11})) + env.svc.RequestReveal(revealRequest(t, env, 1, phase0.Root{0x11})) require.Eventually(t, func() bool { return env.publisher.callCount() == 1 @@ -621,7 +641,7 @@ func TestRevealService_BroadcastValidationPassthrough(t *testing.T) { require.NoError(t, env.svc.Start(context.Background())) defer env.svc.Stop() - env.svc.RequestReveal(revealRequest(1, phase0.Root{0x11})) + env.svc.RequestReveal(revealRequest(t, env, 1, phase0.Root{0x11})) require.Eventually(t, func() bool { return env.publisher.callCount() == 1 @@ -642,7 +662,7 @@ func TestRevealService_RetryPolicyFromConfig(t *testing.T) { require.NoError(t, env.svc.Start(context.Background())) defer env.svc.Stop() - env.svc.RequestReveal(revealRequest(1, phase0.Root{0x11})) + env.svc.RequestReveal(revealRequest(t, env, 1, phase0.Root{0x11})) require.Eventually(t, func() bool { return env.publisher.callCount() == 2 diff --git a/pkg/rpc/beacon/api.go b/pkg/rpc/beacon/api.go index 73f8bcb5..eac4c220 100644 --- a/pkg/rpc/beacon/api.go +++ b/pkg/rpc/beacon/api.go @@ -2,7 +2,9 @@ package beacon import ( "context" + "errors" "fmt" + "net/http" eth2client "github.com/ethpandaops/go-eth2-client" "github.com/ethpandaops/go-eth2-client/api" @@ -30,6 +32,27 @@ func (c *Client) SubmitExecutionPayloadBid(ctx context.Context, bid *eth2all.Sig return nil } +// BidRejected reports whether err is the beacon node answering a bid submission +// with a rejection, as opposed to the submission never reaching it. The +// distinction matters because a rejected bid WAS seen — retrying it unchanged +// only repeats the rejection — while a transport failure leaves the bid unsent +// and worth retrying. +func BidRejected(err error) bool { + // go-eth2-client returns *api.Error; accept the value form too so a future + // change of that detail cannot silently turn every rejection into a retry. + var apiErrPtr *api.Error + if errors.As(err, &apiErrPtr) && apiErrPtr != nil { + return apiErrPtr.StatusCode >= http.StatusBadRequest + } + + var apiErr api.Error + if errors.As(err, &apiErr) { + return apiErr.StatusCode >= http.StatusBadRequest + } + + return false +} + // SubmitExecutionPayloadEnvelope submits a signed execution payload envelope using the // stateless flow (SignedExecutionPayloadEnvelopeContents body, Eth-Blob-Data-Included true). // The stateful flow only works when the beacon node cached the blob data from its own block diff --git a/pkg/rpc/beacon/bid_rejected_test.go b/pkg/rpc/beacon/bid_rejected_test.go new file mode 100644 index 00000000..6746c9ab --- /dev/null +++ b/pkg/rpc/beacon/bid_rejected_test.go @@ -0,0 +1,46 @@ +package beacon + +import ( + "errors" + "fmt" + "testing" + + "github.com/ethpandaops/go-eth2-client/api" + "github.com/stretchr/testify/require" +) + +// The distinction decides whether a builder key is spent or retried, and the +// client returns its error by pointer — a value-only check silently turns every +// rejection into a retry, which at fleet scale spins through every key. +func TestBidRejected(t *testing.T) { + tests := []struct { + name string + err error + want bool + }{ + {name: "nil", err: nil}, + {name: "transport failure", err: errors.New("connection refused")}, + { + name: "wrapped pointer rejection", + err: fmt.Errorf("failed to submit bid: %w", &api.Error{StatusCode: 400}), + want: true, + }, + { + name: "wrapped value rejection", + err: fmt.Errorf("failed to submit bid: %w", api.Error{StatusCode: 400}), + want: true, + }, + { + name: "server error is still a rejection we saw", + err: &api.Error{StatusCode: 500}, + want: true, + }, + {name: "success-ish status is not a rejection", err: &api.Error{StatusCode: 202}}, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + require.Equal(t, test.want, BidRejected(test.err)) + }) + } +} diff --git a/pkg/rpc/beacon/client.go b/pkg/rpc/beacon/client.go index 79157669..bfaa28c5 100644 --- a/pkg/rpc/beacon/client.go +++ b/pkg/rpc/beacon/client.go @@ -284,6 +284,13 @@ type BlockInfo struct { // (the payload is embedded in the block); zero from Gloas on, where the // number requires the revealed envelope. ExecutionBlockNumber uint64 + // BuilderIndex is the builder the block's execution payload bid came from + // (Gloas+). It is the ground truth for which of our keys won a slot — the + // payload alone cannot say, since several of our keys may have bid the very + // same payload. BuilderIndexKnown is false pre-Gloas, where the payload is + // embedded in the block and no builder is named. + BuilderIndex uint64 + BuilderIndexKnown bool } // FinalityInfo contains finality checkpoint execution block hashes. @@ -334,6 +341,7 @@ func (c *Client) GetBlockInfo(ctx context.Context, blockID string) (*BlockInfo, } gasLimit, blockNumber := agnosticExecutionGasLimitAndNumber(msg) + builderIndex, builderIndexKnown := agnosticBidBuilderIndex(msg) return &BlockInfo{ Slot: msg.Slot, @@ -344,9 +352,27 @@ func (c *Client) GetBlockInfo(ctx context.Context, blockID string) (*BlockInfo, StateRoot: msg.StateRoot, GasLimit: gasLimit, ExecutionBlockNumber: blockNumber, + BuilderIndex: builderIndex, + BuilderIndexKnown: builderIndexKnown, }, nil } +// agnosticBidBuilderIndex extracts the builder index of the block's committed +// execution payload bid. Only Gloas+ blocks name a builder; before that the +// payload is part of the block itself. +func agnosticBidBuilderIndex(msg *all.BeaconBlock) (index uint64, known bool) { + if msg.Version < version.DataVersionGloas { + return 0, false + } + + bid := msg.Body.SignedExecutionPayloadBid + if bid == nil || bid.Message == nil { + return 0, false + } + + return uint64(bid.Message.BuilderIndex), true +} + // agnosticExecutionGasLimitAndNumber extracts the committed gas limit and (where // knowable) the execution block number from a fork-agnostic beacon block. // Pre-Gloas both come from the embedded payload; from Gloas on the gas limit is diff --git a/pkg/signer/derive.go b/pkg/signer/derive.go index c403f409..02a1f19e 100644 --- a/pkg/signer/derive.go +++ b/pkg/signer/derive.go @@ -8,6 +8,7 @@ import ( "io" "math" "math/big" + "strings" "github.com/tyler-smith/go-bip39" "golang.org/x/crypto/hkdf" @@ -25,20 +26,77 @@ func builderKeyPath(index uint64) []uint64 { return []uint64{12381, 3600, index, 0, 0} } +// ResolveEntryPrivkey resolves the operator-supplied builder key source — either +// a raw hex private key or a BIP-39 mnemonic + account index — to a 32-byte hex +// private key. The mnemonic takes precedence when set (callers are expected to +// enforce mutual exclusivity via config validation). +// +// The result is the entry key of the internal key set: see DeriveInternalKey. +func ResolveEntryPrivkey(privkeyHex, mnemonic string, index uint64) (string, error) { + if mnemonic == "" { + return privkeyHex, nil + } + + derived, err := DeriveBLSPrivkeyHex(mnemonic, index) + if err != nil { + return "", fmt.Errorf("failed to derive builder key from mnemonic: %w", err) + } + + return derived, nil +} + // NewBuilderSigner builds a BLS signer from a builder key source: either a raw hex -// private key or a BIP-39 mnemonic + account index. The mnemonic takes precedence -// when set (callers are expected to enforce mutual exclusivity via config validation). +// private key or a BIP-39 mnemonic + account index. func NewBuilderSigner(privkeyHex, mnemonic string, index uint64) (*BLSSigner, error) { - if mnemonic != "" { - derived, err := DeriveBLSPrivkeyHex(mnemonic, index) - if err != nil { - return nil, fmt.Errorf("failed to derive builder key from mnemonic: %w", err) - } + entry, err := ResolveEntryPrivkey(privkeyHex, mnemonic, index) + if err != nil { + return nil, err + } + + return NewBLSSigner(entry) +} + +// DeriveInternalKey derives the internal builder key at the given index from an +// entry key (the operator-supplied private key, or the one derived from the +// mnemonic via DeriveBLSPrivkeyHex). +// +// Index 0 returns the entry key itself, so the first managed key is always the +// one the operator configured. Higher indices apply one further EIP-2333 +// derivation level, i.e. derive_child_SK(entry_sk, index) — for a mnemonic entry +// key that is the path m/12381/3600/{account}/0/0/{index}, one node deeper than +// any other participant's account path. That depth is what makes the internal +// key set collision-free with other builders sharing the same mnemonic: no +// amount of walking the account index can reach a node below our own key. +// +// It returns the 32-byte secret key as a 64-character lowercase hex string +// (no 0x prefix), matching the format accepted by NewBLSSigner. +func DeriveInternalKey(entryPrivkeyHex string, index uint64) (string, error) { + entryPrivkeyHex = strings.TrimPrefix(entryPrivkeyHex, "0x") + + entryBytes, err := hex.DecodeString(entryPrivkeyHex) + if err != nil { + return "", fmt.Errorf("failed to decode entry private key hex: %w", err) + } + + if len(entryBytes) != 32 { + return "", fmt.Errorf("entry private key must be 32 bytes, got %d", len(entryBytes)) + } + + if index == 0 { + return hex.EncodeToString(entryBytes), nil + } - privkeyHex = derived + // EIP-2333 encodes the child index as I2OSP(index, 4). + if index > math.MaxUint32 { + return "", fmt.Errorf("internal key index %d exceeds maximum %d", index, uint64(math.MaxUint32)) } - return NewBLSSigner(privkeyHex) + sk := deriveChildSK(new(big.Int).SetBytes(entryBytes), index) + + skBytes := make([]byte, 32) + sk.FillBytes(skBytes) + + return hex.EncodeToString(skBytes), nil } // DeriveBLSPrivkeyHex derives a builder BLS private key from a BIP-39 mnemonic diff --git a/pkg/signer/derive_internal_test.go b/pkg/signer/derive_internal_test.go new file mode 100644 index 00000000..ca8e0605 --- /dev/null +++ b/pkg/signer/derive_internal_test.go @@ -0,0 +1,93 @@ +package signer + +import ( + "math" + "testing" + + "github.com/stretchr/testify/require" +) + +const testEntryKey = "3f2b8e1c9d4a6f70b5c8e2a1d7943f6058ac2be91d3f5074a6b8c2e1d9f30475" + +func TestDeriveInternalKeyIndexZeroIsEntryKey(t *testing.T) { + derived, err := DeriveInternalKey(testEntryKey, 0) + require.NoError(t, err) + require.Equal(t, testEntryKey, derived) + + // The 0x prefix is accepted and normalised away. + prefixed, err := DeriveInternalKey("0x"+testEntryKey, 0) + require.NoError(t, err) + require.Equal(t, testEntryKey, prefixed) +} + +func TestDeriveInternalKeyIsDeterministicAndDistinct(t *testing.T) { + seen := map[string]uint64{testEntryKey: 0} + + for index := uint64(1); index <= 8; index++ { + derived, err := DeriveInternalKey(testEntryKey, index) + require.NoError(t, err) + require.Len(t, derived, 64) + + again, err := DeriveInternalKey(testEntryKey, index) + require.NoError(t, err) + require.Equal(t, derived, again, "derivation must be deterministic") + + previous, collides := seen[derived] + require.False(t, collides, "index %d collides with index %d", index, previous) + + seen[derived] = index + } +} + +// Internal keys sit one node below the entry key, so they can never be reached +// by another builder walking the mnemonic account index — the property that +// makes the key set safe to use in shared test setups. +func TestDeriveInternalKeyDoesNotCollideWithNeighbourAccounts(t *testing.T) { + const mnemonic = "test test test test test test test test test test test junk" + + ours, err := DeriveBLSPrivkeyHex(mnemonic, 7) + require.NoError(t, err) + + neighbours := map[string]uint64{} + + for account := range uint64(24) { + key, err := DeriveBLSPrivkeyHex(mnemonic, account) + require.NoError(t, err) + + neighbours[key] = account + } + + for index := uint64(1); index <= 16; index++ { + derived, err := DeriveInternalKey(ours, index) + require.NoError(t, err) + + account, collides := neighbours[derived] + require.False(t, collides, "internal key %d collides with account %d", index, account) + } +} + +func TestDeriveInternalKeyRejectsBadInput(t *testing.T) { + _, err := DeriveInternalKey("not-hex", 1) + require.Error(t, err) + + _, err = DeriveInternalKey("aabb", 1) + require.ErrorContains(t, err, "32 bytes") + + _, err = DeriveInternalKey(testEntryKey, math.MaxUint32+1) + require.ErrorContains(t, err, "exceeds maximum") +} + +func TestResolveEntryPrivkey(t *testing.T) { + const mnemonic = "test test test test test test test test test test test junk" + + raw, err := ResolveEntryPrivkey(testEntryKey, "", 0) + require.NoError(t, err) + require.Equal(t, testEntryKey, raw) + + fromMnemonic, err := ResolveEntryPrivkey("", mnemonic, 3) + require.NoError(t, err) + + expected, err := DeriveBLSPrivkeyHex(mnemonic, 3) + require.NoError(t, err) + require.Equal(t, expected, fromMnemonic) +} diff --git a/pkg/slot_results/tracker.go b/pkg/slot_results/tracker.go index 3dc9f271..7181f32f 100644 --- a/pkg/slot_results/tracker.go +++ b/pkg/slot_results/tracker.go @@ -13,6 +13,7 @@ import ( "github.com/sirupsen/logrus" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" "github.com/ethpandaops/buildoor/pkg/db" @@ -50,6 +51,9 @@ type Tracker struct { epbsSvc *p2p_bidder.Service // may be nil pre-Gloas revealSvc *payload_bidder.RevealService inclusionTracker *payload_bidder.InclusionTracker + // registry resolves a recorded bid's on-chain builder index back to the + // managed key that signed it. May be nil. + registry *builder_keys.Registry store *memstore.Store[phase0.Slot, *SlotResult] artifacts *ArtifactStore @@ -75,7 +79,8 @@ type Tracker struct { func NewTracker(cfg *config.Config, chainSvc chain.Service, stateDB *db.Database, planSvc *action_plan.PlanService, builderSvc *payload_builder.Service, epbsSvc *p2p_bidder.Service, revealSvc *payload_bidder.RevealService, - inclusionTracker *payload_bidder.InclusionTracker, log logrus.FieldLogger) *Tracker { + inclusionTracker *payload_bidder.InclusionTracker, registry *builder_keys.Registry, + log logrus.FieldLogger) *Tracker { trackerLog := log.WithField("component", "slot-results") return &Tracker{ @@ -87,6 +92,7 @@ func NewTracker(cfg *config.Config, chainSvc chain.Service, stateDB *db.Database epbsSvc: epbsSvc, revealSvc: revealSvc, inclusionTracker: inclusionTracker, + registry: registry, store: memstore.New[phase0.Slot, *SlotResult](), artifacts: NewArtifactStore(stateDB, trackerLog), lastFired: make(map[phase0.Slot]time.Time, 8), @@ -520,8 +526,9 @@ func attributesSnapshot(attrs *beacon.PayloadAttributesEvent) *AttributesSnapsho } // fillBidDetail copies the bid message properties onto the attempt (blob -// commitments aggregated to a count). -func fillBidDetail(attempt *BidAttempt, signedBid *eth2all.SignedExecutionPayloadBid) { +// commitments aggregated to a count) and resolves which of our builder keys +// signed it. +func (t *Tracker) fillBidDetail(attempt *BidAttempt, signedBid *eth2all.SignedExecutionPayloadBid) { if signedBid == nil || signedBid.Message == nil { return } @@ -535,6 +542,13 @@ func fillBidDetail(attempt *BidAttempt, signedBid *eth2all.SignedExecutionPayloa attempt.GasLimit = bid.GasLimit attempt.BuilderIndex = uint64(bid.BuilderIndex) attempt.NumBlobCommitments = len(bid.BlobKZGCommitments) + + if t.registry != nil { + if key := t.registry.ByBuilderIndex(uint64(bid.BuilderIndex)); key != nil { + keyIndex := key.KeyIndex() + attempt.KeyIndex = &keyIndex + } + } } func (t *Tracker) handleBuildStarted(event *payload_builder.PayloadBuildStartedEvent) { @@ -609,7 +623,7 @@ func (t *Tracker) handleBidSubmission(event *p2p_bidder.BidSubmissionEvent) { At: time.Now(), } - fillBidDetail(&attempt, event.SignedBid) + t.fillBidDetail(&attempt, event.SignedBid) switch event.Status { case p2p_bidder.BidStatusSubmitted: @@ -753,7 +767,7 @@ func (t *Tracker) RecordBuilderAPIBid(slot phase0.Slot, forkName string, signedB // The epbs dialect serves Gloas+ bids with the full message; the legacy // dialect's versioned header bids stay aggregate-only. if signed, ok := signedBid.(*eth2all.SignedExecutionPayloadBid); ok { - fillBidDetail(&attempt, signed) + t.fillBidDetail(&attempt, signed) } marshaler, isMarshaler := signedBid.(sszMarshaler) diff --git a/pkg/slot_results/tracker_test.go b/pkg/slot_results/tracker_test.go index 88b60df8..e7e7c025 100644 --- a/pkg/slot_results/tracker_test.go +++ b/pkg/slot_results/tracker_test.go @@ -93,7 +93,7 @@ func newTrackerTestEnv(t *testing.T, withDB bool) *trackerTestEnv { t.Cleanup(func() { _ = stateDB.Close() }) - tracker := NewTracker(cfg, chainSvc, stateDB, planSvc, nil, nil, nil, nil, log) + tracker := NewTracker(cfg, chainSvc, stateDB, planSvc, nil, nil, nil, nil, nil, log) return &trackerTestEnv{ cfg: cfg, @@ -375,7 +375,7 @@ func TestPersistenceRoundTrip(t *testing.T) { log := logrus.New() log.SetOutput(io.Discard) - fresh := NewTracker(env.cfg, env.chainSvc, env.stateDB, env.planSvc, nil, nil, nil, nil, log) + fresh := NewTracker(env.cfg, env.chainSvc, env.stateDB, env.planSvc, nil, nil, nil, nil, nil, log) fresh.SetPersistence(t.Context(), env.stateDB) defer fresh.store.Stop() diff --git a/pkg/slot_results/types.go b/pkg/slot_results/types.go index 308356d0..29541848 100644 --- a/pkg/slot_results/types.go +++ b/pkg/slot_results/types.go @@ -163,14 +163,19 @@ type BidAttempt struct { // Full bid message properties (Gloas+ bids; blob commitments aggregated // to a count). Empty for legacy Builder API bids and pre-construction // failures. - BlockHash string `json:"block_hash,omitempty"` - ParentBlockHash string `json:"parent_block_hash,omitempty"` - ParentBlockRoot string `json:"parent_block_root,omitempty"` - PrevRandao string `json:"prev_randao,omitempty"` - FeeRecipient string `json:"fee_recipient,omitempty"` - GasLimit uint64 `json:"gas_limit,omitempty"` - BuilderIndex uint64 `json:"builder_index,omitempty"` - NumBlobCommitments int `json:"num_blob_commitments,omitempty"` + BlockHash string `json:"block_hash,omitempty"` + ParentBlockHash string `json:"parent_block_hash,omitempty"` + ParentBlockRoot string `json:"parent_block_root,omitempty"` + PrevRandao string `json:"prev_randao,omitempty"` + FeeRecipient string `json:"fee_recipient,omitempty"` + GasLimit uint64 `json:"gas_limit,omitempty"` + BuilderIndex uint64 `json:"builder_index,omitempty"` + // KeyIndex is the internal builder key the bid was signed with, resolved + // from BuilderIndex at record time. Recorded because a builder index is + // reused by other builders after an exit, so the mapping is only reliable + // while the bid is being made. + KeyIndex *uint64 `json:"key_index,omitempty"` + NumBlobCommitments int `json:"num_blob_commitments,omitempty"` // ArtifactIndex references the slot's stored 'bid' SSZ artifact; nil when // no artifact exists (suppressed, pre-construction failure, or capture diff --git a/pkg/wallet/batch.go b/pkg/wallet/batch.go new file mode 100644 index 00000000..40445363 --- /dev/null +++ b/pkg/wallet/batch.go @@ -0,0 +1,214 @@ +package wallet + +import ( + "context" + "fmt" + "math/big" + "sync" + "time" + + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + "github.com/sirupsen/logrus" +) + +// MaxBatchSize bounds how many transactions one batch submits at a time. Larger +// requests are split into consecutive chunks: a long run of pre-signed nonces is +// more exposed to another instance taking one of them, and every displaced +// nonce stalls the ones behind it until it is refilled. +const MaxBatchSize = 10 + +// TxRequest is one transaction of a batch submission. +type TxRequest struct { + To common.Address + Value *big.Int + Data []byte + GasLimit uint64 +} + +// TxResult is the outcome of one batch request. Index is the request's position +// in the input slice, so results stay matchable regardless of completion order. +type TxResult struct { + Index int + Receipt *types.Receipt + Err error +} + +// batchTx is one in-flight transaction of a batch round. +type batchTx struct { + index int // position in the caller's request slice + tx *types.Transaction + // sendErr, if any, only matters as a tie-breaker once node state shows the + // transaction never entered the pool. + sendErr error +} + +// SendBatchAndConfirm submits several transactions from this wallet at once and +// confirms each independently. It returns one result per request, in request +// order; a failure of one request never fails the others. +// +// Batching matters because every lifecycle transaction is serialized on the same +// funding key: depositing a fleet of builder keys one confirmed transaction at a +// time takes a block per key. The batch signs consecutive nonces in one go and +// sends them together, so they land in the same block or two. +// +// Each transaction is still resolved on its own, because sharing the funding key +// with other buildoor instances means any single nonce can be taken by a foreign +// transaction. A displaced transaction is rebuilt with a fresh nonce and retried +// in a later round, while the ones that landed are left alone. +func (w *Wallet) SendBatchAndConfirm( + ctx context.Context, requests []TxRequest, timeout time.Duration, +) []TxResult { + if len(requests) == 0 { + return nil + } + + w.txMu.Lock() + defer w.txMu.Unlock() + + results := make([]TxResult, len(requests)) + for i := range results { + results[i] = TxResult{Index: i} + } + + pending := make([]int, 0, len(requests)) + for i := range requests { + pending = append(pending, i) + } + + for attempt := 1; attempt <= w.maxAttempts && len(pending) > 0; attempt++ { + retry := make([]int, 0, len(pending)) + + for _, chunk := range chunkIndices(pending, MaxBatchSize) { + retry = append(retry, w.runBatchRound(ctx, requests, chunk, results, timeout)...) + } + + pending = retry + + if len(pending) > 0 { + w.log.WithFields(logrus.Fields{ + "pending": len(pending), + "attempt": attempt, + }).Warn("Retrying batch transactions whose nonce slot was taken") + + if err := sleepCtx(ctx, w.conflictBackoff); err != nil { + break + } + } + } + + // Anything still unresolved exhausted its attempts. + for _, index := range pending { + results[index].Err = fmt.Errorf("transaction failed after %d attempts", w.maxAttempts) + } + + return results +} + +// runBatchRound signs and sends one chunk with consecutive nonces, resolves every +// transaction concurrently, writes terminal outcomes into results and returns the +// request indices that must be retried with a fresh nonce. +func (w *Wallet) runBatchRound( + ctx context.Context, + requests []TxRequest, + chunk []int, + results []TxResult, + timeout time.Duration, +) []int { + baseNonce, err := w.nextNonce(ctx) + if err != nil { + for _, index := range chunk { + results[index].Err = fmt.Errorf("failed to read nonce: %w", err) + } + + return nil + } + + sent := make([]batchTx, 0, len(chunk)) + + for offset, index := range chunk { + req := requests[index] + + //nolint:gosec // offset is bounded by MaxBatchSize + tx, err := w.buildAndSignWithNonce(ctx, req, baseNonce+uint64(offset)) + if err != nil { + results[index].Err = err + continue + } + + sendErr := w.rpcClient.SendTransaction(ctx, tx) + if sendErr == nil { + w.log.WithFields(logrus.Fields{ + "hash": tx.Hash().Hex(), + "nonce": tx.Nonce(), + "to": req.To.Hex(), + "value": req.Value.String(), + }).Info("Transaction sent") + } + + sent = append(sent, batchTx{index: index, tx: tx, sendErr: sendErr}) + } + + outcomes := make([]txOutcome, len(sent)) + receipts := make([]*types.Receipt, len(sent)) + errs := make([]error, len(sent)) + + var wg sync.WaitGroup + + for i := range sent { + wg.Add(1) + + go func(i int) { + defer wg.Done() + + entry := sent[i] + receipts[i], outcomes[i], errs[i] = w.resolve( + ctx, entry.tx.Hash(), entry.tx.Nonce(), entry.sendErr, timeout) + }(i) + } + + wg.Wait() + + retry := make([]int, 0, len(sent)) + + for i, entry := range sent { + switch outcomes[i] { + case outcomeIncluded: + results[entry.index].Receipt = receipts[i] + case outcomeReverted: + results[entry.index].Receipt = receipts[i] + results[entry.index].Err = errs[i] + case outcomeRetry: + retry = append(retry, entry.index) + case outcomeFailed, outcomePending: + results[entry.index].Err = fmt.Errorf( + "send transaction (nonce %d): %w", entry.tx.Nonce(), errs[i]) + } + } + + return retry +} + +// buildAndSignWithNonce builds and signs a batch request at an explicit nonce. +func (w *Wallet) buildAndSignWithNonce( + ctx context.Context, req TxRequest, nonce uint64, +) (*types.Transaction, error) { + tx, err := w.buildTransactionWithNonce(ctx, req.To, req.Value, req.Data, req.GasLimit, nonce) + if err != nil { + return nil, err + } + + return w.SignTransaction(tx) +} + +// chunkIndices splits a slice into consecutive chunks of at most size entries. +func chunkIndices(indices []int, size int) [][]int { + chunks := make([][]int, 0, (len(indices)+size-1)/size) + + for start := 0; start < len(indices); start += size { + end := min(start+size, len(indices)) + chunks = append(chunks, indices[start:end]) + } + + return chunks +} diff --git a/pkg/wallet/batch_test.go b/pkg/wallet/batch_test.go new file mode 100644 index 00000000..48f4dc59 --- /dev/null +++ b/pkg/wallet/batch_test.go @@ -0,0 +1,100 @@ +package wallet + +import ( + "context" + "math/big" + "testing" + "time" + + "github.com/ethereum/go-ethereum/common" + "github.com/stretchr/testify/require" +) + +func batchRequests(count int) []TxRequest { + requests := make([]TxRequest, count) + for i := range requests { + requests[i] = TxRequest{ + To: common.HexToAddress("0x000000000000000000000000000000000000dEaD"), + Value: big.NewInt(int64(i + 1)), + Data: []byte{byte(i)}, + GasLimit: 21000, + } + } + + return requests +} + +// A batch signs consecutive nonces from one read, so the transactions can be in +// flight together instead of a block apart. +func TestSendBatchAndConfirmUsesConsecutiveNonces(t *testing.T) { + backend := newFakeBackend() + w := newTestWallet(t, backend) + + backend.pendingNonce = 7 + + results := w.SendBatchAndConfirm(context.Background(), batchRequests(3), time.Second) + require.Len(t, results, 3) + + for i, result := range results { + require.NoError(t, result.Err, "request %d", i) + require.NotNil(t, result.Receipt) + require.Equal(t, i, result.Index) + } + + require.Equal(t, []uint64{7, 8, 9}, backend.sentNonces()) +} + +// A nonce taken by another instance sharing the funding key must not fail the +// whole batch: the displaced transaction is resubmitted while the rest stand. +func TestSendBatchAndConfirmResubmitsDisplacedTransaction(t *testing.T) { + backend := newFakeBackend() + w := newTestWallet(t, backend) + + backend.pendingNonce = 4 + // The second transaction of the batch never lands and its slot is consumed + // by a foreign transaction, so the account nonce moves past it. + backend.dropNonce = 5 + + results := w.SendBatchAndConfirm(context.Background(), batchRequests(3), 2*time.Second) + require.Len(t, results, 3) + + for i, result := range results { + require.NoError(t, result.Err, "request %d", i) + require.NotNil(t, result.Receipt) + } + + // The displaced request was retried on a fresh nonce above the batch. + nonces := backend.sentNonces() + require.Equal(t, []uint64{4, 5, 6}, nonces[:3]) + require.Greater(t, nonces[3], uint64(6)) +} + +// Requests beyond the batch size are sent in consecutive chunks rather than one +// long run of pre-signed nonces. +func TestSendBatchAndConfirmChunksLargeBatches(t *testing.T) { + backend := newFakeBackend() + w := newTestWallet(t, backend) + + count := MaxBatchSize + 3 + + results := w.SendBatchAndConfirm(context.Background(), batchRequests(count), time.Second) + require.Len(t, results, count) + + for i, result := range results { + require.NoError(t, result.Err, "request %d", i) + } + + require.Len(t, backend.sentNonces(), count) +} + +func TestSendBatchAndConfirmEmpty(t *testing.T) { + w := newTestWallet(t, newFakeBackend()) + + require.Nil(t, w.SendBatchAndConfirm(context.Background(), nil, time.Second)) +} + +func TestChunkIndices(t *testing.T) { + require.Equal(t, [][]int{{1, 2}, {3, 4}, {5}}, chunkIndices([]int{1, 2, 3, 4, 5}, 2)) + require.Equal(t, [][]int{{1, 2, 3}}, chunkIndices([]int{1, 2, 3}, 10)) + require.Empty(t, chunkIndices(nil, 4)) +} diff --git a/pkg/wallet/wallet.go b/pkg/wallet/wallet.go index d3620cdc..5327ef58 100644 --- a/pkg/wallet/wallet.go +++ b/pkg/wallet/wallet.go @@ -170,13 +170,37 @@ func (w *Wallet) Sync(ctx context.Context) error { return nil } -// BuildTransaction creates a new unsigned transaction. +// BuildTransaction creates a new unsigned transaction with a freshly read nonce. func (w *Wallet) BuildTransaction( ctx context.Context, to common.Address, value *big.Int, data []byte, gasLimit uint64, +) (*types.Transaction, error) { + nonce, err := w.nextNonce(ctx) + if err != nil { + return nil, err + } + + w.log.WithFields(logrus.Fields{ + "address": w.address.Hex(), + "nonce": nonce, + }).Debug("Fetched next nonce from RPC for transaction build") + + return w.buildTransactionWithNonce(ctx, to, value, data, gasLimit, nonce) +} + +// buildTransactionWithNonce creates an unsigned transaction at an explicit nonce. +// Batch submissions assign consecutive nonces themselves: reading one per +// transaction would hand every transaction of a round the same nonce. +func (w *Wallet) buildTransactionWithNonce( + ctx context.Context, + to common.Address, + value *big.Int, + data []byte, + gasLimit uint64, + nonce uint64, ) (*types.Transaction, error) { // Ensure chain ID is available if w.chainID == nil { @@ -205,24 +229,6 @@ func (w *Wallet) BuildTransaction( gasFeeCap := new(big.Int).Mul(baseFee, big.NewInt(2)) gasFeeCap.Add(gasFeeCap, gasTipCap) - // Always read the next free nonce straight from the node on every build — never - // cached or tracked internally. - // - // Use max(pending, latest): normally the pending nonce already includes in-flight - // mempool txs and is >= latest. But ethrex has been observed returning a pending - // nonce *below* the latest (confirmed) nonce, which would make us stamp an - // already-used nonce and get "nonce too low". The latest nonce is the authoritative - // floor, so taking the larger of the two is correct on every client. - nonce, err := w.nextNonce(ctx) - if err != nil { - return nil, err - } - - w.log.WithFields(logrus.Fields{ - "address": w.address.Hex(), - "nonce": nonce, - }).Debug("Fetched next nonce from RPC for transaction build") - tx := types.NewTx(&types.DynamicFeeTx{ ChainID: w.chainID, Nonce: nonce, @@ -237,10 +243,11 @@ func (w *Wallet) BuildTransaction( return tx, nil } -// nextNonce returns the next usable nonce by reading the node fresh: the larger of the -// pending nonce and the latest (confirmed) nonce. The latest nonce is an authoritative -// floor that protects against clients (e.g. ethrex) whose pending nonce can lag below -// it; the pending nonce covers legitimate in-flight mempool txs on correct clients. +// nextNonce returns the next usable nonce by reading the node fresh — never cached or +// tracked internally: the larger of the pending nonce and the latest (confirmed) nonce. +// The latest nonce is an authoritative floor that protects against clients (e.g. ethrex) +// whose pending nonce can lag below it; the pending nonce covers legitimate in-flight +// mempool txs on correct clients. func (w *Wallet) nextNonce(ctx context.Context) (uint64, error) { pending, err := w.rpcClient.GetNonce(ctx, w.address) if err != nil { diff --git a/pkg/wallet/wallet_test.go b/pkg/wallet/wallet_test.go index f598bda6..57d3166f 100644 --- a/pkg/wallet/wallet_test.go +++ b/pkg/wallet/wallet_test.go @@ -33,6 +33,16 @@ type fakeBackend struct { known map[common.Hash]int accepted int + // sentNoncesLog records the nonce of every accepted tx, in send order. + sentNoncesLog []uint64 + + // dropNonce, when non-zero, models another instance taking that exact nonce + // slot: our tx there is accepted, then disappears before inclusion while the + // account nonce moves past it. + dropNonce uint64 + // displacedNonces holds the hashes whose slot was taken by dropNonce. + displacedNonces []common.Hash + // displaceFirstN: the first N accepted txs are dropped before inclusion — their // first receipt poll returns not-found and removes them from the pool, modelling // another instance replacing our tx at the same nonce. @@ -104,6 +114,11 @@ func (f *fakeBackend) SendTransaction(_ context.Context, tx *types.Transaction) f.accepted++ f.known[tx.Hash()] = f.accepted f.lastNonce = tx.Nonce() + f.sentNoncesLog = append(f.sentNoncesLog, tx.Nonce()) + + if f.dropNonce != 0 && tx.Nonce() == f.dropNonce { + f.displacedNonces = append(f.displacedNonces, tx.Hash()) + } if tx.Nonce()+1 > f.pendingNonce { f.pendingNonce = tx.Nonce() + 1 @@ -120,6 +135,15 @@ func (f *fakeBackend) GetTransactionReceipt(_ context.Context, txHash common.Has f.mu.Lock() defer f.mu.Unlock() + for _, displaced := range f.displacedNonces { + if displaced == txHash { + // The foreign tx took the slot: ours is gone and can never land. + delete(f.known, txHash) + + return nil, errNotFound + } + } + idx, ok := f.known[txHash] if !ok { return nil, errNotFound @@ -133,6 +157,14 @@ func (f *fakeBackend) GetTransactionReceipt(_ context.Context, txHash common.Has return &types.Receipt{Status: types.ReceiptStatusSuccessful, BlockNumber: big.NewInt(100)}, nil } +// sentNonces returns the nonces of every accepted tx, in send order. +func (f *fakeBackend) sentNonces() []uint64 { + f.mu.Lock() + defer f.mu.Unlock() + + return append([]uint64(nil), f.sentNoncesLog...) +} + func (f *fakeBackend) IsTxKnown(_ context.Context, txHash common.Hash) (bool, error) { f.mu.Lock() defer f.mu.Unlock() diff --git a/pkg/webui/handlers/api/action_plan_test.go b/pkg/webui/handlers/api/action_plan_test.go index 73b7fd71..ccd05580 100644 --- a/pkg/webui/handlers/api/action_plan_test.go +++ b/pkg/webui/handlers/api/action_plan_test.go @@ -91,12 +91,12 @@ func newPlanAPITestEnv(t *testing.T) *planAPITestEnv { t.Cleanup(func() { _ = stateDB.Close() }) - tracker := slot_results.NewTracker(cfg, chainSvc, stateDB, planSvc, nil, nil, nil, nil, log) + tracker := slot_results.NewTracker(cfg, chainSvc, stateDB, planSvc, nil, nil, nil, nil, nil, log) authHandler, err := auth.NewAuthHandler(context.Background(), "") require.NoError(t, err) - handler := NewAPIHandler(authHandler, nil, stateDB, nil, nil, nil, chainSvc, + handler := NewAPIHandler(authHandler, nil, stateDB, nil, nil, nil, nil, chainSvc, nil, nil, nil, nil, nil, nil, nil, planSvc, tracker) return &planAPITestEnv{ @@ -483,7 +483,7 @@ func TestUpdateSettingsPathBased(t *testing.T) { authHandler, err := auth.NewAuthHandler(context.Background(), "") require.NoError(t, err) - handler := NewAPIHandler(authHandler, settingsSvc, stateDB, nil, nil, nil, nil, + handler := NewAPIHandler(authHandler, settingsSvc, stateDB, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) post := func(body string) *httptest.ResponseRecorder { diff --git a/pkg/webui/handlers/api/api.go b/pkg/webui/handlers/api/api.go index 5ed9026b..e3fac11d 100644 --- a/pkg/webui/handlers/api/api.go +++ b/pkg/webui/handlers/api/api.go @@ -421,7 +421,7 @@ func (h *APIHandler) PostDeposit(w http.ResponseWriter, r *http.Request) { return } - if err := h.lifecycleMgr.EnsureBuilderRegistered(context.Background()); err != nil { + if err := h.lifecycleMgr.EnsureBuilderRegistered(context.Background(), h.lifecycleMgr.Registry().Primary()); err != nil { h.audit(r, token, "lifecycle.deposit", "", req, "error: "+err.Error()) writeError(w, http.StatusInternalServerError, err.Error()) @@ -459,7 +459,7 @@ func (h *APIHandler) PostTopup(w http.ResponseWriter, r *http.Request) { return } - if err := h.lifecycleMgr.CheckAndTopup(context.Background()); err != nil { + if err := h.lifecycleMgr.CheckAndTopup(context.Background(), h.lifecycleMgr.Registry().Primary()); err != nil { h.audit(r, token, "lifecycle.topup", "", nil, "error: "+err.Error()) writeError(w, http.StatusInternalServerError, err.Error()) @@ -498,7 +498,7 @@ func (h *APIHandler) PostExit(w http.ResponseWriter, r *http.Request) { return } - if err := h.lifecycleMgr.InitiateExit(context.Background()); err != nil { + if err := h.lifecycleMgr.InitiateExit(context.Background(), h.lifecycleMgr.Registry().Primary()); err != nil { h.audit(r, token, "lifecycle.exit", "", nil, "error: "+err.Error()) writeError(w, http.StatusInternalServerError, err.Error()) diff --git a/pkg/webui/handlers/api/builder_keys.go b/pkg/webui/handlers/api/builder_keys.go new file mode 100644 index 00000000..75476319 --- /dev/null +++ b/pkg/webui/handlers/api/builder_keys.go @@ -0,0 +1,354 @@ +package api + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "strconv" + + "github.com/golang-jwt/jwt/v5" + "github.com/gorilla/mux" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" +) + +// BuilderKeysResponse is the full view of the managed builder key set. +type BuilderKeysResponse struct { + Keys []*builder_keys.State `json:"keys"` + Aggregate builder_keys.Aggregate `json:"aggregate"` + Settings BuilderKeysSettings `json:"settings"` +} + +// BuilderKeysSettings mirrors the mutable key-set configuration so the UI can +// render and edit it without a second request. +type BuilderKeysSettings struct { + TargetCount uint64 `json:"target_count"` + MaxIndex uint64 `json:"max_index"` + AutoDeposit bool `json:"auto_deposit"` + AutoExit bool `json:"auto_exit"` +} + +// UpdateBuilderKeyTargetRequest sets how many builder keys are kept registered +// and funded. +type UpdateBuilderKeyTargetRequest struct { + Target uint64 `json:"target"` +} + +// BuilderKeyExitRequest asks for a builder key exit. LowerTarget (the default +// from the UI) decrements the target count in the same write, so the reconciler +// does not immediately deposit a replacement. +type BuilderKeyExitRequest struct { + LowerTarget bool `json:"lower_target"` +} + +// BuilderKeyTopupRequest tops a key up; AmountGwei of 0 uses the configured +// top-up amount. +type BuilderKeyTopupRequest struct { + AmountGwei uint64 `json:"amount_gwei,omitempty"` +} + +// keyRegistry returns the managed key set. It is available even without +// lifecycle management — the keys are derived either way; only deposits, exits +// and top-ups need the lifecycle manager. +func (h *APIHandler) keyRegistry() *builder_keys.Registry { + return h.keys +} + +// requireLifecycle reports whether lifecycle management is available, writing a +// 404 when it is not. Every mutating key operation needs it. +func (h *APIHandler) requireLifecycle(w http.ResponseWriter) bool { + if h.lifecycleMgr == nil { + writeError(w, http.StatusNotFound, "lifecycle management not enabled") + return false + } + + return true +} + +// GetBuilderKeys godoc +// @Id getBuilderKeys +// @Summary Get the managed builder key set +// @Tags Builder Keys +// @Description Returns every managed builder key with its lifecycle status, +// @Description on-chain builder index, balances and usage history, plus fleet +// @Description aggregates and the mutable key-set settings. +// @Produce json +// @Success 200 {object} BuilderKeysResponse +// @Failure 404 {object} map[string]string "Lifecycle management not enabled" +// @Router /api/buildoor/builder-keys [get] +func (h *APIHandler) GetBuilderKeys(w http.ResponseWriter, _ *http.Request) { + registry := h.keyRegistry() + if registry == nil { + writeError(w, http.StatusNotFound, "builder key registry not available") + return + } + + writeJSON(w, http.StatusOK, &BuilderKeysResponse{ + Keys: registry.States(), + Aggregate: registry.Aggregate(), + Settings: h.builderKeysSettings(), + }) +} + +// builderKeysSettings snapshots the mutable key-set configuration. +func (h *APIHandler) builderKeysSettings() BuilderKeysSettings { + cfg := h.settingsSvc.Load() + + return BuilderKeysSettings{ + TargetCount: cfg.BuilderKeys.EffectiveTargetCount(), + MaxIndex: cfg.BuilderKeys.MaxIndex, + AutoDeposit: cfg.BuilderKeys.AutoDeposit, + AutoExit: cfg.BuilderKeys.AutoExit, + } +} + +// UpdateBuilderKeyTarget godoc +// @Id updateBuilderKeyTarget +// @Summary Set the target builder key count +// @Tags Builder Keys +// @Description Sets how many builder keys are kept registered and funded. +// @Description Raising it deposits new keys; lowering it exits surplus keys +// @Description when auto-exit is on — an exited key can never be reactivated. +// @Accept json +// @Produce json +// @Param request body UpdateBuilderKeyTargetRequest true "Target key count" +// @Success 200 {object} BuilderKeysResponse +// @Failure 400 {object} map[string]string "Invalid request" +// @Failure 401 {object} map[string]string "Unauthorized" +// @Router /api/buildoor/builder-keys/target [post] +func (h *APIHandler) UpdateBuilderKeyTarget(w http.ResponseWriter, r *http.Request) { + token := h.authHandler.CheckAuthToken(r.Header.Get("Authorization")) + if token == nil { + writeError(w, http.StatusUnauthorized, "unauthorized") + return + } + + registry := h.keyRegistry() + if registry == nil { + writeError(w, http.StatusNotFound, "builder key registry not available") + return + } + + var req UpdateBuilderKeyTargetRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + writeError(w, http.StatusBadRequest, "invalid request body") + return + } + + if req.Target == 0 { + writeError(w, http.StatusBadRequest, "target must be at least 1") + return + } + + if !h.applyKeyTarget(w, r, token, "builder_keys.target", req, req.Target) { + return + } + + writeJSON(w, http.StatusOK, &BuilderKeysResponse{ + Keys: registry.States(), + Aggregate: registry.Aggregate(), + Settings: h.builderKeysSettings(), + }) +} + +// applyKeyTarget writes the target through the settings service, so persistence, +// CLI/UI recency resolution and the audit log all follow the same path as every +// other setting. It reports whether the write succeeded. +func (h *APIHandler) applyKeyTarget( + w http.ResponseWriter, r *http.Request, token *jwt.Token, action string, detail any, target uint64, +) bool { + encoded, err := json.Marshal(target) + if err != nil { + writeError(w, http.StatusInternalServerError, "failed to encode target") + return false + } + + return h.applySettings(w, r, token, action, detail, map[string]json.RawMessage{ + config.KeyBuilderKeysTargetCount: encoded, + }) +} + +// resolveKeyParam parses the {index} path parameter into a managed key. +func (h *APIHandler) resolveKeyParam(w http.ResponseWriter, r *http.Request) *builder_keys.Key { + registry := h.keyRegistry() + if registry == nil { + writeError(w, http.StatusNotFound, "builder key registry not available") + return nil + } + + keyIndex, err := strconv.ParseUint(mux.Vars(r)["index"], 10, 64) + if err != nil { + writeError(w, http.StatusBadRequest, "invalid key index") + return nil + } + + key, err := registry.Key(keyIndex) + if err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return nil + } + + return key +} + +// DepositBuilderKey godoc +// @Id depositBuilderKey +// @Summary Deposit for a builder key +// @Tags Builder Keys +// @Description Submits a builder deposit for the given key and waits for it to +// @Description register on the beacon chain. +// @Produce json +// @Param index path int true "Internal builder key index" +// @Success 200 {object} map[string]string +// @Failure 400 {object} map[string]string "Invalid key index or deposit failed" +// @Failure 401 {object} map[string]string "Unauthorized" +// @Router /api/buildoor/builder-keys/{index}/deposit [post] +func (h *APIHandler) DepositBuilderKey(w http.ResponseWriter, r *http.Request) { + token := h.authHandler.CheckAuthToken(r.Header.Get("Authorization")) + if token == nil { + writeError(w, http.StatusUnauthorized, "unauthorized") + return + } + + if !h.requireLifecycle(w) { + return + } + + key := h.resolveKeyParam(w, r) + if key == nil { + return + } + + detail := map[string]uint64{"key_index": key.KeyIndex()} + + if err := h.lifecycleMgr.EnsureBuilderRegistered(context.Background(), key); err != nil { + h.audit(r, token, "builder_keys.deposit", "", detail, "error: "+err.Error()) + writeError(w, http.StatusInternalServerError, err.Error()) + + return + } + + h.audit(r, token, "builder_keys.deposit", "", detail, "ok") + writeJSON(w, http.StatusOK, map[string]string{ + "status": "ok", + "key": fmt.Sprintf("#%d", key.KeyIndex()), + }) +} + +// TopupBuilderKey godoc +// @Id topupBuilderKey +// @Summary Top up a builder key +// @Tags Builder Keys +// @Description Submits a top-up deposit for the given key when its balance is +// @Description below the configured threshold. +// @Accept json +// @Produce json +// @Param index path int true "Internal builder key index" +// @Success 200 {object} map[string]string +// @Failure 400 {object} map[string]string "Invalid key index or top-up failed" +// @Failure 401 {object} map[string]string "Unauthorized" +// @Router /api/buildoor/builder-keys/{index}/topup [post] +func (h *APIHandler) TopupBuilderKey(w http.ResponseWriter, r *http.Request) { + token := h.authHandler.CheckAuthToken(r.Header.Get("Authorization")) + if token == nil { + writeError(w, http.StatusUnauthorized, "unauthorized") + return + } + + if !h.requireLifecycle(w) { + return + } + + key := h.resolveKeyParam(w, r) + if key == nil { + return + } + + req := BuilderKeyTopupRequest{} + if r.Body != nil { + _ = json.NewDecoder(r.Body).Decode(&req) + } + + detail := map[string]uint64{"key_index": key.KeyIndex(), "amount_gwei": req.AmountGwei} + + if err := h.lifecycleMgr.TopupKey(context.Background(), key, req.AmountGwei); err != nil { + h.audit(r, token, "builder_keys.topup", "", detail, "error: "+err.Error()) + writeError(w, http.StatusInternalServerError, err.Error()) + + return + } + + h.audit(r, token, "builder_keys.topup", "", detail, "ok") + writeJSON(w, http.StatusOK, map[string]string{ + "status": "ok", + "key": fmt.Sprintf("#%d", key.KeyIndex()), + }) +} + +// ExitBuilderKey godoc +// @Id exitBuilderKey +// @Summary Exit a builder key +// @Tags Builder Keys +// @Description Submits a builder exit request for the given key. Irreversible: +// @Description an exited key cannot be reactivated until its registry entry is +// @Description reused by another builder's deposit. With lower_target the +// @Description target key count is decremented in the same write, so the +// @Description reconciler does not deposit a replacement. +// @Accept json +// @Produce json +// @Param index path int true "Internal builder key index" +// @Param request body BuilderKeyExitRequest false "Exit options" +// @Success 200 {object} map[string]string +// @Failure 400 {object} map[string]string "Invalid key index" +// @Failure 401 {object} map[string]string "Unauthorized" +// @Failure 409 {object} map[string]string "Key cannot be exited yet" +// @Router /api/buildoor/builder-keys/{index}/exit [post] +func (h *APIHandler) ExitBuilderKey(w http.ResponseWriter, r *http.Request) { + token := h.authHandler.CheckAuthToken(r.Header.Get("Authorization")) + if token == nil { + writeError(w, http.StatusUnauthorized, "unauthorized") + return + } + + if !h.requireLifecycle(w) { + return + } + + key := h.resolveKeyParam(w, r) + if key == nil { + return + } + + // The body is optional; an absent one exits without touching the target. + req := BuilderKeyExitRequest{} + if r.Body != nil { + _ = json.NewDecoder(r.Body).Decode(&req) + } + + detail := map[string]any{"key_index": key.KeyIndex(), "lower_target": req.LowerTarget} + + if err := h.lifecycleMgr.InitiateExit(context.Background(), key); err != nil { + h.audit(r, token, "builder_keys.exit", "", detail, "error: "+err.Error()) + writeError(w, http.StatusConflict, err.Error()) + + return + } + + h.audit(r, token, "builder_keys.exit", "", detail, "ok") + + // Lower the target after the exit landed, so a failed exit never shrinks + // the fleet the operator asked for. + if req.LowerTarget { + target := h.settingsSvc.Load().BuilderKeys.EffectiveTargetCount() + if target > 1 { + h.applyKeyTarget(w, r, token, "builder_keys.target", detail, target-1) + } + } + + writeJSON(w, http.StatusOK, map[string]string{ + "status": "ok", + "key": fmt.Sprintf("#%d", key.KeyIndex()), + }) +} diff --git a/pkg/webui/handlers/api/builder_keys_test.go b/pkg/webui/handlers/api/builder_keys_test.go new file mode 100644 index 00000000..95378a91 --- /dev/null +++ b/pkg/webui/handlers/api/builder_keys_test.go @@ -0,0 +1,130 @@ +package api + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/gorilla/mux" + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/require" + + "github.com/ethpandaops/buildoor/pkg/builder_keys" + "github.com/ethpandaops/buildoor/pkg/config" + "github.com/ethpandaops/buildoor/pkg/db" + "github.com/ethpandaops/buildoor/pkg/webui/handlers/auth" +) + +const testEntryPrivkey = "3f2b8e1c9d4a6f70b5c8e2a1d7943f6058ac2be91d3f5074a6b8c2e1d9f30475" + +func keysTestHandler(t *testing.T, target uint64) *APIHandler { + t.Helper() + + log := logrus.New() + log.SetLevel(logrus.PanicLevel) + + keys := config.BuilderKeysConfig{TargetCount: target, DiscoveryGap: 1, MaxIndex: 32} + + cfg := config.DefaultConfig() + cfg.BuilderKeys = keys + + defaults := config.DefaultConfig() + defaults.BuilderKeys = keys + + // The settings service owns cfg, so it must be built before the registry + // reads the resolved values. + settingsSvc, err := config.NewService(cfg, defaults, map[string]bool{}, + db.NewDatabase(&db.Config{}, log), log) + require.NoError(t, err) + + registry, err := builder_keys.NewRegistry(cfg, testEntryPrivkey, log) + require.NoError(t, err) + registry.Refresh() + + authHandler, err := auth.NewAuthHandler(t.Context(), "") + require.NoError(t, err) + + return &APIHandler{authHandler: authHandler, keys: registry, settingsSvc: settingsSvc} +} + +func TestGetBuilderKeysWithoutRegistry(t *testing.T) { + h := &APIHandler{} + + rec := httptest.NewRecorder() + h.GetBuilderKeys(rec, httptest.NewRequest(http.MethodGet, "/api/buildoor/builder-keys", nil)) + + require.Equal(t, http.StatusNotFound, rec.Code) + + var body map[string]string + require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &body)) + require.Contains(t, body["error"], "builder key registry") +} + +// The key set is readable without lifecycle management: the keys are derived +// either way, only mutating them needs the manager. +func TestGetBuilderKeysWithoutLifecycle(t *testing.T) { + h := keysTestHandler(t, 3) + + rec := httptest.NewRecorder() + h.GetBuilderKeys(rec, httptest.NewRequest(http.MethodGet, "/api/buildoor/builder-keys", nil)) + + require.Equal(t, http.StatusOK, rec.Code) + + var resp BuilderKeysResponse + require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp)) + require.Len(t, resp.Keys, 3) + require.Equal(t, uint64(3), resp.Aggregate.Target) + require.Equal(t, uint64(3), resp.Settings.TargetCount) + + for keyIndex, state := range resp.Keys { + require.EqualValues(t, keyIndex, state.KeyIndex) + require.NotEmpty(t, state.PubkeyHex) + require.Equal(t, builder_keys.StatusUnused, state.Status) + } +} + +func TestMutatingBuilderKeyEndpointsRequireLifecycle(t *testing.T) { + h := keysTestHandler(t, 2) + + handlers := map[string]http.HandlerFunc{ + "deposit": h.DepositBuilderKey, + "topup": h.TopupBuilderKey, + "exit": h.ExitBuilderKey, + } + + for action, handler := range handlers { + t.Run(action, func(t *testing.T) { + req := httptest.NewRequest(http.MethodPost, "/api/buildoor/builder-keys/1/"+action, nil) + req = mux.SetURLVars(req, map[string]string{"index": "1"}) + + rec := httptest.NewRecorder() + handler(rec, req) + + require.Equal(t, http.StatusNotFound, rec.Code) + }) + } +} + +func TestResolveKeyParamRejectsBadIndices(t *testing.T) { + h := keysTestHandler(t, 2) + + tests := []struct { + name string + index string + }{ + {name: "not a number", index: "abc"}, + {name: "above the derivation cap", index: "99"}, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + req := httptest.NewRequest(http.MethodPost, "/api/buildoor/builder-keys/x/topup", nil) + req = mux.SetURLVars(req, map[string]string{"index": test.index}) + + rec := httptest.NewRecorder() + require.Nil(t, h.resolveKeyParam(rec, req)) + require.Equal(t, http.StatusBadRequest, rec.Code) + }) + } +} diff --git a/pkg/webui/handlers/api/builder_preferences_test.go b/pkg/webui/handlers/api/builder_preferences_test.go index f0997686..2212df44 100644 --- a/pkg/webui/handlers/api/builder_preferences_test.go +++ b/pkg/webui/handlers/api/builder_preferences_test.go @@ -18,7 +18,7 @@ import ( func TestGetBuilderPreferences_NotEnabled(t *testing.T) { // No builder API service wired → 404. - h := NewAPIHandler(nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) + h := NewAPIHandler(nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) req := httptest.NewRequest(http.MethodGet, "/api/buildoor/builder-preferences", nil) rec := httptest.NewRecorder() @@ -37,7 +37,7 @@ func TestGetBuilderPreferences_ReturnsEntries(t *testing.T) { // builderSvc (4th arg) nil so the event stream manager does not start; // srv is passed as builderAPISvc (9th arg). - h := NewAPIHandler(nil, nil, nil, nil, nil, nil, nil, nil, srv, nil, nil, nil, nil, nil, nil, nil) + h := NewAPIHandler(nil, nil, nil, nil, nil, nil, nil, nil, nil, srv, nil, nil, nil, nil, nil, nil, nil) req := httptest.NewRequest(http.MethodGet, "/api/buildoor/builder-preferences", nil) rec := httptest.NewRecorder() diff --git a/pkg/webui/handlers/api/events.go b/pkg/webui/handlers/api/events.go index 0dfe3ce5..a3562dd2 100644 --- a/pkg/webui/handlers/api/events.go +++ b/pkg/webui/handlers/api/events.go @@ -13,6 +13,7 @@ import ( "github.com/ethpandaops/go-eth2-client/spec/phase0" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/builderapi" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/lifecycle" @@ -43,6 +44,7 @@ const ( EventTypeSlotState EventType = "slot_state" EventTypePayloadAvailable EventType = "payload_available" EventTypeBuilderInfo EventType = "builder_info" + EventTypeBuilderKeys EventType = "builder_keys" EventTypeHeadVotes EventType = "head_votes" EventTypeVoteCoverage EventType = "vote_coverage" EventTypeRevealStarted EventType = "reveal_started" @@ -172,13 +174,17 @@ type BidSubmittedEvent struct { Warning string `json:"warning,omitempty"` // Full bid message properties (blob commitments aggregated to a count). - ExecutionPayment uint64 `json:"execution_payment,omitempty"` - FeeRecipient string `json:"fee_recipient,omitempty"` - GasLimit uint64 `json:"gas_limit,omitempty"` - BuilderIndex uint64 `json:"builder_index,omitempty"` - ParentBlockHash string `json:"parent_block_hash,omitempty"` - ParentBlockRoot string `json:"parent_block_root,omitempty"` - NumBlobCommitments int `json:"num_blob_commitments,omitempty"` + ExecutionPayment uint64 `json:"execution_payment,omitempty"` + FeeRecipient string `json:"fee_recipient,omitempty"` + GasLimit uint64 `json:"gas_limit,omitempty"` + BuilderIndex uint64 `json:"builder_index,omitempty"` + // KeyIndex is the managed builder key that signed the bid. With several + // keys bidding one slot, the builder index alone does not say which of ours + // it was. + KeyIndex *uint64 `json:"key_index,omitempty"` + ParentBlockHash string `json:"parent_block_hash,omitempty"` + ParentBlockRoot string `json:"parent_block_root,omitempty"` + NumBlobCommitments int `json:"num_blob_commitments,omitempty"` } // HeadReceivedEvent is sent when a head event is received. @@ -306,7 +312,9 @@ func fillEnvelopeDetail(detail *EnvelopeDetail, envelope *eth2all.SignedExecutio } } -// BuilderInfoEvent contains builder identity and balance information. +// BuilderInfoEvent contains builder identity and balance information. The +// identity fields describe the primary key; the fleet totals summarise every +// managed key (they equal the primary's values on a single-key deployment). type BuilderInfoEvent struct { BuilderPubkey string `json:"builder_pubkey"` BuilderIndex uint64 `json:"builder_index"` @@ -319,6 +327,21 @@ type BuilderInfoEvent struct { WalletBalance string `json:"wallet_balance_wei,omitempty"` DepositEpoch uint64 `json:"deposit_epoch"` WithdrawableEpoch uint64 `json:"withdrawable_epoch"` + + // Fleet summary of the managed builder key set. + KeyCount uint64 `json:"key_count"` + KeyTarget uint64 `json:"key_target"` + KeysActive uint64 `json:"keys_active"` + TotalBalance uint64 `json:"total_balance_gwei"` + TotalPendingPayments uint64 `json:"total_pending_payments_gwei"` + TotalEffective uint64 `json:"total_effective_gwei"` +} + +// BuilderKeysStreamEvent is the full managed key set, pushed on every change so +// the keys view stays live without polling. +type BuilderKeysStreamEvent struct { + Keys []*builder_keys.State `json:"keys"` + Aggregate builder_keys.Aggregate `json:"aggregate"` } // HeadVotesStreamEvent is sent when head vote participation changes. @@ -484,10 +507,11 @@ type BuilderAPISubmitBlockDeliveredEvent struct { // EventStreamManager manages SSE connections and event broadcasting. type EventStreamManager struct { builderSvc *payload_builder.Service - epbsSvc *p2p_bidder.Service // Optional ePBS service for bid events - lifecycleMgr *lifecycle.Manager // Optional lifecycle manager for balance info - chainSvc chain.Service // Optional chain service for head vote tracking - builderAPISvc *builderapi.Server // Optional Builder API server + epbsSvc *p2p_bidder.Service // Optional ePBS service for bid events + lifecycleMgr *lifecycle.Manager // Optional lifecycle manager for balance info + keys *builder_keys.Registry // Managed builder key set + chainSvc chain.Service // Optional chain service for head vote tracking + builderAPISvc *builderapi.Server // Optional Builder API server revealSvc *payload_bidder.RevealService // Optional shared reveal service (Gloas+) inclusionTracker *payload_bidder.InclusionTracker // Optional shared inclusion tracker @@ -534,6 +558,7 @@ func NewEventStreamManager( builderSvc *payload_builder.Service, epbsSvc *p2p_bidder.Service, lifecycleMgr *lifecycle.Manager, + keys *builder_keys.Registry, chainSvc chain.Service, builderAPISvc *builderapi.Server, revealSvc *payload_bidder.RevealService, @@ -548,6 +573,7 @@ func NewEventStreamManager( builderSvc: builderSvc, epbsSvc: epbsSvc, lifecycleMgr: lifecycleMgr, + keys: keys, chainSvc: chainSvc, builderAPISvc: builderAPISvc, revealSvc: revealSvc, @@ -659,6 +685,18 @@ func (m *EventStreamManager) Start() { resultUpdateChan = resultUpdateSub.Channel() } + // Subscribe to builder key set changes (if lifecycle management available). + // Lossy delivery is fine: every event carries the complete key set, so a + // dropped one is superseded by the next. + var keyChangeSub *utils.Subscription[*builder_keys.ChangeEvent] + + var keyChangeChan <-chan *builder_keys.ChangeEvent + + if registry := m.keyRegistry(); registry != nil { + keyChangeSub = registry.SubscribeChanges(8, false) + keyChangeChan = keyChangeSub.Channel() + } + // Subscribe to head vote + subnet coverage updates (if chain service available) var hvSub *utils.Subscription[*chain.HeadVoteUpdate] @@ -735,6 +773,10 @@ func (m *EventStreamManager) Start() { defer resultUpdateSub.Unsubscribe() } + if keyChangeSub != nil { + defer keyChangeSub.Unsubscribe() + } + // Slot tracking ticker ticker := time.NewTicker(100 * time.Millisecond) defer ticker.Stop() @@ -840,6 +882,21 @@ func (m *EventStreamManager) Start() { Data: event, }) + case event, ok := <-keyChangeChan: + if !ok { + keyChangeChan = nil + continue + } + + m.Broadcast(&StreamEvent{ + Type: EventTypeBuilderKeys, + Timestamp: time.Now().UnixMilli(), + Data: BuilderKeysStreamEvent{ + Keys: event.States, + Aggregate: event.Aggregate, + }, + }) + case event, ok := <-revealChan: if !ok { revealChan = nil @@ -1180,6 +1237,14 @@ func (m *EventStreamManager) handleBidSubmissionEvent(event *p2p_bidder.BidSubmi data.FeeRecipient = fmt.Sprintf("0x%x", bid.FeeRecipient) data.GasLimit = bid.GasLimit data.BuilderIndex = uint64(bid.BuilderIndex) + + if registry := m.keyRegistry(); registry != nil { + if key := registry.ByBuilderIndex(uint64(bid.BuilderIndex)); key != nil { + keyIndex := key.KeyIndex() + data.KeyIndex = &keyIndex + } + } + data.ParentBlockHash = fmt.Sprintf("0x%x", bid.ParentBlockHash[:]) data.ParentBlockRoot = fmt.Sprintf("0x%x", bid.ParentBlockRoot[:]) data.NumBlobCommitments = len(bid.BlobKZGCommitments) @@ -1249,10 +1314,14 @@ func (m *EventStreamManager) handleHeadEvent(event *beacon.HeadEvent) { } func (m *EventStreamManager) handleBidEvent(event *beacon.BidEvent) { - // Determine if this is our own bid. The beacon node echoes our submitted bid - // back over the SSE stream, so without this it would be shown as an external - // bid. Match on builder index, the same way the ePBS service classifies bids. - isOurs := m.epbsSvc != nil && event.BuilderIndex == m.epbsSvc.GetBuilderIndex() + // Determine if this is our own bid. The beacon node echoes our submitted bids + // back over the SSE stream, so without this they would be shown as external + // bids. Every managed key counts: with a fleet, a slot's bids carry several + // different builder indices, all of them ours. + isOurs := false + if registry := m.keyRegistry(); registry != nil { + isOurs = registry.ByBuilderIndex(event.BuilderIndex) != nil + } m.broadcastForSlot(event.Slot, &StreamEvent{ Type: EventTypeBidEvent, @@ -1524,7 +1593,7 @@ func (m *EventStreamManager) getBuilderInfo() BuilderInfoEvent { // Apply local balance adjustment (topups + revealed bid deductions since last state refresh) if m.payments != nil { - adjustment := m.payments.GetBalanceAdjustment() + adjustment := m.payments.GetBalanceAdjustment(m.epbsSvc.Registry().Primary().KeyIndex()) adjusted := int64(info.CLBalance) + adjustment if adjusted < 0 { adjusted = 0 @@ -1552,9 +1621,44 @@ func (m *EventStreamManager) getBuilderInfo() BuilderInfoEvent { info.EffectiveBalance = info.CLBalance - info.PendingPayments } + // Fleet totals. On a single-key deployment these equal the primary key's + // values above; they diverge as soon as the target count grows. + if registry := m.keyRegistry(); registry != nil { + aggregate := registry.Aggregate() + info.KeyCount = uint64(len(registry.Keys())) + info.KeyTarget = aggregate.Target + info.KeysActive = aggregate.Active + info.TotalBalance = aggregate.TotalBalance + info.TotalPendingPayments = aggregate.TotalPendingPayments + info.TotalEffective = aggregate.TotalEffective + } + return info } +// keyRegistry returns the managed builder key set (nil only when the WebUI runs +// without one). +func (m *EventStreamManager) keyRegistry() *builder_keys.Registry { + return m.keys +} + +// BroadcastBuilderKeys pushes the current managed key set to all clients. +func (m *EventStreamManager) BroadcastBuilderKeys() { + registry := m.keyRegistry() + if registry == nil { + return + } + + m.Broadcast(&StreamEvent{ + Type: EventTypeBuilderKeys, + Timestamp: time.Now().UnixMilli(), + Data: BuilderKeysStreamEvent{ + Keys: registry.States(), + Aggregate: registry.Aggregate(), + }, + }) +} + func (m *EventStreamManager) getServiceStatus() ServiceStatusEvent { regState := "unknown" if m.epbsSvc != nil { @@ -1698,6 +1802,20 @@ func (m *EventStreamManager) SendInitialState(ctx context.Context, ch chan *Stre return } + // Send the managed builder key set + if registry := m.keyRegistry(); registry != nil { + if !send(&StreamEvent{ + Type: EventTypeBuilderKeys, + Timestamp: time.Now().UnixMilli(), + Data: BuilderKeysStreamEvent{ + Keys: registry.States(), + Aggregate: registry.Aggregate(), + }, + }) { + return + } + } + // Send service status if !send(&StreamEvent{ Type: EventTypeServiceStatus, diff --git a/pkg/webui/handlers/api/events_test.go b/pkg/webui/handlers/api/events_test.go index a8a0176b..2ae23d28 100644 --- a/pkg/webui/handlers/api/events_test.go +++ b/pkg/webui/handlers/api/events_test.go @@ -13,7 +13,7 @@ import ( // newTestEventStreamManager builds a manager suitable for exercising the // broadcast / replay-cache paths, which touch no injected service. func newTestEventStreamManager() *EventStreamManager { - return NewEventStreamManager(nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) + return NewEventStreamManager(nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) } func slotEvent(slot uint64) *StreamEvent { diff --git a/pkg/webui/handlers/api/handler.go b/pkg/webui/handlers/api/handler.go index e16d17f3..ffd73113 100644 --- a/pkg/webui/handlers/api/handler.go +++ b/pkg/webui/handlers/api/handler.go @@ -5,6 +5,7 @@ import ( "github.com/ethpandaops/go-eth2-client/spec/phase0" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/builderapi" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" @@ -27,6 +28,7 @@ type APIHandler struct { builderSvc *payload_builder.Service epbsSvc *p2p_bidder.Service // May be nil lifecycleMgr *lifecycle.Manager // May be nil + keys *builder_keys.Registry // Managed builder key set chainSvc chain.Service // May be nil validatorStore *memstore.Store[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration] // May be nil (only set when Builder API enabled) builderAPISvc *builderapi.Server // May be nil (only set when Builder API enabled) @@ -49,6 +51,7 @@ func NewAPIHandler( builderSvc *payload_builder.Service, epbsSvc *p2p_bidder.Service, lifecycleMgr *lifecycle.Manager, + keys *builder_keys.Registry, chainSvc chain.Service, validatorStore *memstore.Store[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration], builderAPISvc *builderapi.Server, @@ -67,6 +70,7 @@ func NewAPIHandler( builderSvc: builderSvc, epbsSvc: epbsSvc, lifecycleMgr: lifecycleMgr, + keys: keys, chainSvc: chainSvc, validatorStore: validatorStore, builderAPISvc: builderAPISvc, @@ -83,7 +87,7 @@ func NewAPIHandler( // Create and start event stream manager if builderSvc != nil { h.eventStreamMgr = NewEventStreamManager( - builderSvc, epbsSvc, lifecycleMgr, chainSvc, + builderSvc, epbsSvc, lifecycleMgr, keys, chainSvc, builderAPISvc, revealSvc, inclusionTracker, payments, planSvc, resultTracker, ) diff --git a/pkg/webui/handlers/api/overview.go b/pkg/webui/handlers/api/overview.go index de9ce139..9ef57416 100644 --- a/pkg/webui/handlers/api/overview.go +++ b/pkg/webui/handlers/api/overview.go @@ -2,6 +2,7 @@ package api import ( "net/http" + "slices" "github.com/ethpandaops/buildoor/pkg/p2p_bidder" "github.com/ethpandaops/buildoor/version" @@ -46,8 +47,45 @@ type OverviewStats struct { BuilderAPIRegisteredValidators int `json:"builder_api_registered_validators"` } +// OverviewBuilderKey is one managed builder key, kept small enough that an +// instance can list its whole fleet in the overview payload. +type OverviewBuilderKey struct { + KeyIndex uint64 `json:"key_index"` + Pubkey string `json:"pubkey"` + Status string `json:"status"` + // BuilderIndex is the on-chain registry index, only meaningful when + // HasBuilderIndex is set (index 0 is a valid builder index). + BuilderIndex uint64 `json:"builder_index"` + HasBuilderIndex bool `json:"has_builder_index"` + BalanceGwei uint64 `json:"balance_gwei,omitempty"` +} + +// OverviewBuilders is the managed fleet: the counts a consumer needs to render +// a summary, plus every registered builder index so an indexer can attribute +// blocks and bids to this instance. +// +// Consumers must treat it as optional — an instance predating the managed key +// set omits it, and the legacy top-level builder_pubkey/builder_index fields +// remain populated with the primary key for exactly that reason. +type OverviewBuilders struct { + Count uint64 `json:"count"` + Target uint64 `json:"target"` + Active uint64 `json:"active"` + // Indexes are the on-chain indexes of every key currently in the registry, + // ascending. This is the field an indexer wants. + Indexes []uint64 `json:"indexes"` + Keys []OverviewBuilderKey `json:"keys,omitempty"` + TotalBalanceGwei uint64 `json:"total_balance_gwei"` + TotalPendingGwei uint64 `json:"total_pending_payments_gwei"` + TotalEffectiveGwei uint64 `json:"total_effective_gwei"` +} + // OverviewResponse is the response payload of /api/buildoor/overview — a compact // summary used by the multi-instance overview UI. +// +// BuilderPubkey/BuilderIndex/IsRegistered describe the PRIMARY key only and are +// retained for consumers that predate the managed key set; Builders carries the +// whole fleet. type OverviewResponse struct { Version string `json:"version"` Running bool `json:"running"` @@ -59,6 +97,7 @@ type OverviewResponse struct { Services OverviewServices `json:"services"` Balances OverviewBalances `json:"balances"` Stats OverviewStats `json:"stats"` + Builders *OverviewBuilders `json:"builders,omitempty"` } // GetOverview godoc @@ -120,6 +159,46 @@ func (h *APIHandler) GetOverview(w http.ResponseWriter, r *http.Request) { } } + // The managed fleet, when the key registry is available. Its aggregate + // replaces the primary key's balances: a multi-key instance's funds are the + // fleet's, and on a single-key deployment the two are identical. + if registry := h.keyRegistry(); registry != nil { + states := registry.States() + aggregate := registry.Aggregate() + + builders := &OverviewBuilders{ + Count: aggregate.Managed, + Target: aggregate.Target, + Active: aggregate.Active, + Indexes: make([]uint64, 0, len(states)), + Keys: make([]OverviewBuilderKey, 0, len(states)), + TotalBalanceGwei: aggregate.TotalBalance, + TotalPendingGwei: aggregate.TotalPendingPayments, + TotalEffectiveGwei: aggregate.TotalEffective, + } + + for _, state := range states { + if state.HasBuilderIndex { + builders.Indexes = append(builders.Indexes, state.BuilderIndex) + } + + builders.Keys = append(builders.Keys, OverviewBuilderKey{ + KeyIndex: state.KeyIndex, + Pubkey: state.PubkeyHex, + Status: string(state.Status), + BuilderIndex: state.BuilderIndex, + HasBuilderIndex: state.HasBuilderIndex, + BalanceGwei: state.Balance, + }) + } + + slices.Sort(builders.Indexes) + + resp.Builders = builders + resp.Balances.CLBalanceGwei = aggregate.TotalBalance + resp.Balances.PendingPaymentsGwei = aggregate.TotalPendingPayments + } + if resp.Balances.CLBalanceGwei > resp.Balances.PendingPaymentsGwei { resp.Balances.EffectiveBalanceGwei = resp.Balances.CLBalanceGwei - resp.Balances.PendingPaymentsGwei } diff --git a/pkg/webui/handlers/docs/docs.go b/pkg/webui/handlers/docs/docs.go index e6f48300..8c3672e7 100644 --- a/pkg/webui/handlers/docs/docs.go +++ b/pkg/webui/handlers/docs/docs.go @@ -425,6 +425,265 @@ const docTemplate = `{ } } }, + "/api/buildoor/builder-keys": { + "get": { + "description": "Returns every managed builder key with its lifecycle status,\non-chain builder index, balances and usage history, plus fleet\naggregates and the mutable key-set settings.", + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Get the managed builder key set", + "operationId": "getBuilderKeys", + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/api.BuilderKeysResponse" + } + }, + "404": { + "description": "Lifecycle management not enabled", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/target": { + "post": { + "description": "Sets how many builder keys are kept registered and funded.\nRaising it deposits new keys; lowering it exits surplus keys\nwhen auto-exit is on — an exited key can never be reactivated.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Set the target builder key count", + "operationId": "updateBuilderKeyTarget", + "parameters": [ + { + "description": "Target key count", + "name": "request", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/api.UpdateBuilderKeyTargetRequest" + } + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/api.BuilderKeysResponse" + } + }, + "400": { + "description": "Invalid request", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/{index}/deposit": { + "post": { + "description": "Submits a builder deposit for the given key and waits for it to\nregister on the beacon chain.", + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Deposit for a builder key", + "operationId": "depositBuilderKey", + "parameters": [ + { + "type": "integer", + "description": "Internal builder key index", + "name": "index", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "400": { + "description": "Invalid key index or deposit failed", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/{index}/exit": { + "post": { + "description": "Submits a builder exit request for the given key. Irreversible:\nan exited key cannot be reactivated until its registry entry is\nreused by another builder's deposit. With lower_target the\ntarget key count is decremented in the same write, so the\nreconciler does not deposit a replacement.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Exit a builder key", + "operationId": "exitBuilderKey", + "parameters": [ + { + "type": "integer", + "description": "Internal builder key index", + "name": "index", + "in": "path", + "required": true + }, + { + "description": "Exit options", + "name": "request", + "in": "body", + "schema": { + "$ref": "#/definitions/api.BuilderKeyExitRequest" + } + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "400": { + "description": "Invalid key index", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "409": { + "description": "Key cannot be exited yet", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/{index}/topup": { + "post": { + "description": "Submits a top-up deposit for the given key when its balance is\nbelow the configured threshold.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Top up a builder key", + "operationId": "topupBuilderKey", + "parameters": [ + { + "type": "integer", + "description": "Internal builder key index", + "name": "index", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "400": { + "description": "Invalid key index or top-up failed", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, "/api/buildoor/builder-preferences": { "get": { "description": "Returns all builder preferences currently in the cache, submitted by proposers via the submitBuilderPreferences API.", @@ -822,6 +1081,12 @@ const docTemplate = `{ "name": "slot", "in": "path", "required": true + }, + { + "type": "string", + "description": "Build-parent candidate key (parent_full, parent_empty, grandparent_full, grandparent_empty)", + "name": "candidate", + "in": "query" } ], "responses": { @@ -862,6 +1127,72 @@ const docTemplate = `{ } } }, + "/api/buildoor/slot-results/{slot}/payload/{index}": { + "get": { + "description": "Returns one of the execution payloads built for the slot by\nartifact index (a slot may build several candidate payloads on\ndifferent parents; the slot result's build entries carry each\nbuild's artifact index). Content negotiation as with the\ndefault payload artifact endpoint.", + "produces": [ + "application/json", + "application/octet-stream" + ], + "tags": [ + "ActionPlan" + ], + "summary": "Get one of a slot's built candidate payloads", + "operationId": "getSlotPayloadArtifactByIndex", + "parameters": [ + { + "type": "integer", + "description": "Slot", + "name": "slot", + "in": "path", + "required": true + }, + { + "type": "integer", + "description": "Payload artifact index", + "name": "index", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "Versioned payload (or raw SSZ)", + "schema": { + "type": "object", + "additionalProperties": true + } + }, + "400": { + "description": "Bad Request", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "404": { + "description": "No artifact for this slot/index", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "406": { + "description": "No acceptable content type", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, "/api/buildoor/validators": { "get": { "description": "Returns the list of validators registered via the Builder API (fee recipient preferences). Not paginated.", @@ -1473,6 +1804,10 @@ const docTemplate = `{ "action_plan.BidPlan": { "type": "object", "properties": { + "bid_candidate": { + "description": "BidCandidate overrides which built candidate payload this slot's p2p\nbids commit to: auto, all, or a specific candidate key.", + "type": "string" + }, "bid_end_time": { "type": "integer" }, @@ -1484,6 +1819,10 @@ const docTemplate = `{ "description": "ms, \u003e= 0, 0 = single bid", "type": "integer" }, + "bid_keys_per_slot": { + "description": "BidKeysPerSlot overrides how many distinct builder keys may bid this\nslot (0 = one key per selected candidate payload).", + "type": "integer" + }, "bid_min_amount": { "description": "gwei", "type": "integer" @@ -1503,6 +1842,10 @@ const docTemplate = `{ "description": "IgnoreMissingPrefs bids with the payload's fee recipient when no gossip\nproposer preferences arrived for the slot, bypassing the skip gate.", "type": "boolean" }, + "key_strategy": { + "description": "KeyStrategy overrides which managed builder key signs this slot's bids:\nround_robin, single, random or least_used.", + "type": "string" + }, "mode": { "$ref": "#/definitions/action_plan.Mode" } @@ -1511,6 +1854,13 @@ const docTemplate = `{ "action_plan.BuildPlan": { "type": "object", "properties": { + "candidates": { + "description": "Candidates overrides the global build-candidate policy for this slot:\ncandidate key (parent_full, parent_empty, grandparent_full,\ngrandparent_empty) -\u003e mode (auto, always, never). Absent keys inherit\nthe global policy.", + "type": "object", + "additionalProperties": { + "type": "string" + } + }, "reorg_parent_payload": { "description": "ReorgParentPayload builds on the grandparent (n-2) execution payload\ninstead of the immediate parent: the FCU head block hash and the payload\nattributes' withdrawals are taken from the PARENT slot's payload\nattributes (whose parent is n-2), while every other property comes from\nthe current slot. This is a deliberate parent-payload reorg attempt —\nrejected by mainnet forkchoice, but useful for exercising the reveal /\ninclusion path against a withheld parent.", "type": "boolean" @@ -1520,6 +1870,10 @@ const docTemplate = `{ "action_plan.BuilderAPIPlan": { "type": "object", "properties": { + "key_strategy": { + "description": "KeyStrategy overrides which managed builder key signs this slot's served\nbids: round_robin, single, random or least_used.", + "type": "string" + }, "mode": { "$ref": "#/definitions/action_plan.Mode" }, @@ -1527,6 +1881,10 @@ const docTemplate = `{ "description": "ResponseDelayMs delays the bid response by this many milliseconds\n(context-cancellable, capped at one slot).", "type": "integer" }, + "serve_candidates": { + "description": "ServeCandidates overrides which built candidate payloads bid requests\nfor this slot may be answered from: all, canonical_only, or a\ncomma-separated candidate key list.", + "type": "string" + }, "total_value_override_gwei": { "description": "TotalValueOverrideGwei is the absolute total proposer-visible bid value\n(before the Gloas execution-payment split), replacing block value +\nsubsidy. May exceed the block value to test payment edge cases.", "type": "integer" @@ -1667,6 +2025,14 @@ const docTemplate = `{ "action_plan.ResolvedBidSettings": { "type": "object", "properties": { + "bid_candidate": { + "description": "BidCandidate is the effective bid candidate selection for the slot:\nauto, all, or a specific candidate key.", + "type": "string" + }, + "bid_keys_per_slot": { + "description": "BidKeysPerSlot caps how many distinct builder keys bid the slot\n(0 = one key per selected candidate payload).", + "type": "integer" + }, "end_ms": { "type": "integer" }, @@ -1684,6 +2050,10 @@ const docTemplate = `{ "interval_ms": { "type": "integer" }, + "key_strategy": { + "description": "KeyStrategy is the effective builder key selection strategy for the\nslot's bids.", + "type": "string" + }, "min_gwei": { "type": "integer" }, @@ -1710,6 +2080,13 @@ const docTemplate = `{ "description": "BuildStartTimeMs is the effective build start time, milliseconds\nrelative to slot start (signed).", "type": "integer" }, + "candidate_modes": { + "description": "CandidateModes is the effective build-candidate policy for the slot:\ncandidate key -\u003e auto/always/never, merged from the global config and\nthe plan's build.candidates overrides.", + "type": "object", + "additionalProperties": { + "type": "string" + } + }, "forced": { "description": "Forced marks builds the plan pushed past the schedule (they never\nconsume the next_n budget).", "type": "boolean" @@ -1738,6 +2115,14 @@ const docTemplate = `{ "description": "Forced marks that the plan activated serving although the module is\nglobally disabled.", "type": "boolean" }, + "key_strategy": { + "description": "KeyStrategy is the effective builder key selection strategy for the\nslot's served bids.", + "type": "string" + }, + "serve_candidates": { + "description": "ServeCandidates is the effective serve-candidates policy for the slot\n(all, canonical_only, or a comma-separated candidate key list).", + "type": "string" + }, "subsidy_gwei": { "type": "integer" }, @@ -2036,6 +2421,48 @@ const docTemplate = `{ } } }, + "api.BuilderKeyExitRequest": { + "type": "object", + "properties": { + "lower_target": { + "type": "boolean" + } + } + }, + "api.BuilderKeysResponse": { + "type": "object", + "properties": { + "aggregate": { + "$ref": "#/definitions/builder_keys.Aggregate" + }, + "keys": { + "type": "array", + "items": { + "$ref": "#/definitions/builder_keys.State" + } + }, + "settings": { + "$ref": "#/definitions/api.BuilderKeysSettings" + } + } + }, + "api.BuilderKeysSettings": { + "type": "object", + "properties": { + "auto_deposit": { + "type": "boolean" + }, + "auto_exit": { + "type": "boolean" + }, + "max_index": { + "type": "integer" + }, + "target_count": { + "type": "integer" + } + } + }, "api.BuilderPreferencesEntry": { "type": "object", "properties": { @@ -2488,6 +2915,14 @@ const docTemplate = `{ } } }, + "api.UpdateBuilderKeyTargetRequest": { + "type": "object", + "properties": { + "target": { + "type": "integer" + } + } + }, "api.UpdateEPBSRequest": { "type": "object", "properties": { @@ -2559,6 +2994,129 @@ const docTemplate = `{ } } }, + "builder_keys.Aggregate": { + "type": "object", + "properties": { + "active": { + "type": "integer" + }, + "depositing": { + "type": "integer" + }, + "exited": { + "type": "integer" + }, + "exiting": { + "type": "integer" + }, + "managed": { + "type": "integer" + }, + "pending": { + "type": "integer" + }, + "target": { + "type": "integer" + }, + "total_balance_gwei": { + "type": "integer" + }, + "total_effective_gwei": { + "type": "integer" + }, + "total_pending_payments_gwei": { + "type": "integer" + }, + "unused": { + "type": "integer" + }, + "withdrawn": { + "type": "integer" + } + } + }, + "builder_keys.State": { + "type": "object", + "properties": { + "balance_adjustment_gwei": { + "type": "integer" + }, + "balance_gwei": { + "type": "integer" + }, + "bids_submitted": { + "type": "integer" + }, + "bids_won": { + "type": "integer" + }, + "builder_index": { + "description": "BuilderIndex is the on-chain registry index; only meaningful when\nHasBuilderIndex is set (index 0 is a valid builder index).", + "type": "integer" + }, + "deposit_epoch": { + "type": "integer" + }, + "effective_balance_gwei": { + "description": "EffectiveBalance is the balance the key can actually bid against:\nchain balance plus local adjustments minus pending payments.", + "type": "integer" + }, + "has_builder_index": { + "type": "boolean" + }, + "key_index": { + "type": "integer" + }, + "last_deposit_at": { + "type": "integer" + }, + "last_exit_at": { + "type": "integer" + }, + "last_topup_epoch": { + "description": "LastTopupEpoch guards the per-key top-up cooldown (0 = never topped up).", + "type": "integer" + }, + "pending_payments_gwei": { + "type": "integer" + }, + "pubkey": { + "description": "PubkeyHex is the 0x-prefixed public key, for JSON consumers.", + "type": "string" + }, + "status": { + "$ref": "#/definitions/builder_keys.Status" + }, + "use_count": { + "description": "UseCount is how many deposit generations this key has gone through.", + "type": "integer" + }, + "withdrawable_epoch": { + "type": "integer" + } + } + }, + "builder_keys.Status": { + "type": "string", + "enum": [ + "unused", + "depositing", + "pending", + "active", + "exiting", + "exited", + "withdrawn" + ], + "x-enum-varnames": [ + "StatusUnused", + "StatusDepositing", + "StatusPending", + "StatusActive", + "StatusExiting", + "StatusExited", + "StatusWithdrawn" + ] + }, "db.AuditLog": { "type": "object", "properties": { @@ -2689,6 +3247,10 @@ const docTemplate = `{ "gas_limit": { "type": "integer" }, + "key_index": { + "description": "KeyIndex is the internal builder key the bid was signed with, resolved\nfrom BuilderIndex at record time. Recorded because a builder index is\nreused by other builders after an exit, so the mapping is only reliable\nwhile the bid is being made.", + "type": "integer" + }, "num_blob_commitments": { "type": "integer" }, @@ -2753,6 +3315,10 @@ const docTemplate = `{ "slot_results.BuildOutcome": { "type": "object", "properties": { + "artifact_idx": { + "description": "ArtifactIdx is the per-slot payload artifact index of this build's\ncaptured payload (nil when no artifact was captured).", + "type": "integer" + }, "at": { "type": "string" }, @@ -2781,6 +3347,10 @@ const docTemplate = `{ "block_value_wei": { "type": "string" }, + "candidate": { + "description": "Candidate classifies which build-parent candidate this outcome belongs\nto (parent_full, parent_empty, grandparent_full, grandparent_empty;\nempty = unclassified or single-build slot).", + "type": "string" + }, "error": { "type": "string" }, @@ -2976,7 +3546,18 @@ const docTemplate = `{ } }, "build": { - "$ref": "#/definitions/slot_results.BuildOutcome" + "description": "Build is the slot's primary build outcome (the most canonical ready\ncandidate, or the single lifecycle record). Builds lists every\ncandidate build the slot produced when more than one ran.", + "allOf": [ + { + "$ref": "#/definitions/slot_results.BuildOutcome" + } + ] + }, + "builds": { + "type": "array", + "items": { + "$ref": "#/definitions/slot_results.BuildOutcome" + } }, "dropped_attempts": { "description": "DroppedAttempts counts attempts beyond the per-kind retention cap,\nkeyed by kind (\"bids\", \"block_submissions\", \"reveal_attempts\").", diff --git a/pkg/webui/handlers/docs/swagger.json b/pkg/webui/handlers/docs/swagger.json index 5b1ed565..b777c967 100644 --- a/pkg/webui/handlers/docs/swagger.json +++ b/pkg/webui/handlers/docs/swagger.json @@ -414,6 +414,265 @@ } } }, + "/api/buildoor/builder-keys": { + "get": { + "description": "Returns every managed builder key with its lifecycle status,\non-chain builder index, balances and usage history, plus fleet\naggregates and the mutable key-set settings.", + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Get the managed builder key set", + "operationId": "getBuilderKeys", + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/api.BuilderKeysResponse" + } + }, + "404": { + "description": "Lifecycle management not enabled", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/target": { + "post": { + "description": "Sets how many builder keys are kept registered and funded.\nRaising it deposits new keys; lowering it exits surplus keys\nwhen auto-exit is on — an exited key can never be reactivated.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Set the target builder key count", + "operationId": "updateBuilderKeyTarget", + "parameters": [ + { + "description": "Target key count", + "name": "request", + "in": "body", + "required": true, + "schema": { + "$ref": "#/definitions/api.UpdateBuilderKeyTargetRequest" + } + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "$ref": "#/definitions/api.BuilderKeysResponse" + } + }, + "400": { + "description": "Invalid request", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/{index}/deposit": { + "post": { + "description": "Submits a builder deposit for the given key and waits for it to\nregister on the beacon chain.", + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Deposit for a builder key", + "operationId": "depositBuilderKey", + "parameters": [ + { + "type": "integer", + "description": "Internal builder key index", + "name": "index", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "400": { + "description": "Invalid key index or deposit failed", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/{index}/exit": { + "post": { + "description": "Submits a builder exit request for the given key. Irreversible:\nan exited key cannot be reactivated until its registry entry is\nreused by another builder's deposit. With lower_target the\ntarget key count is decremented in the same write, so the\nreconciler does not deposit a replacement.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Exit a builder key", + "operationId": "exitBuilderKey", + "parameters": [ + { + "type": "integer", + "description": "Internal builder key index", + "name": "index", + "in": "path", + "required": true + }, + { + "description": "Exit options", + "name": "request", + "in": "body", + "schema": { + "$ref": "#/definitions/api.BuilderKeyExitRequest" + } + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "400": { + "description": "Invalid key index", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "409": { + "description": "Key cannot be exited yet", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, + "/api/buildoor/builder-keys/{index}/topup": { + "post": { + "description": "Submits a top-up deposit for the given key when its balance is\nbelow the configured threshold.", + "consumes": [ + "application/json" + ], + "produces": [ + "application/json" + ], + "tags": [ + "Builder Keys" + ], + "summary": "Top up a builder key", + "operationId": "topupBuilderKey", + "parameters": [ + { + "type": "integer", + "description": "Internal builder key index", + "name": "index", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "OK", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "400": { + "description": "Invalid key index or top-up failed", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "401": { + "description": "Unauthorized", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, "/api/buildoor/builder-preferences": { "get": { "description": "Returns all builder preferences currently in the cache, submitted by proposers via the submitBuilderPreferences API.", @@ -811,6 +1070,12 @@ "name": "slot", "in": "path", "required": true + }, + { + "type": "string", + "description": "Build-parent candidate key (parent_full, parent_empty, grandparent_full, grandparent_empty)", + "name": "candidate", + "in": "query" } ], "responses": { @@ -851,6 +1116,72 @@ } } }, + "/api/buildoor/slot-results/{slot}/payload/{index}": { + "get": { + "description": "Returns one of the execution payloads built for the slot by\nartifact index (a slot may build several candidate payloads on\ndifferent parents; the slot result's build entries carry each\nbuild's artifact index). Content negotiation as with the\ndefault payload artifact endpoint.", + "produces": [ + "application/json", + "application/octet-stream" + ], + "tags": [ + "ActionPlan" + ], + "summary": "Get one of a slot's built candidate payloads", + "operationId": "getSlotPayloadArtifactByIndex", + "parameters": [ + { + "type": "integer", + "description": "Slot", + "name": "slot", + "in": "path", + "required": true + }, + { + "type": "integer", + "description": "Payload artifact index", + "name": "index", + "in": "path", + "required": true + } + ], + "responses": { + "200": { + "description": "Versioned payload (or raw SSZ)", + "schema": { + "type": "object", + "additionalProperties": true + } + }, + "400": { + "description": "Bad Request", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "404": { + "description": "No artifact for this slot/index", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + }, + "406": { + "description": "No acceptable content type", + "schema": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } + } + } + }, "/api/buildoor/validators": { "get": { "description": "Returns the list of validators registered via the Builder API (fee recipient preferences). Not paginated.", @@ -1462,6 +1793,10 @@ "action_plan.BidPlan": { "type": "object", "properties": { + "bid_candidate": { + "description": "BidCandidate overrides which built candidate payload this slot's p2p\nbids commit to: auto, all, or a specific candidate key.", + "type": "string" + }, "bid_end_time": { "type": "integer" }, @@ -1473,6 +1808,10 @@ "description": "ms, \u003e= 0, 0 = single bid", "type": "integer" }, + "bid_keys_per_slot": { + "description": "BidKeysPerSlot overrides how many distinct builder keys may bid this\nslot (0 = one key per selected candidate payload).", + "type": "integer" + }, "bid_min_amount": { "description": "gwei", "type": "integer" @@ -1492,6 +1831,10 @@ "description": "IgnoreMissingPrefs bids with the payload's fee recipient when no gossip\nproposer preferences arrived for the slot, bypassing the skip gate.", "type": "boolean" }, + "key_strategy": { + "description": "KeyStrategy overrides which managed builder key signs this slot's bids:\nround_robin, single, random or least_used.", + "type": "string" + }, "mode": { "$ref": "#/definitions/action_plan.Mode" } @@ -1500,6 +1843,13 @@ "action_plan.BuildPlan": { "type": "object", "properties": { + "candidates": { + "description": "Candidates overrides the global build-candidate policy for this slot:\ncandidate key (parent_full, parent_empty, grandparent_full,\ngrandparent_empty) -\u003e mode (auto, always, never). Absent keys inherit\nthe global policy.", + "type": "object", + "additionalProperties": { + "type": "string" + } + }, "reorg_parent_payload": { "description": "ReorgParentPayload builds on the grandparent (n-2) execution payload\ninstead of the immediate parent: the FCU head block hash and the payload\nattributes' withdrawals are taken from the PARENT slot's payload\nattributes (whose parent is n-2), while every other property comes from\nthe current slot. This is a deliberate parent-payload reorg attempt —\nrejected by mainnet forkchoice, but useful for exercising the reveal /\ninclusion path against a withheld parent.", "type": "boolean" @@ -1509,6 +1859,10 @@ "action_plan.BuilderAPIPlan": { "type": "object", "properties": { + "key_strategy": { + "description": "KeyStrategy overrides which managed builder key signs this slot's served\nbids: round_robin, single, random or least_used.", + "type": "string" + }, "mode": { "$ref": "#/definitions/action_plan.Mode" }, @@ -1516,6 +1870,10 @@ "description": "ResponseDelayMs delays the bid response by this many milliseconds\n(context-cancellable, capped at one slot).", "type": "integer" }, + "serve_candidates": { + "description": "ServeCandidates overrides which built candidate payloads bid requests\nfor this slot may be answered from: all, canonical_only, or a\ncomma-separated candidate key list.", + "type": "string" + }, "total_value_override_gwei": { "description": "TotalValueOverrideGwei is the absolute total proposer-visible bid value\n(before the Gloas execution-payment split), replacing block value +\nsubsidy. May exceed the block value to test payment edge cases.", "type": "integer" @@ -1656,6 +2014,14 @@ "action_plan.ResolvedBidSettings": { "type": "object", "properties": { + "bid_candidate": { + "description": "BidCandidate is the effective bid candidate selection for the slot:\nauto, all, or a specific candidate key.", + "type": "string" + }, + "bid_keys_per_slot": { + "description": "BidKeysPerSlot caps how many distinct builder keys bid the slot\n(0 = one key per selected candidate payload).", + "type": "integer" + }, "end_ms": { "type": "integer" }, @@ -1673,6 +2039,10 @@ "interval_ms": { "type": "integer" }, + "key_strategy": { + "description": "KeyStrategy is the effective builder key selection strategy for the\nslot's bids.", + "type": "string" + }, "min_gwei": { "type": "integer" }, @@ -1699,6 +2069,13 @@ "description": "BuildStartTimeMs is the effective build start time, milliseconds\nrelative to slot start (signed).", "type": "integer" }, + "candidate_modes": { + "description": "CandidateModes is the effective build-candidate policy for the slot:\ncandidate key -\u003e auto/always/never, merged from the global config and\nthe plan's build.candidates overrides.", + "type": "object", + "additionalProperties": { + "type": "string" + } + }, "forced": { "description": "Forced marks builds the plan pushed past the schedule (they never\nconsume the next_n budget).", "type": "boolean" @@ -1727,6 +2104,14 @@ "description": "Forced marks that the plan activated serving although the module is\nglobally disabled.", "type": "boolean" }, + "key_strategy": { + "description": "KeyStrategy is the effective builder key selection strategy for the\nslot's served bids.", + "type": "string" + }, + "serve_candidates": { + "description": "ServeCandidates is the effective serve-candidates policy for the slot\n(all, canonical_only, or a comma-separated candidate key list).", + "type": "string" + }, "subsidy_gwei": { "type": "integer" }, @@ -2025,6 +2410,48 @@ } } }, + "api.BuilderKeyExitRequest": { + "type": "object", + "properties": { + "lower_target": { + "type": "boolean" + } + } + }, + "api.BuilderKeysResponse": { + "type": "object", + "properties": { + "aggregate": { + "$ref": "#/definitions/builder_keys.Aggregate" + }, + "keys": { + "type": "array", + "items": { + "$ref": "#/definitions/builder_keys.State" + } + }, + "settings": { + "$ref": "#/definitions/api.BuilderKeysSettings" + } + } + }, + "api.BuilderKeysSettings": { + "type": "object", + "properties": { + "auto_deposit": { + "type": "boolean" + }, + "auto_exit": { + "type": "boolean" + }, + "max_index": { + "type": "integer" + }, + "target_count": { + "type": "integer" + } + } + }, "api.BuilderPreferencesEntry": { "type": "object", "properties": { @@ -2477,6 +2904,14 @@ } } }, + "api.UpdateBuilderKeyTargetRequest": { + "type": "object", + "properties": { + "target": { + "type": "integer" + } + } + }, "api.UpdateEPBSRequest": { "type": "object", "properties": { @@ -2548,6 +2983,129 @@ } } }, + "builder_keys.Aggregate": { + "type": "object", + "properties": { + "active": { + "type": "integer" + }, + "depositing": { + "type": "integer" + }, + "exited": { + "type": "integer" + }, + "exiting": { + "type": "integer" + }, + "managed": { + "type": "integer" + }, + "pending": { + "type": "integer" + }, + "target": { + "type": "integer" + }, + "total_balance_gwei": { + "type": "integer" + }, + "total_effective_gwei": { + "type": "integer" + }, + "total_pending_payments_gwei": { + "type": "integer" + }, + "unused": { + "type": "integer" + }, + "withdrawn": { + "type": "integer" + } + } + }, + "builder_keys.State": { + "type": "object", + "properties": { + "balance_adjustment_gwei": { + "type": "integer" + }, + "balance_gwei": { + "type": "integer" + }, + "bids_submitted": { + "type": "integer" + }, + "bids_won": { + "type": "integer" + }, + "builder_index": { + "description": "BuilderIndex is the on-chain registry index; only meaningful when\nHasBuilderIndex is set (index 0 is a valid builder index).", + "type": "integer" + }, + "deposit_epoch": { + "type": "integer" + }, + "effective_balance_gwei": { + "description": "EffectiveBalance is the balance the key can actually bid against:\nchain balance plus local adjustments minus pending payments.", + "type": "integer" + }, + "has_builder_index": { + "type": "boolean" + }, + "key_index": { + "type": "integer" + }, + "last_deposit_at": { + "type": "integer" + }, + "last_exit_at": { + "type": "integer" + }, + "last_topup_epoch": { + "description": "LastTopupEpoch guards the per-key top-up cooldown (0 = never topped up).", + "type": "integer" + }, + "pending_payments_gwei": { + "type": "integer" + }, + "pubkey": { + "description": "PubkeyHex is the 0x-prefixed public key, for JSON consumers.", + "type": "string" + }, + "status": { + "$ref": "#/definitions/builder_keys.Status" + }, + "use_count": { + "description": "UseCount is how many deposit generations this key has gone through.", + "type": "integer" + }, + "withdrawable_epoch": { + "type": "integer" + } + } + }, + "builder_keys.Status": { + "type": "string", + "enum": [ + "unused", + "depositing", + "pending", + "active", + "exiting", + "exited", + "withdrawn" + ], + "x-enum-varnames": [ + "StatusUnused", + "StatusDepositing", + "StatusPending", + "StatusActive", + "StatusExiting", + "StatusExited", + "StatusWithdrawn" + ] + }, "db.AuditLog": { "type": "object", "properties": { @@ -2678,6 +3236,10 @@ "gas_limit": { "type": "integer" }, + "key_index": { + "description": "KeyIndex is the internal builder key the bid was signed with, resolved\nfrom BuilderIndex at record time. Recorded because a builder index is\nreused by other builders after an exit, so the mapping is only reliable\nwhile the bid is being made.", + "type": "integer" + }, "num_blob_commitments": { "type": "integer" }, @@ -2742,6 +3304,10 @@ "slot_results.BuildOutcome": { "type": "object", "properties": { + "artifact_idx": { + "description": "ArtifactIdx is the per-slot payload artifact index of this build's\ncaptured payload (nil when no artifact was captured).", + "type": "integer" + }, "at": { "type": "string" }, @@ -2770,6 +3336,10 @@ "block_value_wei": { "type": "string" }, + "candidate": { + "description": "Candidate classifies which build-parent candidate this outcome belongs\nto (parent_full, parent_empty, grandparent_full, grandparent_empty;\nempty = unclassified or single-build slot).", + "type": "string" + }, "error": { "type": "string" }, @@ -2965,7 +3535,18 @@ } }, "build": { - "$ref": "#/definitions/slot_results.BuildOutcome" + "description": "Build is the slot's primary build outcome (the most canonical ready\ncandidate, or the single lifecycle record). Builds lists every\ncandidate build the slot produced when more than one ran.", + "allOf": [ + { + "$ref": "#/definitions/slot_results.BuildOutcome" + } + ] + }, + "builds": { + "type": "array", + "items": { + "$ref": "#/definitions/slot_results.BuildOutcome" + } }, "dropped_attempts": { "description": "DroppedAttempts counts attempts beyond the per-kind retention cap,\nkeyed by kind (\"bids\", \"block_submissions\", \"reveal_attempts\").", diff --git a/pkg/webui/handlers/docs/swagger.yaml b/pkg/webui/handlers/docs/swagger.yaml index 43d78049..d90dce66 100644 --- a/pkg/webui/handlers/docs/swagger.yaml +++ b/pkg/webui/handlers/docs/swagger.yaml @@ -1,6 +1,11 @@ definitions: action_plan.BidPlan: properties: + bid_candidate: + description: |- + BidCandidate overrides which built candidate payload this slot's p2p + bids commit to: auto, all, or a specific candidate key. + type: string bid_end_time: type: integer bid_increase: @@ -9,6 +14,11 @@ definitions: bid_interval: description: ms, >= 0, 0 = single bid type: integer + bid_keys_per_slot: + description: |- + BidKeysPerSlot overrides how many distinct builder keys may bid this + slot (0 = one key per selected candidate payload). + type: integer bid_min_amount: description: gwei type: integer @@ -28,11 +38,25 @@ definitions: IgnoreMissingPrefs bids with the payload's fee recipient when no gossip proposer preferences arrived for the slot, bypassing the skip gate. type: boolean + key_strategy: + description: |- + KeyStrategy overrides which managed builder key signs this slot's bids: + round_robin, single, random or least_used. + type: string mode: $ref: '#/definitions/action_plan.Mode' type: object action_plan.BuildPlan: properties: + candidates: + additionalProperties: + type: string + description: |- + Candidates overrides the global build-candidate policy for this slot: + candidate key (parent_full, parent_empty, grandparent_full, + grandparent_empty) -> mode (auto, always, never). Absent keys inherit + the global policy. + type: object reorg_parent_payload: description: |- ReorgParentPayload builds on the grandparent (n-2) execution payload @@ -46,6 +70,11 @@ definitions: type: object action_plan.BuilderAPIPlan: properties: + key_strategy: + description: |- + KeyStrategy overrides which managed builder key signs this slot's served + bids: round_robin, single, random or least_used. + type: string mode: $ref: '#/definitions/action_plan.Mode' response_delay_ms: @@ -53,6 +82,12 @@ definitions: ResponseDelayMs delays the bid response by this many milliseconds (context-cancellable, capped at one slot). type: integer + serve_candidates: + description: |- + ServeCandidates overrides which built candidate payloads bid requests + for this slot may be answered from: all, canonical_only, or a + comma-separated candidate key list. + type: string total_value_override_gwei: description: |- TotalValueOverrideGwei is the absolute total proposer-visible bid value @@ -154,6 +189,16 @@ definitions: type: object action_plan.ResolvedBidSettings: properties: + bid_candidate: + description: |- + BidCandidate is the effective bid candidate selection for the slot: + auto, all, or a specific candidate key. + type: string + bid_keys_per_slot: + description: |- + BidKeysPerSlot caps how many distinct builder keys bid the slot + (0 = one key per selected candidate payload). + type: integer end_ms: type: integer forced: @@ -168,6 +213,11 @@ definitions: type: integer interval_ms: type: integer + key_strategy: + description: |- + KeyStrategy is the effective builder key selection strategy for the + slot's bids. + type: string min_gwei: type: integer start_ms: @@ -191,6 +241,14 @@ definitions: BuildStartTimeMs is the effective build start time, milliseconds relative to slot start (signed). type: integer + candidate_modes: + additionalProperties: + type: string + description: |- + CandidateModes is the effective build-candidate policy for the slot: + candidate key -> auto/always/never, merged from the global config and + the plan's build.candidates overrides. + type: object forced: description: |- Forced marks builds the plan pushed past the schedule (they never @@ -222,6 +280,16 @@ definitions: Forced marks that the plan activated serving although the module is globally disabled. type: boolean + key_strategy: + description: |- + KeyStrategy is the effective builder key selection strategy for the + slot's served bids. + type: string + serve_candidates: + description: |- + ServeCandidates is the effective serve-candidates policy for the slot + (all, canonical_only, or a comma-separated candidate key list). + type: string subsidy_gwei: type: integer total_value_gwei: @@ -447,6 +515,33 @@ definitions: validator_count: type: integer type: object + api.BuilderKeyExitRequest: + properties: + lower_target: + type: boolean + type: object + api.BuilderKeysResponse: + properties: + aggregate: + $ref: '#/definitions/builder_keys.Aggregate' + keys: + items: + $ref: '#/definitions/builder_keys.State' + type: array + settings: + $ref: '#/definitions/api.BuilderKeysSettings' + type: object + api.BuilderKeysSettings: + properties: + auto_deposit: + type: boolean + auto_exit: + type: boolean + max_index: + type: integer + target_count: + type: integer + type: object api.BuilderPreferencesEntry: properties: max_execution_payment: @@ -750,6 +845,11 @@ definitions: status: type: string type: object + api.UpdateBuilderKeyTargetRequest: + properties: + target: + type: integer + type: object api.UpdateEPBSRequest: properties: bid_end_time: @@ -799,6 +899,98 @@ definitions: description: Unix timestamp type: integer type: object + builder_keys.Aggregate: + properties: + active: + type: integer + depositing: + type: integer + exited: + type: integer + exiting: + type: integer + managed: + type: integer + pending: + type: integer + target: + type: integer + total_balance_gwei: + type: integer + total_effective_gwei: + type: integer + total_pending_payments_gwei: + type: integer + unused: + type: integer + withdrawn: + type: integer + type: object + builder_keys.State: + properties: + balance_adjustment_gwei: + type: integer + balance_gwei: + type: integer + bids_submitted: + type: integer + bids_won: + type: integer + builder_index: + description: |- + BuilderIndex is the on-chain registry index; only meaningful when + HasBuilderIndex is set (index 0 is a valid builder index). + type: integer + deposit_epoch: + type: integer + effective_balance_gwei: + description: |- + EffectiveBalance is the balance the key can actually bid against: + chain balance plus local adjustments minus pending payments. + type: integer + has_builder_index: + type: boolean + key_index: + type: integer + last_deposit_at: + type: integer + last_exit_at: + type: integer + last_topup_epoch: + description: LastTopupEpoch guards the per-key top-up cooldown (0 = never + topped up). + type: integer + pending_payments_gwei: + type: integer + pubkey: + description: PubkeyHex is the 0x-prefixed public key, for JSON consumers. + type: string + status: + $ref: '#/definitions/builder_keys.Status' + use_count: + description: UseCount is how many deposit generations this key has gone through. + type: integer + withdrawable_epoch: + type: integer + type: object + builder_keys.Status: + enum: + - unused + - depositing + - pending + - active + - exiting + - exited + - withdrawn + type: string + x-enum-varnames: + - StatusUnused + - StatusDepositing + - StatusPending + - StatusActive + - StatusExiting + - StatusExited + - StatusWithdrawn db.AuditLog: properties: action: @@ -894,6 +1086,13 @@ definitions: type: string gas_limit: type: integer + key_index: + description: |- + KeyIndex is the internal builder key the bid was signed with, resolved + from BuilderIndex at record time. Recorded because a builder index is + reused by other builders after an exit, so the mapping is only reliable + while the bid is being made. + type: integer num_blob_commitments: type: integer parent_block_hash: @@ -940,6 +1139,11 @@ definitions: type: object slot_results.BuildOutcome: properties: + artifact_idx: + description: |- + ArtifactIdx is the per-slot payload artifact index of this build's + captured payload (nil when no artifact was captured). + type: integer at: type: string attributes: @@ -960,6 +1164,12 @@ definitions: type: integer block_value_wei: type: string + candidate: + description: |- + Candidate classifies which build-parent candidate this outcome belongs + to (parent_full, parent_empty, grandparent_full, grandparent_empty; + empty = unclassified or single-build slot). + type: string error: type: string excess_blob_gas: @@ -1103,7 +1313,16 @@ definitions: $ref: '#/definitions/slot_results.BlockSubmission' type: array build: - $ref: '#/definitions/slot_results.BuildOutcome' + allOf: + - $ref: '#/definitions/slot_results.BuildOutcome' + description: |- + Build is the slot's primary build outcome (the most canonical ready + candidate, or the single lifecycle record). Builds lists every + candidate build the slot produced when more than one ran. + builds: + items: + $ref: '#/definitions/slot_results.BuildOutcome' + type: array dropped_attempts: additionalProperties: type: integer @@ -1441,6 +1660,193 @@ paths: summary: Get Builder API status tags: - Buildoor + /api/buildoor/builder-keys: + get: + description: |- + Returns every managed builder key with its lifecycle status, + on-chain builder index, balances and usage history, plus fleet + aggregates and the mutable key-set settings. + operationId: getBuilderKeys + produces: + - application/json + responses: + "200": + description: OK + schema: + $ref: '#/definitions/api.BuilderKeysResponse' + "404": + description: Lifecycle management not enabled + schema: + additionalProperties: + type: string + type: object + summary: Get the managed builder key set + tags: + - Builder Keys + /api/buildoor/builder-keys/{index}/deposit: + post: + description: |- + Submits a builder deposit for the given key and waits for it to + register on the beacon chain. + operationId: depositBuilderKey + parameters: + - description: Internal builder key index + in: path + name: index + required: true + type: integer + produces: + - application/json + responses: + "200": + description: OK + schema: + additionalProperties: + type: string + type: object + "400": + description: Invalid key index or deposit failed + schema: + additionalProperties: + type: string + type: object + "401": + description: Unauthorized + schema: + additionalProperties: + type: string + type: object + summary: Deposit for a builder key + tags: + - Builder Keys + /api/buildoor/builder-keys/{index}/exit: + post: + consumes: + - application/json + description: |- + Submits a builder exit request for the given key. Irreversible: + an exited key cannot be reactivated until its registry entry is + reused by another builder's deposit. With lower_target the + target key count is decremented in the same write, so the + reconciler does not deposit a replacement. + operationId: exitBuilderKey + parameters: + - description: Internal builder key index + in: path + name: index + required: true + type: integer + - description: Exit options + in: body + name: request + schema: + $ref: '#/definitions/api.BuilderKeyExitRequest' + produces: + - application/json + responses: + "200": + description: OK + schema: + additionalProperties: + type: string + type: object + "400": + description: Invalid key index + schema: + additionalProperties: + type: string + type: object + "401": + description: Unauthorized + schema: + additionalProperties: + type: string + type: object + "409": + description: Key cannot be exited yet + schema: + additionalProperties: + type: string + type: object + summary: Exit a builder key + tags: + - Builder Keys + /api/buildoor/builder-keys/{index}/topup: + post: + consumes: + - application/json + description: |- + Submits a top-up deposit for the given key when its balance is + below the configured threshold. + operationId: topupBuilderKey + parameters: + - description: Internal builder key index + in: path + name: index + required: true + type: integer + produces: + - application/json + responses: + "200": + description: OK + schema: + additionalProperties: + type: string + type: object + "400": + description: Invalid key index or top-up failed + schema: + additionalProperties: + type: string + type: object + "401": + description: Unauthorized + schema: + additionalProperties: + type: string + type: object + summary: Top up a builder key + tags: + - Builder Keys + /api/buildoor/builder-keys/target: + post: + consumes: + - application/json + description: |- + Sets how many builder keys are kept registered and funded. + Raising it deposits new keys; lowering it exits surplus keys + when auto-exit is on — an exited key can never be reactivated. + operationId: updateBuilderKeyTarget + parameters: + - description: Target key count + in: body + name: request + required: true + schema: + $ref: '#/definitions/api.UpdateBuilderKeyTargetRequest' + produces: + - application/json + responses: + "200": + description: OK + schema: + $ref: '#/definitions/api.BuilderKeysResponse' + "400": + description: Invalid request + schema: + additionalProperties: + type: string + type: object + "401": + description: Unauthorized + schema: + additionalProperties: + type: string + type: object + summary: Set the target builder key count + tags: + - Builder Keys /api/buildoor/builder-preferences: get: description: Returns all builder preferences currently in the cache, submitted @@ -1733,6 +2139,11 @@ paths: name: slot required: true type: integer + - description: Build-parent candidate key (parent_full, parent_empty, grandparent_full, + grandparent_empty) + in: query + name: candidate + type: string produces: - application/json - application/octet-stream @@ -1763,6 +2174,56 @@ paths: summary: Get the built execution payload of a slot tags: - ActionPlan + /api/buildoor/slot-results/{slot}/payload/{index}: + get: + description: |- + Returns one of the execution payloads built for the slot by + artifact index (a slot may build several candidate payloads on + different parents; the slot result's build entries carry each + build's artifact index). Content negotiation as with the + default payload artifact endpoint. + operationId: getSlotPayloadArtifactByIndex + parameters: + - description: Slot + in: path + name: slot + required: true + type: integer + - description: Payload artifact index + in: path + name: index + required: true + type: integer + produces: + - application/json + - application/octet-stream + responses: + "200": + description: Versioned payload (or raw SSZ) + schema: + additionalProperties: true + type: object + "400": + description: Bad Request + schema: + additionalProperties: + type: string + type: object + "404": + description: No artifact for this slot/index + schema: + additionalProperties: + type: string + type: object + "406": + description: No acceptable content type + schema: + additionalProperties: + type: string + type: object + summary: Get one of a slot's built candidate payloads + tags: + - ActionPlan /api/buildoor/validators: get: description: Returns the list of validators registered via the Builder API (fee diff --git a/pkg/webui/src/App.tsx b/pkg/webui/src/App.tsx index dea0fb3e..a4e2635b 100644 --- a/pkg/webui/src/App.tsx +++ b/pkg/webui/src/App.tsx @@ -6,6 +6,7 @@ const DashboardPage = React.lazy(() => import('./pages/DashboardPage')); const ActionPlanPage = React.lazy(() => import('./pages/ActionPlanPage')); const ValidatorsPage = React.lazy(() => import('./pages/ValidatorsPage')); const BidsWonPage = React.lazy(() => import('./pages/BidsWonPage')); +const BuilderKeysPage = React.lazy(() => import('./pages/BuilderKeysPage')); const ProposerPreferencesPage = React.lazy(() => import('./pages/ProposerPreferencesPage')); const BuilderPreferencesPage = React.lazy(() => import('./pages/BuilderPreferencesPage')); const AuditLogPage = React.lazy(() => import('./pages/AuditLogPage')); @@ -23,6 +24,7 @@ export const App: React.FC = () => { {currentView === 'action-plan' && } {currentView === 'validators' && } {currentView === 'bids-won' && } + {currentView === 'builder-keys' && } {currentView === 'proposer-preferences' && } {currentView === 'builder-preferences' && } {currentView === 'audit-log' && } diff --git a/pkg/webui/src/components/BuilderInfo.tsx b/pkg/webui/src/components/BuilderInfo.tsx index e62ee070..63eac70b 100644 --- a/pkg/webui/src/components/BuilderInfo.tsx +++ b/pkg/webui/src/components/BuilderInfo.tsx @@ -1,7 +1,9 @@ -import React, { useState } from 'react'; +import React, { useEffect, useState } from 'react'; import { useAuthContext } from '../context/AuthContext'; import type { BuilderInfo as BuilderInfoType, ServiceStatus, Config } from '../types'; import { CopyableHash } from './CopyableHash'; +import { useBuilderKeyActions } from '../hooks/useBuilderKeys'; +import { setView } from '../stores/viewStore'; interface BuilderInfoProps { builderInfo: BuilderInfoType | null; @@ -28,13 +30,25 @@ export const BuilderInfo: React.FC = ({ builderInfo, serviceSt const [editingLifecycle, setEditingLifecycle] = useState(false); const [lcThreshold, setLcThreshold] = useState(''); const [lcAmount, setLcAmount] = useState(''); - const [exitState, setExitState] = useState<'idle' | 'confirm' | 'submitting' | 'done' | 'error'>('idle'); - const [exitError, setExitError] = useState(''); + const [editingTarget, setEditingTarget] = useState(false); + const [targetInput, setTargetInput] = useState(''); + const [targetError, setTargetError] = useState(''); + const keyActions = useBuilderKeyActions(); const lifecycleAvailable = serviceStatus?.lifecycle_available ?? false; const lifecycleEnabled = serviceStatus?.lifecycle_enabled ?? false; const registrationState = serviceStatus?.epbs_registration_state; - const canExit = registrationState === 'registered' || registrationState === 'pending_finalization'; + // A managed fleet reports totals; a single key reports its own values, which + // are the same numbers. + const keyCount = builderInfo?.key_count ?? 0; + const keyTarget = builderInfo?.key_target ?? 0; + const isFleet = keyCount > 1 || keyTarget > 1; + + useEffect(() => { + if (!editingTarget && keyTarget > 0) { + setTargetInput(String(keyTarget)); + } + }, [keyTarget, editingTarget]); const startEditingLifecycle = () => { setLcThreshold(config ? String(config.topup_threshold / 1e9) : ''); @@ -64,25 +78,18 @@ export const BuilderInfo: React.FC = ({ builderInfo, serviceSt } }; - const handleExit = async () => { - if (!isLoggedIn) return; - const headers: HeadersInit = { 'Content-Type': 'application/json' }; - const authToken = await getAuthHeader(); - if (authToken) { - headers['Authorization'] = `Bearer ${authToken}`; + const handleTargetSave = async () => { + const target = parseInt(targetInput, 10); + if (!Number.isFinite(target) || target < 1) { + setTargetError('target must be at least 1'); + return; } - setExitState('submitting'); - try { - const resp = await fetch('/api/lifecycle/exit', { method: 'POST', headers }); - if (!resp.ok) { - const body = await resp.json().catch(() => null); - throw new Error(body?.error || `HTTP ${resp.status}`); - } - setExitState('done'); - } catch (err) { - console.error('Failed to submit builder exit:', err); - setExitError(err instanceof Error ? err.message : String(err)); - setExitState('error'); + + const result = await keyActions.setTarget(target); + setTargetError(result.ok ? '' : (result.error ?? 'failed to set target')); + + if (result.ok) { + setEditingTarget(false); } }; @@ -124,6 +131,14 @@ export const BuilderInfo: React.FC = ({ builderInfo, serviceSt
Builder Info
+ {isFleet && ( + + {builderInfo?.keys_active ?? 0} / {keyTarget} keys + + )} {lifecycleAvailable && ( <> {lifecycleEnabled ? ( @@ -196,6 +211,67 @@ export const BuilderInfo: React.FC = ({ builderInfo, serviceSt + {/* Target key count. A fleet is maintained against it: raising it + deposits new keys, lowering it exits surplus ones. */} + {lifecycleAvailable && ( + + Target keys: + + {!editingTarget ? ( + <> + {keyTarget} + {isLoggedIn && ( + + )} + + ) : ( +
+ setTargetInput(e.target.value)} + /> + + +
+ )} + + + )} + {editingTarget && parseInt(targetInput, 10) < keyTarget && ( + + + Lowering the target exits surplus keys, highest index first. An exited builder + key can never be reactivated. + + + )} + {targetError && ( + + {targetError} + + )} + {/* Wallet Info (if lifecycle enabled) */} {builderInfo.lifecycle_enabled && builderInfo.wallet_address && ( @@ -213,10 +289,10 @@ export const BuilderInfo: React.FC = ({ builderInfo, serviceSt {/* CL Balance */} - CL Balance: + {isFleet ? 'CL Balance (all keys):' : 'CL Balance:'} - {formatGwei(builderInfo.cl_balance_gwei)} ETH + {formatGwei(isFleet ? builderInfo.total_balance_gwei : builderInfo.cl_balance_gwei)} ETH @@ -225,18 +301,23 @@ export const BuilderInfo: React.FC = ({ builderInfo, serviceSt Pending Payments: - {builderInfo.pending_payments_gwei > 0 - ? `-${formatGwei(builderInfo.pending_payments_gwei)} ETH` - : '0 ETH'} + {(() => { + const pending = isFleet + ? builderInfo.total_pending_payments_gwei + : builderInfo.pending_payments_gwei; + return pending > 0 ? `-${formatGwei(pending)} ETH` : '0 ETH'; + })()} {/* Effective Balance (shown when pending payments reduce it) */} - {builderInfo.pending_payments_gwei > 0 && ( + {(isFleet ? builderInfo.total_pending_payments_gwei : builderInfo.pending_payments_gwei) > 0 && ( Effective Balance: - {formatGwei(builderInfo.effective_balance_gwei)} ETH + {formatGwei( + isFleet ? builderInfo.total_effective_gwei : builderInfo.effective_balance_gwei, + )} ETH )} @@ -343,43 +424,20 @@ export const BuilderInfo: React.FC = ({ builderInfo, serviceSt - {/* Builder exit (irreversible: submits an exit request to the builder - exit system contract; the exit then proceeds through the exit queue) */} - {lifecycleAvailable && isLoggedIn && (canExit || exitState === 'done') && ( + {/* Per-key operations (deposit, top-up, exit) live on the Builder Keys + page: with a fleet, "exit the builder" has no single meaning. */} + {lifecycleAvailable && (
- {exitState === 'idle' && ( - - )} - {exitState === 'confirm' && ( -
- Exit builder from the builder set? - - -
- )} - {exitState === 'submitting' && ( - - Submitting exit transaction... - - )} - {exitState === 'done' && ( - - Exit submitted — waiting for the exit queue - - )} - {exitState === 'error' && ( -
- Exit failed: {exitError} - - -
- )} + { + e.preventDefault(); + setView('builder-keys'); + }} + > + Manage keys +
)}
diff --git a/pkg/webui/src/components/BuilderKeysTable.tsx b/pkg/webui/src/components/BuilderKeysTable.tsx new file mode 100644 index 00000000..23e9b8ef --- /dev/null +++ b/pkg/webui/src/components/BuilderKeysTable.tsx @@ -0,0 +1,204 @@ +import React, { useState } from 'react'; +import { CopyableHash } from './CopyableHash'; +import type { BuilderKeyState, BuilderKeyStatus } from '../types'; + +interface BuilderKeysTableProps { + keys: BuilderKeyState[]; + loading: boolean; + error: string | null; + target: number; + isLoggedIn: boolean; + busyKey: number | null; + actionError: string | null; + onDeposit: (keyIndex: number) => void; + onTopup: (keyIndex: number) => void; + onExit: (keyIndex: number, lowerTarget: boolean) => void; +} + +const FAR_FUTURE_EPOCH = 18446744073709551615; + +const STATUS_BADGE: Record = { + unused: { className: 'bg-dark', label: 'Unused' }, + depositing: { className: 'bg-warning text-dark', label: 'Depositing' }, + pending: { className: 'bg-info', label: 'Pending' }, + active: { className: 'bg-success', label: 'Active' }, + exiting: { className: 'bg-warning text-dark', label: 'Exiting' }, + exited: { className: 'bg-secondary', label: 'Exited' }, + withdrawn: { className: 'bg-dark', label: 'Withdrawn' }, +}; + +function formatGwei(gwei: number): string { + return (gwei / 1e9).toFixed(4); +} + +export const BuilderKeysTable: React.FC = ({ + keys, + loading, + error, + target, + isLoggedIn, + busyKey, + actionError, + onDeposit, + onTopup, + onExit, +}) => { + // Key index whose exit confirmation is open; the confirmation names the + // consequences because an exit can never be undone. + const [confirmExit, setConfirmExit] = useState(null); + const [lowerTarget, setLowerTarget] = useState(true); + + if (error) { + return
{error}
; + } + + if (loading && keys.length === 0) { + return
Loading builder keys...
; + } + + if (keys.length === 0) { + return
No builder keys derived yet.
; + } + + return ( + <> + {actionError &&
{actionError}
} +
+ + + + + + + + + + + + + {isLoggedIn && } + + + + {keys.map((key) => { + const badge = STATUS_BADGE[key.status] ?? STATUS_BADGE.unused; + const busy = busyKey === key.key_index; + const canExit = key.status === 'active' || key.status === 'pending'; + const canDeposit = key.status === 'unused' || key.status === 'withdrawn'; + const canTopup = key.status === 'active' || key.status === 'pending'; + + return ( + + + + + + + + + + + + {isLoggedIn && ( + + )} + + {confirmExit === key.key_index && ( + + + + )} + + ); + })} + +
#PubkeyStatusBuilder indexBalancePendingEffectiveUsedBids / WonActions
{key.key_index} + + + {badge.label} + + {key.has_builder_index ? key.builder_index : —} + {formatGwei(key.balance_gwei)} + {key.pending_payments_gwei > 0 ? `-${formatGwei(key.pending_payments_gwei)}` : '—'} + {formatGwei(key.effective_balance_gwei)} + {key.use_count} + {key.withdrawable_epoch > 0 && key.withdrawable_epoch < FAR_FUTURE_EPOCH && ( + + (w{key.withdrawable_epoch}) + + )} + + {key.bids_submitted} / {key.bids_won} + + {canDeposit && ( + + )} + {canTopup && ( + + )} + {canExit && ( + + )} + {busy && } +
+
+ Exit builder key #{key.key_index}? This is irreversible — + the key cannot be reactivated until its registry entry is reused by another + builder's deposit. +
+
+
+ setLowerTarget(e.target.checked)} + /> + +
+ + +
+
+
+ + ); +}; diff --git a/pkg/webui/src/components/BuilderKeysView.tsx b/pkg/webui/src/components/BuilderKeysView.tsx new file mode 100644 index 00000000..f3f7c5ca --- /dev/null +++ b/pkg/webui/src/components/BuilderKeysView.tsx @@ -0,0 +1,152 @@ +import React, { useEffect, useState } from 'react'; +import { BuilderKeysTable } from './BuilderKeysTable'; +import { useBuilderKeyActions, useBuilderKeys } from '../hooks/useBuilderKeys'; +import type { BuilderKeysAggregate } from '../types'; + +interface BuilderKeysViewProps { + /** + * Key set pushed by the SSE stream. It is the live source; the REST fetch + * supplies the settings and covers the moment before the first event. + */ + streamKeys: BuilderKeysAggregate | null; + connectionGeneration: number; +} + +function formatGwei(gwei: number): string { + return (gwei / 1e9).toFixed(4); +} + +export const BuilderKeysView: React.FC = ({ + streamKeys, + connectionGeneration, +}) => { + const { keys, aggregate, settings, loading, error, refetch } = useBuilderKeys( + `${connectionGeneration}:${streamKeys?.managed ?? 0}:${streamKeys?.active ?? 0}:${streamKeys?.target ?? 0}`, + ); + const actions = useBuilderKeyActions(); + + const [targetInput, setTargetInput] = useState(''); + const [editingTarget, setEditingTarget] = useState(false); + const [busyKey, setBusyKey] = useState(null); + const [actionError, setActionError] = useState(null); + + useEffect(() => { + if (!editingTarget) { + setTargetInput(String(settings.target_count)); + } + }, [settings.target_count, editingTarget]); + + const runAction = async (keyIndex: number, action: () => Promise<{ ok: boolean; error?: string }>) => { + setBusyKey(keyIndex); + setActionError(null); + + const result = await action(); + if (!result.ok) { + setActionError(result.error ?? 'request failed'); + } + + setBusyKey(null); + refetch(); + }; + + const saveTarget = async () => { + const target = parseInt(targetInput, 10); + if (!Number.isFinite(target) || target < 1) { + setActionError('target must be at least 1'); + return; + } + + setActionError(null); + + const result = await actions.setTarget(target); + if (!result.ok) { + setActionError(result.error ?? 'failed to set target'); + } + + setEditingTarget(false); + refetch(); + }; + + const lowering = editingTarget && parseInt(targetInput, 10) < settings.target_count; + + return ( +
+
+
Builder Keys
+ {aggregate.active} active + {aggregate.managed} managed + target {settings.target_count} + {aggregate.depositing > 0 && ( + + + {aggregate.depositing} depositing + + )} + {!settings.auto_deposit && auto-deposit off} + {!settings.auto_exit && auto-exit off} + +
+ Total effective: + {formatGwei(aggregate.total_effective_gwei)} ETH + + {actions.isLoggedIn && !editingTarget && ( + + )} + + {actions.isLoggedIn && editingTarget && ( + <> + setTargetInput(e.target.value)} + /> + + + + )} +
+
+ + {lowering && ( +
+ + Lowering the target exits surplus keys, highest index first. An exited builder key can + never be reactivated. +
+ )} + +
+ runAction(keyIndex, () => actions.depositKey(keyIndex))} + onTopup={(keyIndex) => runAction(keyIndex, () => actions.topupKey(keyIndex))} + onExit={(keyIndex, lowerTarget) => + runAction(keyIndex, () => actions.exitKey(keyIndex, lowerTarget)) + } + /> +
+
+ ); +}; diff --git a/pkg/webui/src/components/ConfigPanel.tsx b/pkg/webui/src/components/ConfigPanel.tsx index 960f4682..96ae0f73 100644 --- a/pkg/webui/src/components/ConfigPanel.tsx +++ b/pkg/webui/src/components/ConfigPanel.tsx @@ -1,5 +1,6 @@ import React, { useState, useEffect } from 'react'; import { useAuthContext } from '../context/AuthContext'; +import { KeySelectionSection } from './KeySelectionSection'; import type { Config, EPBSConfig, ServiceStatus } from '../types'; interface ConfigPanelProps { @@ -268,6 +269,8 @@ export const ConfigPanel: React.FC = ({ config, serviceStatus
)} + + )} diff --git a/pkg/webui/src/components/HeaderNav.tsx b/pkg/webui/src/components/HeaderNav.tsx index 78c24a66..865c8b15 100644 --- a/pkg/webui/src/components/HeaderNav.tsx +++ b/pkg/webui/src/components/HeaderNav.tsx @@ -9,6 +9,7 @@ const NAV_ITEMS: Array<{ view: ViewType; label: string; requiresAuth?: boolean } { view: 'dashboard', label: 'Dashboard' }, { view: 'action-plan', label: 'Action Plan' }, { view: 'bids-won', label: 'Bids Won' }, + { view: 'builder-keys', label: 'Builder Keys' }, { view: 'validators', label: 'Validators' }, { view: 'proposer-preferences', label: 'Proposer Prefs' }, { view: 'builder-preferences', label: 'Builder Prefs' }, diff --git a/pkg/webui/src/components/KeySelectionSection.tsx b/pkg/webui/src/components/KeySelectionSection.tsx new file mode 100644 index 00000000..0baa6502 --- /dev/null +++ b/pkg/webui/src/components/KeySelectionSection.tsx @@ -0,0 +1,236 @@ +import React, { useState, useEffect } from 'react'; +import { useAuthContext } from '../context/AuthContext'; +import type { Config, EPBSConfig } from '../types'; + +interface KeySelectionSectionProps { + config: Config | null; +} + +// Which built candidate payload the slot's p2p bids commit to. "all" gossips a +// bid for every built candidate, so one slot can carry bids on several parents. +const CANDIDATE_LABELS: Record = { + auto: 'auto (match the chain view)', + all: 'all built candidates', + parent_full: 'parent (full)', + parent_empty: 'parent (empty payload)', + grandparent_full: 'grandparent (reorg)', + grandparent_empty: 'grandparent (empty payload)', +}; + +// Order the fleet is walked in when picking the key for a bid. +const STRATEGY_LABELS: Record = { + round_robin: 'round robin', + single: 'single (primary key only)', + random: 'random', + least_used: 'least used', +}; + +interface KeySelectionForm { + bid_candidate: string; + key_strategy: string; + bid_keys_per_slot: number; + bid_keys_per_step: number; +} + +const DEFAULT_FORM: KeySelectionForm = { + bid_candidate: 'auto', + key_strategy: 'round_robin', + bid_keys_per_slot: 0, + bid_keys_per_step: 1, +}; + +const formFromConfig = (epbs: EPBSConfig | undefined): KeySelectionForm => ({ + bid_candidate: epbs?.bid_candidate || DEFAULT_FORM.bid_candidate, + key_strategy: epbs?.key_strategy || DEFAULT_FORM.key_strategy, + bid_keys_per_slot: epbs?.bid_keys_per_slot ?? DEFAULT_FORM.bid_keys_per_slot, + bid_keys_per_step: epbs?.bid_keys_per_step ?? DEFAULT_FORM.bid_keys_per_step, +}); + +// KeySelectionSection is the global bid key selection policy, rendered as a +// section of the ePBS Bidder card: which candidate payloads the slot bids on +// and how many of the managed keys are spent doing it. Per-slot overrides live +// in the action plan; edits here go through the generic path-based settings +// endpoint with epbs.* keys. +export const KeySelectionSection: React.FC = ({ config }) => { + const { isLoggedIn, getAuthHeader } = useAuthContext(); + const [editing, setEditing] = useState(false); + + const epbs = config?.epbs; + + const [form, setForm] = useState(DEFAULT_FORM); + + useEffect(() => { + if (!editing) { + setForm(formFromConfig(epbs)); + } + }, [epbs, editing]); + + const handleSave = async (e: React.FormEvent) => { + e.preventDefault(); + + const headers: HeadersInit = { 'Content-Type': 'application/json' }; + const authToken = await getAuthHeader(); + if (authToken) { + headers['Authorization'] = `Bearer ${authToken}`; + } + + try { + const response = await fetch('/api/config/settings', { + method: 'POST', + headers, + body: JSON.stringify({ + 'epbs.bid_candidate': form.bid_candidate, + 'epbs.key_strategy': form.key_strategy, + 'epbs.bid_keys_per_slot': form.bid_keys_per_slot, + 'epbs.bid_keys_per_step': form.bid_keys_per_step, + }), + }); + const result = await response.json(); + if (result.error) { + alert('Failed to update: ' + result.error); + return; + } + setEditing(false); + } catch (err) { + alert('Error: ' + err); + } + }; + + return ( + <> +
+
Key Selection
+ {isLoggedIn && !editing && ( + + )} +
+
+ Gossip ignores every bid a builder makes after its first for a slot, so + each bid spends a managed key that has not bid yet. These settings decide + how many of the fleet a slot spends, and on which candidate payloads. +
+ + {!editing ? ( +
+
+
+
Bid Candidate
+
+ {CANDIDATE_LABELS[epbs?.bid_candidate || 'auto'] ?? epbs?.bid_candidate} +
+
+
+
+
+
Key Strategy
+
+ {STRATEGY_LABELS[epbs?.key_strategy || 'round_robin'] ?? epbs?.key_strategy} +
+
+
+
+
+
Keys / Slot
+
+ {epbs?.bid_keys_per_slot ? epbs.bid_keys_per_slot : 'whole fleet'} +
+
+
+
+
+
Keys / Step
+
+ {epbs?.bid_keys_per_step ? epbs.bid_keys_per_step : 'all remaining'} +
+
+
+
+ ) : ( +
+
+ + +
+ Which built payload the bids commit to. all bids + every candidate the slot built, spending keys per candidate — a + deliberate multi-parent gossip scenario, since only one of those + parents can end up canonical. +
+
+ +
+ + +
+ Order the active keys are offered in. Balance is a preference, not + a filter: an underfunded key sorts last but is still used when + nothing else can cover the bid. +
+
+ +
+ + + setForm({ ...form, bid_keys_per_slot: parseInt(e.target.value) || 0 }) + } + required + /> +
+ Caps how many distinct keys one slot may spend. 0 means every + active key is available. +
+
+ +
+ + + setForm({ ...form, bid_keys_per_step: parseInt(e.target.value) || 0 }) + } + required + /> +
+ Keys spent on a payload per bid interval, each bidding one increase + above the last. 1 walks the fleet up the price ladder; 0 spends + every remaining key in a single step. +
+
+ +
+ + +
+
+ )} + + ); +}; diff --git a/pkg/webui/src/components/SlotGraph.tsx b/pkg/webui/src/components/SlotGraph.tsx index 0a53f697..4d02713c 100644 --- a/pkg/webui/src/components/SlotGraph.tsx +++ b/pkg/webui/src/components/SlotGraph.tsx @@ -1073,6 +1073,11 @@ export const SlotGraph: React.FC = ({ copyValue: bid.blockHash }] : []), ...candidateRows(candidateBuilds, bid.blockHash), + ...(bid.keyIndex !== undefined ? [{ + label: 'Builder Key', + value: `#${bid.keyIndex}` + + (bid.builderIndex !== undefined ? ` (builder ${bid.builderIndex})` : '') + }] : []), ...(bid.parentBlockHash ? [{ label: 'Parent Hash', value: truncateHash(bid.parentBlockHash), diff --git a/pkg/webui/src/hooks/useBuilderKeys.ts b/pkg/webui/src/hooks/useBuilderKeys.ts new file mode 100644 index 00000000..dc2439a0 --- /dev/null +++ b/pkg/webui/src/hooks/useBuilderKeys.ts @@ -0,0 +1,134 @@ +import { useCallback, useEffect, useRef, useState } from 'react'; +import { useAuthContext } from '../context/AuthContext'; +import type { + BuilderKeyState, + BuilderKeysAggregate, + BuilderKeysResponse, + BuilderKeysSettings, +} from '../types'; + +const EMPTY_AGGREGATE: BuilderKeysAggregate = { + target: 0, + managed: 0, + unused: 0, + depositing: 0, + pending: 0, + active: 0, + exiting: 0, + exited: 0, + withdrawn: 0, + total_balance_gwei: 0, + total_pending_payments_gwei: 0, + total_effective_gwei: 0, +}; + +const EMPTY_SETTINGS: BuilderKeysSettings = { + target_count: 0, + max_index: 0, + auto_deposit: false, + auto_exit: false, +}; + +/** + * Loads the managed builder key set. The SSE `builder_keys` event carries the + * whole set on every change, so callers pass a change counter (e.g. the stream's + * key generation) to refetch settings alongside it. + */ +export function useBuilderKeys(changeToken?: unknown) { + const [keys, setKeys] = useState([]); + const [aggregate, setAggregate] = useState(EMPTY_AGGREGATE); + const [settings, setSettings] = useState(EMPTY_SETTINGS); + const [loading, setLoading] = useState(true); + const [error, setError] = useState(null); + const initialFetchDone = useRef(false); + + const fetchKeys = useCallback(async () => { + try { + const response = await fetch('/api/buildoor/builder-keys'); + + if (response.status === 404) { + setKeys([]); + setError('builder key registry not available'); + return; + } + + if (!response.ok) { + throw new Error(`Failed to fetch builder keys: ${response.statusText}`); + } + + const data: BuilderKeysResponse = await response.json(); + setKeys(data.keys || []); + setAggregate(data.aggregate ?? EMPTY_AGGREGATE); + setSettings(data.settings ?? EMPTY_SETTINGS); + setError(null); + } catch (err) { + setError(err instanceof Error ? err.message : 'Unknown error'); + } finally { + if (!initialFetchDone.current) { + initialFetchDone.current = true; + setLoading(false); + } + } + }, []); + + useEffect(() => { + fetchKeys(); + }, [fetchKeys, changeToken]); + + return { keys, aggregate, settings, loading, error, refetch: fetchKeys }; +} + +/** Result of a mutating builder key request. */ +export interface KeyActionResult { + ok: boolean; + error?: string; +} + +/** + * Provides the mutating builder key actions (target, deposit, top-up, exit). + * Every call carries the auth header the rest of the UI uses. + */ +export function useBuilderKeyActions() { + const { isLoggedIn, getAuthHeader } = useAuthContext(); + + const post = useCallback( + async (path: string, body?: unknown): Promise => { + if (!isLoggedIn) { + return { ok: false, error: 'not logged in' }; + } + + const headers: HeadersInit = { 'Content-Type': 'application/json' }; + const authToken = await getAuthHeader(); + if (authToken) { + headers['Authorization'] = `Bearer ${authToken}`; + } + + try { + const response = await fetch(path, { + method: 'POST', + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + + if (!response.ok) { + const payload = await response.json().catch(() => null); + return { ok: false, error: payload?.error || `HTTP ${response.status}` }; + } + + return { ok: true }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : 'Unknown error' }; + } + }, + [isLoggedIn, getAuthHeader], + ); + + return { + isLoggedIn, + setTarget: (target: number) => post('/api/buildoor/builder-keys/target', { target }), + depositKey: (keyIndex: number) => post(`/api/buildoor/builder-keys/${keyIndex}/deposit`), + topupKey: (keyIndex: number) => post(`/api/buildoor/builder-keys/${keyIndex}/topup`, {}), + exitKey: (keyIndex: number, lowerTarget: boolean) => + post(`/api/buildoor/builder-keys/${keyIndex}/exit`, { lower_target: lowerTarget }), + }; +} diff --git a/pkg/webui/src/hooks/useEventStream.ts b/pkg/webui/src/hooks/useEventStream.ts index f994416f..4dad27cf 100644 --- a/pkg/webui/src/hooks/useEventStream.ts +++ b/pkg/webui/src/hooks/useEventStream.ts @@ -1,5 +1,5 @@ import { useState, useEffect, useCallback, useRef, useSyncExternalStore } from 'react'; -import type { Config, ChainInfo, Stats, SlotState, LogEvent, OurBid, ExternalBid, BuilderInfo, HeadVoteDataPoint, ServiceStatus, RevealAttempt, VoteCoverage } from '../types'; +import type { Config, ChainInfo, Stats, SlotState, LogEvent, OurBid, ExternalBid, BuilderInfo, HeadVoteDataPoint, ServiceStatus, RevealAttempt, VoteCoverage, BuilderKeyState, BuilderKeysAggregate } from '../types'; // --------------------------------------------------------------------------- // Module-level SSE fan-out: lets other hooks/components subscribe to raw @@ -74,6 +74,8 @@ interface UseEventStreamResult { chainInfo: ChainInfo | null; stats: Stats | null; builderInfo: BuilderInfo | null; + builderKeys: BuilderKeyState[]; + builderKeysAggregate: BuilderKeysAggregate | null; serviceStatus: ServiceStatus | null; voteCoverage: VoteCoverage | null; currentSlot: number; @@ -91,6 +93,9 @@ export function useEventStream(): UseEventStreamResult { const [chainInfo, setChainInfo] = useState(null); const [stats, setStats] = useState(null); const [builderInfo, setBuilderInfo] = useState(null); + const [builderKeys, setBuilderKeys] = useState([]); + const [builderKeysAggregate, setBuilderKeysAggregate] = + useState(null); const [serviceStatus, setServiceStatus] = useState(null); const [voteCoverage, setVoteCoverage] = useState(null); const [currentSlot, setCurrentSlot] = useState(0); @@ -233,6 +238,18 @@ export function useEventStream(): UseEventStreamResult { setBuilderInfo(event.data as BuilderInfo); break; + case 'builder_keys': { + // Every event carries the whole key set, so the latest one always + // wins and a dropped event is simply superseded. + const data = event.data as { + keys: BuilderKeyState[]; + aggregate: BuilderKeysAggregate; + }; + setBuilderKeys(data.keys || []); + setBuilderKeysAggregate(data.aggregate ?? null); + break; + } + case 'service_status': setServiceStatus(event.data as ServiceStatus); break; @@ -423,7 +440,8 @@ export function useEventStream(): UseEventStreamResult { const data = event.data as { slot: number; block_hash: string; value: number; bid_count: number; success: boolean; error?: string; warning?: string; execution_payment?: number; fee_recipient?: string; - gas_limit?: number; builder_index?: number; parent_block_hash?: string; + gas_limit?: number; builder_index?: number; key_index?: number; + parent_block_hash?: string; parent_block_root?: string; num_blob_commitments?: number; }; const bidSuccess = data.success !== false; @@ -462,6 +480,7 @@ export function useEventStream(): UseEventStreamResult { feeRecipient: data.fee_recipient, gasLimit: data.gas_limit, builderIndex: data.builder_index, + keyIndex: data.key_index, parentBlockHash: data.parent_block_hash, parentBlockRoot: data.parent_block_root, numBlobCommitments: data.num_blob_commitments @@ -773,6 +792,8 @@ export function useEventStream(): UseEventStreamResult { chainInfo, stats, builderInfo, + builderKeys, + builderKeysAggregate, serviceStatus, voteCoverage, currentSlot, diff --git a/pkg/webui/src/overview/InstanceCard.tsx b/pkg/webui/src/overview/InstanceCard.tsx index 2f8ae35c..053bcf55 100644 --- a/pkg/webui/src/overview/InstanceCard.tsx +++ b/pkg/webui/src/overview/InstanceCard.tsx @@ -112,6 +112,75 @@ const Row: React.FC = ({ label, children, mono, title }) => ( ); +// BuilderSummary renders the instance's builder identity. A managed fleet is +// shown as a count with the individual keys behind a disclosure, since listing +// every pubkey inline makes the card unreadable once an instance runs hundreds +// of them. Instances predating the managed key set send no `builders` object, +// so the single primary key is rendered instead. +const BuilderSummary: React.FC<{ data: OverviewResponse }> = ({ data }) => { + const registeredBadge = ( + + {data.is_registered ? 'registered' : 'unregistered'} + + ); + + const builders = data.builders; + + if (!builders) { + // Legacy instance: one key, reported at the top level. + if (!data.builder_pubkey) return –; + + return ( + <> + + {typeof data.builder_index === 'number' && ( + idx {data.builder_index} + )} + {registeredBadge} + + ); + } + + const keys = builders.keys ?? []; + + return ( + <> + {builders.active} + / {builders.count} active + {builders.target > 0 && builders.target !== builders.count && ( + (target {builders.target}) + )} + {registeredBadge} + + {keys.length > 0 && ( +
+ + {keys.length} {keys.length === 1 ? 'key' : 'keys'} + +
+ {keys.map((key) => ( +
+ #{key.key_index} + + + {key.has_builder_index ? `idx ${key.builder_index}` : '–'} + + {key.status} +
+ ))} +
+
+ )} + + ); +}; + interface Props { host: OverviewHost; status: InstanceStatus; @@ -181,26 +250,8 @@ export const InstanceCard: React.FC = ({ host, status }) => { )} - - {data.builder_pubkey ? ( - <> - - {typeof data.builder_index === 'number' && ( - idx {data.builder_index} - )} - - {data.is_registered ? 'registered' : 'unregistered'} - - - ) : ( - – - )} + + {data.version && ( diff --git a/pkg/webui/src/overview/types.ts b/pkg/webui/src/overview/types.ts index 04f9c6d9..ed75868c 100644 --- a/pkg/webui/src/overview/types.ts +++ b/pkg/webui/src/overview/types.ts @@ -39,6 +39,29 @@ export interface OverviewStats { builder_api_registered_validators: number; } +// One managed builder key, as listed by the overview endpoint. +export interface OverviewBuilderKey { + key_index: number; + pubkey: string; + status: string; + builder_index: number; + has_builder_index: boolean; + balance_gwei?: number; +} + +// The managed fleet. Absent on instances predating the managed key set, which +// is why builder_pubkey/builder_index below are still populated (primary key). +export interface OverviewBuilders { + count: number; + target: number; + active: number; + indexes: number[]; + keys?: OverviewBuilderKey[]; + total_balance_gwei: number; + total_pending_payments_gwei: number; + total_effective_gwei: number; +} + export interface OverviewResponse { version: string; running: boolean; @@ -50,6 +73,7 @@ export interface OverviewResponse { services: OverviewServices; balances: OverviewBalances; stats: OverviewStats; + builders?: OverviewBuilders; } export type InstanceStatus = diff --git a/pkg/webui/src/pages/BuilderKeysPage.tsx b/pkg/webui/src/pages/BuilderKeysPage.tsx new file mode 100644 index 00000000..bf5e2ca5 --- /dev/null +++ b/pkg/webui/src/pages/BuilderKeysPage.tsx @@ -0,0 +1,16 @@ +import React from 'react'; +import { BuilderKeysView } from '../components/BuilderKeysView'; +import { useEventStream } from '../hooks/useEventStream'; + +const BuilderKeysPage: React.FC = () => { + const { builderKeysAggregate, connectionGeneration } = useEventStream(); + + return ( + + ); +}; + +export default BuilderKeysPage; diff --git a/pkg/webui/src/stores/viewStore.ts b/pkg/webui/src/stores/viewStore.ts index 778c3654..a65be24d 100644 --- a/pkg/webui/src/stores/viewStore.ts +++ b/pkg/webui/src/stores/viewStore.ts @@ -4,6 +4,7 @@ export type ViewType = | 'dashboard' | 'action-plan' | 'bids-won' + | 'builder-keys' | 'validators' | 'proposer-preferences' | 'builder-preferences' @@ -14,6 +15,7 @@ const VIEW_PATHS: Record = { dashboard: '/', 'action-plan': '/action-plan', 'bids-won': '/bids-won', + 'builder-keys': '/builder-keys', validators: '/validators', 'proposer-preferences': '/proposer-preferences', 'builder-preferences': '/builder-preferences', diff --git a/pkg/webui/src/types.ts b/pkg/webui/src/types.ts index bc830840..69d8908d 100644 --- a/pkg/webui/src/types.ts +++ b/pkg/webui/src/types.ts @@ -72,6 +72,10 @@ export interface EPBSConfig { bid_interval: number; bid_subsidy: number; payload_build_delay?: number; + bid_candidate?: string; + key_strategy?: string; + bid_keys_per_slot?: number; + bid_keys_per_step?: number; } export interface ServiceStatus { @@ -116,6 +120,71 @@ export interface BuilderInfo { wallet_balance_wei?: string; deposit_epoch: number; withdrawable_epoch: number; + // Fleet summary of the managed builder key set. Equal to the fields above on + // a single-key deployment. + key_count: number; + key_target: number; + keys_active: number; + total_balance_gwei: number; + total_pending_payments_gwei: number; + total_effective_gwei: number; +} + +export type BuilderKeyStatus = + | 'unused' + | 'depositing' + | 'pending' + | 'active' + | 'exiting' + | 'exited' + | 'withdrawn'; + +export interface BuilderKeyState { + key_index: number; + pubkey: string; + status: BuilderKeyStatus; + builder_index: number; + has_builder_index: boolean; + balance_gwei: number; + pending_payments_gwei: number; + balance_adjustment_gwei: number; + effective_balance_gwei: number; + deposit_epoch: number; + withdrawable_epoch: number; + use_count: number; + last_deposit_at?: number; + last_exit_at?: number; + last_topup_epoch?: number; + bids_submitted: number; + bids_won: number; +} + +export interface BuilderKeysAggregate { + target: number; + managed: number; + unused: number; + depositing: number; + pending: number; + active: number; + exiting: number; + exited: number; + withdrawn: number; + total_balance_gwei: number; + total_pending_payments_gwei: number; + total_effective_gwei: number; +} + +export interface BuilderKeysSettings { + target_count: number; + max_index: number; + auto_deposit: boolean; + auto_exit: boolean; +} + +export interface BuilderKeysResponse { + keys: BuilderKeyState[]; + aggregate: BuilderKeysAggregate; + settings: BuilderKeysSettings; } export interface SlotStartEvent { @@ -139,6 +208,8 @@ export interface BidSubmittedEvent { timestamp: number; success: boolean; error?: string; + builder_index?: number; + key_index?: number; } export interface HeadReceivedEvent { @@ -418,6 +489,9 @@ export interface OurBid { feeRecipient?: string; gasLimit?: number; builderIndex?: number; + // The managed builder key that signed the bid: with several keys bidding one + // slot, the builder index alone does not say which of ours it was. + keyIndex?: number; parentBlockHash?: string; parentBlockRoot?: string; numBlobCommitments?: number; @@ -793,6 +867,7 @@ export interface SlotBidAttempt { fee_recipient?: string; gas_limit?: number; builder_index?: number; + key_index?: number; num_blob_commitments?: number; error?: string; at: string; diff --git a/pkg/webui/webui.go b/pkg/webui/webui.go index af93b61a..b4332b01 100644 --- a/pkg/webui/webui.go +++ b/pkg/webui/webui.go @@ -12,6 +12,7 @@ import ( "github.com/ethpandaops/go-eth2-client/spec/phase0" "github.com/ethpandaops/buildoor/pkg/action_plan" + "github.com/ethpandaops/buildoor/pkg/builder_keys" "github.com/ethpandaops/buildoor/pkg/builderapi" "github.com/ethpandaops/buildoor/pkg/chain" "github.com/ethpandaops/buildoor/pkg/config" @@ -43,7 +44,7 @@ var ( staticEmbedFS embed.FS ) -func StartHttpServer(frontendConfig *types.FrontendConfig, settingsSvc *config.Service, stateDB *db.Database, builderSvc *payload_builder.Service, epbsSvc *p2p_bidder.Service, lifecycleMgr *lifecycle.Manager, chainSvc chain.Service, validatorStore *memstore.Store[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration], builderAPISvc *builderapi.Server, propPrefSvc *payload_bidder.ProposerPreferencesService, valRanges *validatorranges.Resolver, revealSvc *payload_bidder.RevealService, inclusionTracker *payload_bidder.InclusionTracker, payments *payload_bidder.PaymentTracker, planSvc *action_plan.PlanService, resultTracker *slot_results.Tracker) *api.APIHandler { +func StartHttpServer(frontendConfig *types.FrontendConfig, settingsSvc *config.Service, stateDB *db.Database, builderSvc *payload_builder.Service, epbsSvc *p2p_bidder.Service, lifecycleMgr *lifecycle.Manager, keys *builder_keys.Registry, chainSvc chain.Service, validatorStore *memstore.Store[phase0.BLSPubKey, *apiv1.SignedValidatorRegistration], builderAPISvc *builderapi.Server, propPrefSvc *payload_bidder.ProposerPreferencesService, valRanges *validatorranges.Resolver, revealSvc *payload_bidder.RevealService, inclusionTracker *payload_bidder.InclusionTracker, payments *payload_bidder.PaymentTracker, planSvc *action_plan.PlanService, resultTracker *slot_results.Tracker) *api.APIHandler { authHandler, err := auth.NewAuthHandler(context.Background(), frontendConfig.AuthProviderURL) if err != nil { logrus.WithError(err).Fatal("failed to initialize auth handler") @@ -61,7 +62,7 @@ func StartHttpServer(frontendConfig *types.FrontendConfig, settingsSvc *config.S } // API routes - apiHandler := api.NewAPIHandler(authHandler, settingsSvc, stateDB, builderSvc, epbsSvc, lifecycleMgr, chainSvc, validatorStore, builderAPISvc, propPrefSvc, valRanges, revealSvc, inclusionTracker, payments, planSvc, resultTracker) + apiHandler := api.NewAPIHandler(authHandler, settingsSvc, stateDB, builderSvc, epbsSvc, lifecycleMgr, keys, chainSvc, validatorStore, builderAPISvc, propPrefSvc, valRanges, revealSvc, inclusionTracker, payments, planSvc, resultTracker) apiRouter := router.PathPrefix("/api").Subrouter() apiRouter.HandleFunc("/version", apiHandler.GetVersion).Methods("GET") apiRouter.HandleFunc("/status", apiHandler.GetStatus).Methods(http.MethodGet) @@ -105,6 +106,13 @@ func StartHttpServer(frontendConfig *types.FrontendConfig, settingsSvc *config.S apiRouter.HandleFunc("/buildoor/validators", apiHandler.GetValidators).Methods(http.MethodGet) apiRouter.HandleFunc("/buildoor/bids-won", apiHandler.GetBidsWon).Methods(http.MethodGet) apiRouter.HandleFunc("/buildoor/builder-api-status", apiHandler.GetBuilderAPIStatus).Methods(http.MethodGet) + + // Managed builder key set + apiRouter.HandleFunc("/buildoor/builder-keys", apiHandler.GetBuilderKeys).Methods(http.MethodGet) + apiRouter.HandleFunc("/buildoor/builder-keys/target", apiHandler.UpdateBuilderKeyTarget).Methods(http.MethodPost) + apiRouter.HandleFunc("/buildoor/builder-keys/{index:[0-9]+}/deposit", apiHandler.DepositBuilderKey).Methods(http.MethodPost) + apiRouter.HandleFunc("/buildoor/builder-keys/{index:[0-9]+}/topup", apiHandler.TopupBuilderKey).Methods(http.MethodPost) + apiRouter.HandleFunc("/buildoor/builder-keys/{index:[0-9]+}/exit", apiHandler.ExitBuilderKey).Methods(http.MethodPost) apiRouter.HandleFunc("/buildoor/overview", apiHandler.GetOverview).Methods(http.MethodGet, http.MethodOptions) apiRouter.HandleFunc("/buildoor/proposer-preferences", apiHandler.GetProposerPreferences).Methods(http.MethodGet) apiRouter.HandleFunc("/buildoor/builder-preferences", apiHandler.GetBuilderPreferences).Methods(http.MethodGet)