Head monitor service resilience improvements

[jimmygchen]

Description

Follow-up improvements for the beacon head monitor service introduced in #7892. The current implementation is a solid MVP, but several enhancements were identified during review that would improve resilience and robustness.

Improvements

1. Per-stream error handling and retry

Currently, if a beacon node is restarted after the VC starts, the event stream for that node is never re-established until the VC itself restarts. The service only restarts when all streams fail.

The current error handling returns from the entire function on any stream error:

lighthouse/validator_client/beacon_node_fallback/src/beacon_head_monitor.rs

Lines 173 to 178 in 0e9dab3

	return Err("Head monitoring service channel closed".into());
	}
	}
	Ok(event) => {
	warn!(
	event_kind = event.topic_name(),

A more resilient approach would handle errors per-stream independently and retry failed streams, so a flaky BN doesn't disrupt monitoring of healthy BNs.

2. Handle dynamic beacon node list updates

When update_candidates_list is called via API with a new BN list, stale SSE connections are maintained and the index mapping can become outdated. For example, if connected to nodes [A, B, C] and the list changes to [B, C, D], head events from B would still report index 1 (its original position) even though B is now at index 0 in the candidate list.

Options discussed:

• Trigger service restart when candidate list changes
• Key cache on endpoint/ID rather than index

The cache purge mechanism exists but only runs on service restart:

lighthouse/validator_client/beacon_node_fallback/src/beacon_head_monitor.rs

Lines 97 to 98 in 0e9dab3

	};

3. Unit tests

Add unit tests for:

• BeaconHeadCache methods
• first_success_or_fallback function (similar to existing first_success_should_try_nodes_in_order test)

Additional Info

• Discussion: Use events API to eager send attestations  #7892
• Related to parallel BN handling: Consider flag for making VC first_success parallel #7908

[peter941221]

I dug into the current head monitor implementation (a select_all over per-BN SSE streams) and I think the two issues in #8741 are real:

1. Per-stream disconnect: if a beacon node restarts, that stream can end and gets silently dropped from select_all, so it never reconnects unless the whole monitor restarts.
2. Candidate list update: update_candidates_list re-enumerates indices; any stale SSE connection can keep reporting the old index mapping.

I opened a PR implementing a minimal/merge-first fix here:

• PR: VC: Reconnect head SSE streams and restart on candidate updates #9018

Summary of changes:

• Per-candidate self-reconnecting head stream (end/error triggers cache removal + bounded retry)
• Bounded connect timeout around get_events so a hung reconnect can't stall progress
• watch generation counter bumped by update_candidates_list, observed by the monitor to restart cleanly (indices + cache stay consistent)
• Failure logs are throttled (warn at 1,2,4,8,... consecutive failures; debug otherwise); stream end is debug-only to avoid false-alarm warnings

Known limitation / follow-up idea:

• This bounds connect time, but doesn't add an idle/read-timeout on established SSE connections. If a stream becomes stuck-silent without ending/erroring, it may not reconnect until the underlying connection fails.