server: make etcd start timeout a no-progress timeout instead of a fixed deadline

[bufferflies]

Enhancement Task

Background

server.EtcdStartTimeout (default 5m) is currently applied as a fixed overall deadline in (*Server).startEtcd:

newCtx, cancel := context.WithTimeout(ctx, EtcdStartTimeout)
...
select {
case <-etcd.Server.ReadyNotify():
case <-newCtx.Done():
    return errs.ErrCancelStartEtcd.FastGenByArgs()
}

The deadline counts down regardless of whether etcd is making progress.

Problem

When a PD/etcd member restarts and has to catch up a large raft log (applying a backlog of committed entries after a restore/restart), reaching ReadyNotify() can legitimately take longer than 5 minutes. In that case:

• startEtcd returns ErrCancelStartEtcd and the process exits, even though etcd was healthy and steadily making progress;
• under a process supervisor (systemd/k8s), it restarts, replays from the start again, hits the same fixed deadline, and loops forever — the node can never come up.

The timeout cannot distinguish a slow-but-healthy startup from a genuine hang (e.g. a removed member that can never rejoin the quorum).

Proposal

Treat EtcdStartTimeout as a no-progress (stall) timeout rather than an absolute deadline:

• Poll etcd.Server.AppliedIndex() as a liveness signal while waiting for ReadyNotify().
• Keep waiting as long as the applied index advances (member is applying its raft log → healthy).
• Give up only when there is no apply progress for EtcdStartTimeout → genuine hang, still fail fast.
• Continue to honor parent ctx cancellation for normal shutdown.

AppliedIndex is the reliable progress signal for the catch-up phase (committed index may already sit at the target while the state machine is still applying, so it can look "stuck" while healthy).

Known limitation

This covers the "wait until ready" phase after embed.StartEtcd returns. The bulk snapshot load / WAL replay happens inside embed.StartEtcd, which takes no ctx and runs before this wait — so that phase is neither interrupted nor observed by this change. Bounding/observing it would require running StartEtcd in a goroutine and is out of scope here.

Origin / history

The 5-minute etcd start timeout dates back to:

• server, client: fix hanging problem when etcd failed to start #1267 — first introduced it as a private const etcdStartTimeout = time.Minute * 5, wrapping startEtcd in a fixed context deadline to fix a hang where etcd got stuck on startup (port listened but not serving), which also hung the client.
• *: add keyvisual service #1948 — promoted it to the exported var EtcdStartTimeout = time.Minute * 5 (so tests can override it); the fixed-deadline semantics were carried over unchanged.