Problem
The gRPC server plugin has a hardcoded 1-second timeout for graceful shutdown in both v4 and v5:
select {
case <-exit:
case <-time.After(time.Second):
g.srv.Stop()
}
After calling GracefulStop(), the server waits only 1 second before calling Stop(), which forcefully terminates all active connections. This is problematic for applications with long-lived gRPC streams (e.g., server-side streaming RPCs) that need more than 1 second to drain.
In Kubernetes environments, this causes ECONNRESET errors for clients with active streams during pod rolling updates, even when terminationGracePeriodSeconds is configured to allow for graceful draining.
Proposed Solution
Make the graceful shutdown timeout configurable via a new GracefulStopTimeout server option, defaulting to time.Second for backwards compatibility.
Usage:
import (
grpcsrv "github.com/go-micro/plugins/v4/server/grpc"
)
srv := micro.NewService(
micro.Server(grpcsrv.NewServer(
grpcsrv.GracefulStopTimeout(3 * time.Minute),
)),
)
I have a PR ready that implements this for both v4 and v5.
Environment
- go-micro plugins v4 (
v1.2.0) and v5
- Kubernetes with gRPC streaming services
- Observed with
terminationGracePeriodSeconds: 120 — streams still killed after 1s
Problem
The gRPC server plugin has a hardcoded 1-second timeout for graceful shutdown in both v4 and v5:
After calling
GracefulStop(), the server waits only 1 second before callingStop(), which forcefully terminates all active connections. This is problematic for applications with long-lived gRPC streams (e.g., server-side streaming RPCs) that need more than 1 second to drain.In Kubernetes environments, this causes
ECONNRESETerrors for clients with active streams during pod rolling updates, even whenterminationGracePeriodSecondsis configured to allow for graceful draining.Proposed Solution
Make the graceful shutdown timeout configurable via a new
GracefulStopTimeoutserver option, defaulting totime.Secondfor backwards compatibility.Usage:
I have a PR ready that implements this for both v4 and v5.
Environment
v1.2.0) and v5terminationGracePeriodSeconds: 120— streams still killed after 1s