Skip to content

server/grpc: Hardcoded 1s graceful shutdown timeout causes active gRPC streams to be forcefully terminated #162

@Grantmartin2002

Description

@Grantmartin2002

Problem

The gRPC server plugin has a hardcoded 1-second timeout for graceful shutdown in both v4 and v5:

select {
case <-exit:
case <-time.After(time.Second):
    g.srv.Stop()
}

After calling GracefulStop(), the server waits only 1 second before calling Stop(), which forcefully terminates all active connections. This is problematic for applications with long-lived gRPC streams (e.g., server-side streaming RPCs) that need more than 1 second to drain.

In Kubernetes environments, this causes ECONNRESET errors for clients with active streams during pod rolling updates, even when terminationGracePeriodSeconds is configured to allow for graceful draining.

Proposed Solution

Make the graceful shutdown timeout configurable via a new GracefulStopTimeout server option, defaulting to time.Second for backwards compatibility.

Usage:

import (
    grpcsrv "github.com/go-micro/plugins/v4/server/grpc"
)

srv := micro.NewService(
    micro.Server(grpcsrv.NewServer(
        grpcsrv.GracefulStopTimeout(3 * time.Minute),
    )),
)

I have a PR ready that implements this for both v4 and v5.

Environment

  • go-micro plugins v4 (v1.2.0) and v5
  • Kubernetes with gRPC streaming services
  • Observed with terminationGracePeriodSeconds: 120 — streams still killed after 1s

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions