What would you like to be added: An interceptor to help server recovery from panic
Why is this needed: pipecd server crashes and restarts when panic occurs, we should avoid this happens in production environment
There are 2 ways to do this:
- directly use https://github.com/grpc-ecosystem/go-grpc-middleware/blob/v1.4.0/recovery/interceptors.go
- implement by our own to avoid dependency
// PanicRecoveryUnaryServerInterceptor returns a gRPC unary interceptor that
// recovers from panics in request handlers to prevent the server from crashing.
func PanicRecoveryUnaryServerInterceptor(logger *zap.Logger) grpc.UnaryServerInterceptor {
return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (_ interface{}, err error) {
panicked := true
defer func() {
if r := recover(); r != nil || panicked {
var panicErr error
switch v := r.(type) {
case error:
panicErr = v
case nil:
panicErr = fmt.Errorf("runtime.Goexit() called")
default:
panicErr = fmt.Errorf("%v", v)
}
logger.Error("panic recovered in gRPC handler",
zap.String("method", info.FullMethod),
zap.Error(panicErr),
zap.Stack("stack"),
)
err = status.Error(codes.Internal, "internal server error")
}
}()
resp, err := handler(ctx, req)
panicked = false
return resp, err
}
}Explain a little bit for the usage of panicked: when panic occurs, recover() returns the value of panic (panic != nil) but when Goexit, recover() return nil -> the condition r != nil won't detect it
panicked flag helps us distinguishing "handler returns normally" or "handler is terminated by Goexit" (this technique I learn from https://github.com/grpc-ecosystem/go-grpc-middleware/blob/v1.4.0/recovery/interceptors.go)
What would you like to be added: An interceptor to help server recovery from panic
Why is this needed: pipecd server crashes and restarts when panic occurs, we should avoid this happens in production environment
There are 2 ways to do this:
Explain a little bit for the usage of
panicked: when panic occurs,recover()returns the value of panic (panic != nil) but when Goexit,recover()return nil -> the conditionr != nilwon't detect itpanicked flag helps us distinguishing "handler returns normally" or "handler is terminated by Goexit" (this technique I learn from https://github.com/grpc-ecosystem/go-grpc-middleware/blob/v1.4.0/recovery/interceptors.go)