WAL client reports unacknowledged flush position to Postgres, breaking RPO 0 guarantee

[gabriele-wolfox]

Summary

Klio's WAL streaming client reports a flush LSN back to PostgreSQL that does not reflect data actually durable (fsynced) on the Klio server. This makes it unsafe to configure PostgreSQL synchronous replication (synchronous_standby_names + synchronous_commit) with Klio as the synchronous target to achieve RPO 0: Postgres can acknowledge commits as durable before the corresponding WAL bytes are guaranteed to survive a Klio server or network failure.

Technical detail

• core/internal/client/sendwal/receiver.go (sendFeedback) sends Postgres a WALFlushPosition (and WALApplyPosition) derived from buffer.FlushLSN().
• core/internal/client/sendwal/buffer/buffer.go sets flushLSN = writeLSN immediately after wal.handler.Write(...) returns successfully, with no server round-trip involved.
• That Write (buffer/grpc.go) calls SendBlock (core/internal/client/klioclient/grpcclient/waluploader.go), which is just g.innerStream.Send(&PutRequest{...}), a gRPC client-streaming Send(). Per grpc-go's documented contract for ClientStream.SendMsg: "SendMsg does not wait until the message is received by the server."
• The only genuine server acknowledgment in this protocol is CloseAndRecv() (grpcclient/connection.go), which fires once per whole WAL segment (16MB by default), not on the ~200ms cadence (FlushTimeoutMilliseconds, core/pkg/config/client.go) at which flush feedback is actually sent to Postgres.
• Server-side durability itself is fine (internal/server/walserver/upload.go -> internal/repository/writer.go's DirectWriter.Flush() does call file.Sync()). The gap is purely that the client never learns about it before reporting its own optimistic position upstream.

Impact

• With synchronous_commit = remote_flush (or on) and Klio in synchronous_standby_names (its replication connection identifies as application_name=klio), Postgres routinely acknowledges commits as durable based on a flush LSN that only reflects "handed to the local send buffer," not "confirmed durable on Klio." A Klio server crash or network partition in that window loses data the application was told was safe, the opposite of RPO 0.
• The same flush position drives the physical replication slot's restart_lsn advancement on the Postgres side, so Postgres can be told it is safe to recycle WAL segments Klio was never guaranteed to have durably received. This is a WAL-loss risk independent of synchronous replication.

Suggested fix direction

Gate flushLSN advancement (and thus both the feedback sent to Postgres and slot restart_lsn advancement) on a genuine acknowledgment from the Klio server confirming durable persistence, at a granularity finer than "once per 16MB segment." This likely needs either periodic acks on the Put stream or a bidirectional-streaming protocol change. Evaluate the added per-feedback-interval latency against what the synchronous-replication use case can tolerate.

Related docs to revisit once resolved

documentation/web/docs/user/wal_streaming.md and the "Synchronous replication" bullet in documentation/web/docs/user/index.mdx currently claim zero RPO in synchronous mode.