Problem
When PostgreSQL asks the plugin for a WAL file (RESTORE_WAL, called by
restore_command), the plugin already times the whole thing — but only writes it
to the log. There's no metric, so you can't graph it or alert on it.
This hurts most in a replica cluster. The designated primary there keeps up to
date by restoring WAL files from an external Klio source
(core/internal/cnpgi/wal.go, the cluster.IsReplica() branch; there's an
example in documentation/web/docs/user/plugin_configuration.md). So WAL restore
is how that cluster replicates, and how fast it runs sets the replica lag. When
such a replica falls behind today, there's no data showing whether the hold-up is
prefetch, the tier, or the network.
The same applies during recovery: restore speed decides how fast a cluster
catches up.
klio.server.wal.get_duration doesn't cover this. It only times the server's
side of a single WAL.Get call, while the plugin's restore also picks the config
and tier, fails over between tiers, checks the prefetch spool, and renames the
file into place. More important: a prefetch hit never reaches the server — it's
served from the local spool with a rename, so no WAL.Get is sent and
get_duration sees nothing. Those hits are the normal case while replaying WAL
in order.
Requirements
- Export the restore time the plugin already measures, as a histogram.
- Tag it with the outcome, which tier served it, and whether it was a prefetch
hit or a real download.
- Keep prefetch hits and misses separable. A hit is a local rename, a miss waits
on the network, so the two are orders of magnitude apart: mixed together, a
percentile just drifts with the hit rate. Kept apart, the hit rate also shows
whether prefetch is keeping up and whether its settings need tuning.
Verification
Problem
When PostgreSQL asks the plugin for a WAL file (
RESTORE_WAL, called byrestore_command), the plugin already times the whole thing — but only writes itto the log. There's no metric, so you can't graph it or alert on it.
This hurts most in a replica cluster. The designated primary there keeps up to
date by restoring WAL files from an external Klio source
(
core/internal/cnpgi/wal.go, thecluster.IsReplica()branch; there's anexample in
documentation/web/docs/user/plugin_configuration.md). So WAL restoreis how that cluster replicates, and how fast it runs sets the replica lag. When
such a replica falls behind today, there's no data showing whether the hold-up is
prefetch, the tier, or the network.
The same applies during recovery: restore speed decides how fast a cluster
catches up.
klio.server.wal.get_durationdoesn't cover this. It only times the server'sside of a single
WAL.Getcall, while the plugin's restore also picks the configand tier, fails over between tiers, checks the prefetch spool, and renames the
file into place. More important: a prefetch hit never reaches the server — it's
served from the local spool with a rename, so no
WAL.Getis sent andget_durationsees nothing. Those hits are the normal case while replaying WALin order.
Requirements
hit or a real download.
on the network, so the two are orders of magnitude apart: mixed together, a
percentile just drifts with the hit rate. Kept apart, the hit rate also shows
whether prefetch is keeping up and whether its settings need tuning.
Verification
RESTORE_WALis recorded, whether it succeeds or fails.visible.