Add opt-in stale-on-error recovery from retained Redis values

[lan17]

Priority

P2 — Medium — opt-in availability behavior for selected use cases. Default behavior remains unchanged.

Summary

Allow DialCache to return a physically retained Redis value after the source of truth (SoT) rejects.

For an opted-in use case:

• F = ttlSec[CacheLayer.REMOTE] is the logical fresh age.
• M = staleOnErrorMaxAgeSec is the absolute recovery age.

age < F        fresh
F <= age < M   retained; recoverable only after an SoT rejection
age >= M       unavailable

This is stale-on-error, not stale-while-revalidate: stale data is never returned while the SoT is healthy, and there is no background refresh or circuit breaker.

Implementation: #121

Confirmed decisions

• Opt-in, per use case, runtime-overridable, and default-off.
• One existing Redis value key and frame-v1 envelope; no parallel stale key.
• staleOnErrorMaxAgeSec?: number is the only new policy scalar.
• Omission is off by default and inherits in a sparse overlay; 0 explicitly disables an inherited policy.
• Enabled policy requires 0 < F < M <= 31,536,000 seconds.
• Writes retain the value physically to M; every ordinary read still enforces F.
• Redis server time, not application wall time, determines serving age.
• After a definitive ordinary miss and any SoT rejection, one independent bounded reread may serve below M.
• Initial Redis errors/timeouts and initial decode/decompression failures do not trigger recovery.
• A recovery failure never replaces the original SoT rejection.
• Tracked recovery rechecks the current invalidation watermark.
• A recovered value is never written back, refreshed, put in process-local cache, or admitted to shadow work.
• Request-local memoization may retain a recovered value only inside the already-active request scope.
• One bounded recovery metric; no new public error class or duration histogram.
• Same-key rollout is readers-first; backward compatibility with readers that ignore logical age is not required.

Public configuration

new DialCacheKeyConfig({
  ttlSec: { [CacheLayer.REMOTE]: 300 }, // F: fresh for 5 minutes
  staleOnErrorMaxAgeSec: 3_600,         // M: recoverable through 1 hour
});

Static invalid combinations fail fast. An invalid runtime M records configuration telemetry, disables only recovery for that invocation, and preserves an otherwise valid fresh Redis policy. The invocation's once-resolved F/M snapshot governs its entire read/SoT/recovery chain.

Lowering F or M applies immediately to retained keys. Raising M cannot resurrect or extend a key written with a shorter physical TTL; only a later successful write receives the longer retention. M remains an upper bound rather than a guarantee because invalidation, eviction, and physical expiry may remove data sooner.

Execution flow

flowchart TD
  A[Redis read with maxAge F] -->|hit| B[Return fresh]
  A -->|definitive miss| C[Call SoT]
  A -->|error or timeout| D[Call SoT; recovery forbidden]
  C -->|success| E[Publish normally with Redis PX M]
  C -->|rejection| F[Redis reread with maxAge M]
  F -->|eligible| G[Return retained value without publication]
  F -->|miss, error, timeout, or decode failure| H[Throw identical SoT rejection]
  D -->|rejection| H

Loading

Rereading after the rejection avoids transferring and deserializing stale payloads on healthy SoT refreshes, rechecks the hard M boundary at serving time, and observes invalidation or a concurrent refresh that happened during the SoT attempt.

Redis protocol

The v0.20 design keeps payload I/O native and uses Redis server time without read Lua:

• Untracked read: ordered same-primary GET, then TIME.
• Tracked read: atomic same-primary MGET(value, watermark), then TIME.
• Adapter acceptance: 0 <= serverNowMs - createdAtMs < maxAgeMs.
• Every ordinary/shadow read passes F; only the recovery reread passes M.

Both writes use a native SET PX of an unreadable version-0 payload placeholder followed by a small payload-free stamp script. The script verifies a per-write nonce and stamps the frame with Redis time; the tracked variant additionally fences and maintains the invalidation watermark. Payload bytes never cross the Lua boundary.

RedisReadRequest.maxAgeMs and DialCacheRedisClient.enforcesMaxAge: true are required. DialCache rejects old/custom semantic clients that do not attest to this contract.

Failure, coalescing, and observability

• Recovery receives a new effective remoteReadTimeoutMs budget.
• All SoT rejections qualify, including arbitrary rejection values and FallbackTimeoutError.
• Existing coalescing shares one initial read, one SoT attempt, and at most one recovery read; coalesce: false preserves independent chains.
• Late SoT/Redis settlement is consumed and cannot publish.
• One attempted recovery emits exactly one bounded outcome: served, miss, read_error, read_timeout, or deserialization_error.
• Existing fallback error and duration telemetry remains truthful even when retained data reaches the caller.

Rollout and rollback

An older reader treats physical Redis presence as freshness. Once a new writer retains the same frame-v1 key through M, that reader could serve ages F..M as ordinary fresh hits.

Required rollout:

1. Deploy readers/adapters that enforce F while M remains omitted or 0.
2. Upgrade the entire reader fleet.
3. Enable positive M only for selected use cases.
4. Monitor Redis CPU, memory, evictions, SoT failures, and recovery outcomes.

Disabling recovery on new readers is immediate and safe because they still enforce F. Reintroducing an old reader is unsafe until the largest previously enabled M has elapsed since the final PX M write, or the affected keys are isolated/removed. If mixed-version or immediate rollback safety becomes required, use a new key/frame version instead.

Cost model

• Read: one pipeline/batch RTT with two top-level commands (GET/MGET + TIME).
• Write: one pipeline/batch RTT with native SET + a small header-only stamp script.
• Qualifying SoT rejection: one additional read pair.
• Payloads never cross Lua; longer M can increase resident memory, watermark lifetime, expiration work, and eviction pressure.

The implementation includes a no-threshold benchmark that asserts physical retention near M, logical expiry at F, source/recovery counts, compression, and coalescing while reporting Redis command CPU, network deltas, and throughput.

Alternatives considered

Parallel fresh and stale keys

Rejected. It duplicates writes and payload storage, creates divergence and partial-write cases, complicates invalidation, and expands Cluster handling.

Hold the initial stale candidate through the SoT call

Rejected. It transfers/deserializes stale data when the SoT succeeds and risks serving across M or after an intervening invalidation.

Tri-state initial read

Deferred. It could avoid a second lookup for cold misses but expands the semantic client/result contract. The simpler miss plus bounded reread is measurable and preserves existing core abstractions.

Process-local stale retention

Deferred. It requires dual-expiry local entries and stronger invalidation analysis. Redis is the cross-process stale reservoir in v1.

Acceptance criteria

• Default-off, omission/inheritance, explicit 0, and 0 < F < M <= 365 days validation.
• Strict F and M boundaries with Redis server time for tracked and untracked values.
• Physical F retention when off and M retention when on, including shadow clean-miss fills.
• Fresh hit, successful SoT refresh, arbitrary SoT rejection, fallback timeout, and identical-error propagation.
• Initial Redis error/timeout/decode failures never trigger recovery.
• Recovery never publishes to Redis, process-local cache, or shadow work.
• Invalidation during the SoT attempt prevents tracked recovery.
• Coalescing on/off, request-local-only memoization, cached undefined, compression, and runtime policy changes.
• Bounded metrics in the core, Prometheus, and Datadog adapters.
• Node-redis, Valkey GLIDE, fake/custom clients, packed consumers, Redis 6.2, Valkey 8, and Cluster routing coverage.
• Readers-first rollout and downgrade barrier documented.