Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 10 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,30 +185,28 @@ groups:
# stays connected and healthy, and neither notices. This one fires on
# silence instead, keyed on the reconcile plane's completion line.
#
# 50h, not 30h, and the size comes from the RESTART behaviour rather than
# the interval. The startup pass runs only when the last completion is
# already older than one interval; otherwise it starts a fresh interval
# from boot. So a restart shortly before the deadline defers the next pass
# by almost a full interval, and two consecutive completions can sit ~47h
# apart with nothing wrong. 50h absorbs one such restart plus runtime. It
# does NOT absorb repeated restarts inside the interval, which no
# completion-only window can; a container that restart-loops needs its own
# alert, not a wider window here.
# 26h: one interval plus runtime. The startup pass runs only when the
# last completion is already older than one interval, and the schedule
# keeps its phase across restarts (the last-run record on /config
# carries it), so two consecutive completions sit at most ~24h plus a
# pass's runtime apart, restarts included. A container that
# restart-loops faster than the time left in its period still starves
# the pass, and this window is what catches it.
#
# DROP this rule if you set DEEP_SCAN_INTERVAL to off/disabled/0, which
# runs the app WebSocket-only with no periodic pass, so there is no
# heartbeat to miss and the rule would fire forever.
- alert: PlexLanguageSyncDeepScanStalled
expr: |
absent_over_time({container="plex-language-sync"} |= `deep analysis completed` [50h])
absent_over_time({container="plex-language-sync"} |= `deep analysis completed` [26h])
for: 1h
labels:
severity: warning
annotations:
summary: "no plex-language-sync deep-analysis heartbeat in 50h"
summary: "no plex-language-sync deep-analysis heartbeat in 26h"
description: >
The reconcile plane logs `deep analysis completed` at the end of
every pass, and none has arrived in 50h (DEEP_SCAN_INTERVAL defaults
every pass, and none has arrived in 26h (DEEP_SCAN_INTERVAL defaults
to 24h). The usual cause is the periodic safety net wedged, so
replayed history items stop being reconciled while the container
stays healthy and connected. Check the container logs for the last
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ require (
github.com/cplieger/langtag/v2 v2.0.1
github.com/cplieger/plexapi/v2 v2.0.7
github.com/cplieger/runesafe/v2 v2.0.1
github.com/cplieger/scheduler/v4 v4.1.0
github.com/cplieger/scheduler/v4 v4.2.0
github.com/cplieger/slogx v1.6.4
golang.org/x/sync v0.22.0
pgregory.net/rapid v1.3.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 8 additions & 4 deletions internal/deepscan/deepscan.go
Original file line number Diff line number Diff line change
Expand Up @@ -186,13 +186,17 @@ func (s *Scheduler) Run(ctx context.Context) {
// docker-*-scheduler convention). FireOnStart is false: the conditional
// startup pass above already handled the immediate run (RunLoop's
// unconditional FireOnStart would ignore the last-run stamp and double-run
// on a recent restart). Overlapping ticks collapse via the singleflight in
// deepAnalysis, RunLoop is sequential, so no wall-clock slot-dedup is needed
// and no local wall-clock time is read.
// on a recent restart), and FirstDelay phases the first tick from the
// recorded previous pass so a restart does not delay the cadence.
// Overlapping ticks collapse via the singleflight in deepAnalysis, RunLoop
// is sequential, so no wall-clock slot-dedup is needed.
scheduler.RunLoop(ctx, func(ctx context.Context) {
slog.Info("scheduled deep analysis starting")
s.deepAnalysis(ctx)
}, scheduler.LoopOptions{Interval: s.cfg.Interval})
}, scheduler.LoopOptions{
Interval: s.cfg.Interval,
FirstDelay: s.stamp.Remaining(s.cfg.Interval, time.Now(), scheduler.CountFailed),
})
}

// deepAnalysis runs the recent-history replay + recently-added sweep,
Expand Down