Freshness: know when a refresh mechanism has stopped - #15
Draft
aurph wants to merge 1 commit into
Draft
Conversation
The interconnection queue was 77 days old and nothing anywhere knew. That is not a data problem, it is a missing sense: an unattended pipeline cannot tell you it stopped running. A flow on the Jetson dies quietly when the box reboots, n8n is switched off, or a PAT expires. Nothing errors, because nothing watches. The only symptom is a date that stops moving. server/freshness-registry.ts declares every dataset: file, how to read its timestamp, how old it may get, and what is supposed to refresh it. That expectation previously lived only in ops/n8n/README.md and in my head. Adding a dataset here is the whole integration. server/freshness.ts classifies. Pure and injected like indices.ts and clusters.ts, so the clock and the IO stay in the route layer: - ok: within cadence - aging: overdue but under 2x, which is one missed run - stale: past 2x, the mechanism has probably stopped - manual: hand-curated on no schedule, reported but never alarms - unknown: no readable timestamp, an instrumentation gap not a failure GET /api/admin/freshness returns the report. GET /api/admin/freshness/check is the deadman: 200 when nothing is stale, 503 naming the offenders and the exact mechanism to restart. Both admin-gated and both added to the auth-boundary route table. Freshness is the floor, not a feature, and publishing a page about how current we are would be advertising the bare minimum. The watchdog is pointed at deliberately from off the Jetson (cron-job.org, which already fires the daily tweet and alerts on non-2xx). A watchdog hosted on the box it watches dies with it, which is the failure this exists to catch. Setup in ops/freshness-monitor.md. Two supporting honesty fixes: - The news scanner now stamps lastChecked on every run and lastRefreshed only on a real change. Staleness follows lastChecked, otherwise a dataset checked daily and correctly unchanged reads as abandoned and you cannot tell that apart from the scanner never running. - Dates are read only from fields the file actually carries. Never file mtime, which would report a fresh checkout as fresh data. A bare date parses as the start of its UTC day, so nothing is ever reported fresher than it can prove. Against real data today it returns 503 on clusters (40d), interconnection-queue (77d), and gpu-rental-prices (39d), all stale because their flows were built but never mounted. datacenters.json reports unknown: it is a bare array with no date field, so its age genuinely cannot be checked. tsc clean, 307 tests pass, build passes. Verified live end to end: 401 without a key, 503 with real data, and 200 after temporarily stamping the three files current, so the alarm is proven to go green and not just stay red.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
interconnection-queue.jsonwas 77 days old and nothing anywhere knew.That is not a data problem, it is a missing sense. An unattended pipeline cannot tell you it stopped running. A flow on the Jetson dies quietly when the box reboots, n8n is switched off, or a PAT expires. Nothing errors, because nothing watches. The only symptom is a date that stops moving, and by then it has been months.
What it does against real data right now
GET /api/admin/freshnessreturns 503 today:Every stale row is a flow that was built and shipped to main but never mounted. The monitor does not fix staleness. It makes staleness loud.
Design
server/freshness-registry.tsdeclares each dataset: file, timestamp strategy, max age, and the mechanism responsible. That expectation previously lived only inops/n8n/README.mdand in my head. Adding a dataset here is the whole integration.server/freshness.tsclassifies. Pure and injected likeindices.ts/clusters.ts, so clock and IO stay in the route layer.GET /api/admin/freshnessfull report.GET /api/admin/freshness/checkis the deadman: 200 clean, 503 naming offenders and the mechanism to restart.Both admin-gated, both added to the
PROTECTED_ROUTEStable inauth-boundary.test.ts. Freshness is the floor, not a feature; a public "how current are we" page advertises the bare minimum.The watchdog runs off the Jetson, on purpose
A watchdog hosted on the box it watches dies with it, which is exactly the failure this catches. Point cron-job.org (already firing the daily tweet, alerts on any non-2xx) at the check endpoint with an
x-admin-keyheader. Setup inops/freshness-monitor.md.Path is
/freshness/check, never/health, so the platform cannot mistake a stale-data 503 for an unhealthy instance and start recycling it.agingis deliberately a 200One missed run should not page anyone or the alert stops meaning anything within a month. Only past 2x cadence trips it.
unknowndoes not alarm either: it marks an instrumentation gap, not evidence of failure.Two supporting honesty fixes
lastCheckedon every run, andlastRefreshedonly on a real change. Staleness followslastChecked. Otherwise a dataset checked daily and correctly unchanged reads as abandoned, and you cannot tell that apart from the scanner never running at all. Consequence worth knowing: once the scan is scheduled, the monitor can be green while the homepage still shows an older date on that stat. Both are honest, they answer different questions.Verification
tscclean, 307 tests pass (287 baseline + 20 new), build passes.Verified live end to end, not just unit tested:
Not in this pass
The deals n8n flow, the
datacenters.jsondurability backup, and stamping the datacenter ingester so that dataset can report at all. Those are the obvious next steps.