Scheduled publishing breaks on-demand revalidation: future-dated documents 404 until ISR fallback

[rchlfryn]

Note

For now, the times the future publish will get used is not worth the development effort so we are just creating so we do not forget

Summary

When a user sets a future publishedAt date on a document (posts & pages) our on-demand revalidation system can leave that document's path stuck serving a cached 404 even after publishedAt passes and the document should be live. The path only recovers via the 1-hour time-based ISR fallback, which means scheduled publishing is effectively up to 1 hour late.

Originally raised in NWACus/web#1122
(comment).

The problem

1. A user saves a post with publishedAt set in the future.
2. The save fires our revalidation hook (e.g. revalidatePost.ts). At that moment the post is not considered published (its publishedAt is in the future), so the path is revalidated to a non-published / 404 state and that 404 gets cached.
3. The clock passes publishedAt. The document is now logically published — but no event fires. Nothing tells Next.js to revalidate the path.
4. The cached 404 keeps being served. SWR does not help here because the 404 response itself is cached, so subsequent visitors continue to get the stale 404.
5. The path only self-corrects when the 1-hour time-based revalidation (ISR) safety net kicks in. Net effect: scheduled publishing is always up to ~1 hour behind the intended publish time.

This affects any routable collection that combines a future publishedAt with on-demand revalidation — currently posts, pages, homePages (see docs/revalidation.md → Routable Collections).

Proposed solution

Use Payload's scheduled publish feature, which is built on Payload's Jobs Queue. At the scheduled publish time, the job modifies the document, which fires our existing afterChange revalidation hooks. This correctly revalidates the path right when the content goes live, instead of waiting for the ISR fallback.

This requires standing up the Jobs Queue in this project (not currently configured).

Notes/ TODOs

• Payload Jobs Queue is configured in the project
• Scheduled publish via publishedAt triggers on-demand revalidation at publish time (not via the 1-hour ISR fallback)
• A document with a future publishedAt becomes reachable at its path within a small tolerance of publishedAt, with no stale cached 404
• Behavior verified for all routable collections (posts, pages, homePages)
• The existing 1-hour ISR fallback should remain as a safety net.

[busbyk]

This was generated by AI during triage.

Agent Brief

Category: bug
Summary: Future-dated documents serve a cached 404 until the 1-hour ISR fallback because "becomes live" is time-driven (a publishedAt threshold) while on-demand revalidation is write-driven (fires on save). Stand up Payload's Jobs Queue + scheduled-publish so a re-render is triggered at the actual publish moment.

Root cause (confirmed by tracing):

• Public routes are force-static with export const revalidate = 3600. Reads apply publishedFilter(), which requires _status === 'published' and (publishedAt <= now or absent). A doc with a future publishedAt fails the filter → the route renders notFound() → Next.js caches that 404 as the path's static output.
• The afterChange revalidation hooks (revalidatePost / revalidatePage / revalidateHomePage, dispatched via ROUTABLE_COLLECTIONS in the revalidate-document utility) gate only on doc._status === 'published' — they have no publishedAt awareness. Saving a "published, future-dated" doc revalidates the path now, re-caching another 404. Nothing re-runs when publishedAt actually passes, so the 404 persists until the 3600s ISR timer regenerates the page.

So there are really two things to fix: (1) provide a publish-time trigger (the Jobs Queue), and (2) close the manual "published + future publishedAt" footgun that produces the stuck 404 in the first place.

Desired behavior:
A document scheduled to publish at a future time becomes reachable at its path within a small tolerance of that time, with no stale cached 404 — driven by an event at publish time, not the 1-hour fallback (which stays as a safety net).

Key interfaces / approach:

• Enable scheduled publish. Add versions.drafts.schedulePublish: true to the routable draft-enabled collections (posts, pages; homePages if it should support scheduling). Payload auto-registers a schedulePublish job task and the payload-jobs collection from this alone. The task runs payload.update({ _status }) at the scheduled time, which fires the existing afterChange revalidation hooks at the real publish moment — exactly the missing trigger.
• Run the queue (the load-bearing infra piece). Enqueued jobs don't execute themselves — something must run the queue. Two hosting shapes:
  • Preferred (maintainer direction): a dedicated long-running consumer. Deploy the Payload jobs-queue consumer as a separate long-running server/worker process that runs the job runner continuously (Payload's in-process autoRun/polling, which works in a persistent process even though it does not work on Vercel serverless). We expect to use the jobs queue for other workloads beyond scheduled publish — notably the native AFP product pages (Native forecast pages #1135/Add Posthog analytics to native forecast pages #1136) and other future background jobs — so a persistent worker is the better long-term fit: it isn't bound by serverless function time limits, can process multiple queues, and gives one place to scale background work. Design the queue setup so the consumer can run as (or move to) a dedicated server without rework. Where that worker is hosted is a deployment decision to confirm with the maintainer.
  • Acceptable interim/fallback: a Vercel Cron. An external trigger hitting Payload's POST /api/payload-jobs/run endpoint on a schedule, via a crons entry in vercel.json (fronted by an authenticated thin route or a CRON_SECRET check — the runner must not be publicly invocable). Fine to ship this first if the long-running worker isn't stood up in this pass. vercel.json currently has no crons and sets git.deploymentEnabled: false, and there are no existing cron endpoints — so either path is greenfield.
• Close the manual footgun. Decide how to reconcile the existing direct path (an editor setting _status: 'published' with a future publishedAt, which populatePublishedAt preserves). Recommended: prevent that combination (validation error steering the editor to the schedule-publish control) so a "published" doc is never future-dated into a stuck-404 state. Confirm the exact UX with the maintainer.
• Keep publishedFilter() and the 1-hour ISR fallback unchanged as the safety net.

Acceptance criteria:

• Payload Jobs Queue is configured and the schedulePublish task/payload-jobs collection exist.
• Scheduling a post/page to publish at a future time makes it reachable at its path within a small tolerance of that time (bounded by cron granularity — see below), via on-demand revalidation, not the 1-hour ISR fallback — verified with no stale cached 404.
• The manual "published + future publishedAt" combination no longer produces a stuck 404 (prevented, or itself scheduled — per the chosen approach).
• Behavior verified for all routable collections in scope (posts, pages, and homePages if included).
• The 1-hour ISR fallback remains as a safety net.
• The job-runner endpoint is authenticated (not publicly runnable).
• Tests cover the crossing: a future-dated doc is 404 before its time and live after the job runs (mirror the existing publishedFilter / populatePublishedAt / revalidateDocument server tests; consider a scheduled-publish e2e alongside the posts/pages admin specs).

Requires human / deploy verification (flag in the PR — cannot be fully verified by the agent):

• Consumer hosting is a deployment decision: whether to stand up the dedicated long-running worker now vs ship the Vercel Cron interim first, and where that worker runs. Confirm with the maintainer before finalizing the runner setup.
• Vercel Cron granularity is plan-dependent (lower plans are as coarse as once/day; sub-hourly needs the appropriate plan). Cron frequency directly bounds publish precision, so the maintainer must confirm the plan supports the needed frequency (e.g. every 1–5 min) and that the cron actually fires in the deployed environment.
• git.deploymentEnabled: false in vercel.json means deploy/cron behavior here is non-standard — confirm how this project deploys and that crons register.

Out of scope:

• Changing the read-time publishedFilter semantics or the static/ISR strategy generally.
• Draft preview behavior (already correct — draft mode bypasses the filter).

Context: setting up the Jobs Queue here is shared infra the native AFP forecast-pages effort (#1135/#1136) also wants — per the originating discussion in PR #1122, this was expected to be built alongside that work. Worth coordinating so the queue is stood up once.