Feed refetch polls the published head, so late-arriving articles are never shown

[makeev]

Problem

The feed refetch polls the published feed head, so it never sees articles that reach the feed after their publish time — and those are the majority.

ensure_feed (src/app/feeds.rs:248-262) refetches with head_cmd, i.e. cursor: None, whenever the bundle is older than alphai_ttl and the cursor sits at the top. head_cmd calls /api/news/ with no sort, which is the reverse-chronological feed. apply_alphai then diffs the returned uids against feed_seen to render unseen markers (src/app/feeds.rs:180-196).

That works only if new articles always arrive at the top of the published ordering. They do not. An article ingested now can carry a publish timestamp from an hour ago, which places it below the first page and outside every subsequent refetch. Backend measurement when delta polling was designed: 75.8% of rows arrive behind the published watermark, with general news landing a median of ~33 minutes after publication and SEC filings 5 to 9 minutes after. With the default page_size of 10 the window is narrow, so the miss is not an edge case.

Two visible consequences: the feed silently omits articles, and the unseen-marker count under-reports how much is actually new.

Scope note: I traced this from the code and the API's ordering semantics. I have not instrumented a running TUI to measure its actual miss rate, so treat the percentage as the API-side figure it is, not a measured TUI number.

What the API now offers

GET /api/news/?sort=ingested returns rows in the order they became available (ascending). Its cursor is a polling position rather than an end-of-feed marker: next_cursor is always set, and an empty results means caught up. Each item carries original.created_at, the moment AlphaAI received the article, next to time_published.

Both official SDKs picked this up in 0.4.0. The TUI has its own client (src/alphai.rs), so there is nothing to inherit — this is a local change.

Suggested shape

Do not simply switch the head refetch to sort=ingested. That mode is ascending, which is the wrong reading order for the news pane, and paging back through history still needs published.

Better split:

• keep published for what is displayed and for request_more_articles paging (unchanged);
• add a separate lightweight ingested poll whose only job is to detect arrivals, merging anything new into the top of the bundle and into the unseen set.

src/alphai.rs:news() needs a sort argument for that; everything else is orchestration in feeds.rs.

Two traps worth knowing before implementing

Cursors are mode-specific and no longer fail quietly. The two sort modes mint separate cursor families. As of 2026-07-25 replaying a cursor into the other mode returns 400, and so does a corrupted cursor, instead of silently restarting at the feed head. So the ingested poll cursor and the published paging cursor must be stored separately and never crossed. The strictness is deliberate: in delta mode a silent restart skips a range of rows that the client cannot detect afterwards, because a legitimate delta page never contains previously-seen rows either, and a head jump returns rows newer than the last seen. Only the server can catch it.

archive_horizon also applies to where a poll resumes. On Free and Basic keys a cursor left unused for longer than the plan's window (30 / 90 days) comes back 403 with extra.reason = "archive_horizon". Realistic here: the TUI is closed for a month and reopens with a stale stored cursor. is_archive_gate already recognizes the shape for paging, but the poll path should treat it as "reprime from scratch", not as an error to surface. Pro has no window.

[makeev]

Partial mitigation in 0.10.2

Feeds now request page_size=20 explicitly instead of letting the server apply its default of 10. Two facts behind that, both verified against the live API on a Free key:

• page_size accepts 1 to 20 on every tier. Only 21 and above need Pro: 400 page_size above 20 requires a Pro plan; Free and Basic are limited to 20.
• The client was sending no page_size at all on head fetches. fetch_feed_page probes 50 only when a cursor is present, that is on the first explicit "load more", so every tier including Pro opened a feed at 10 rows.

The page width is exactly the window a refetch can see, so doubling it removes the narrowest part of the problem at no extra request cost. It does not close this issue: rows landing below row 20 stay invisible to any later refetch.

What the API does and does not offer

Checked against alphai.io/api/schema/ (1.15.0) plus live calls:

• sort=published|ingested exists on /api/news/ only. /api/news/insider/?sort=ingested returns 400 Validation error with extra_forbidden. The insider feed therefore has no delta mode, and the Form 4 rows that arrive 5 to 9 minutes behind publication cannot be caught this way. A wider page is all that helps there, which is worth reflecting in the issue's scope.
• Mode-specific cursors are confirmed in the schema text: a cursor is only valid with the sort mode that issued it, a mismatch returns 400.
• Delta responses are cached 60s server side, so polling more often than once a minute buys nothing. That sits comfortably under the 300s default TTL.

Notes for the delta poll itself

• It should replace the TTL head refetch for news feeds, not run alongside it. Then the request budget does not grow at all, and the current "refetch only when news_selected == 0" restriction can go away, because a merge does not drop loaded pages the way a bundle replacement does. That also fixes a second symptom: a reader who has scrolled down currently gets no updates at all.
• Selection stability is the sharp edge. Inserting rows above the cursor shifts news_selected, and mark_selected_seen runs every frame by index, so without a compensating shift the reader silently marks a different row as read. Worth a test of its own.
• Ordering is an open decision, not an implementation detail. Late arrivals placed at the top break strict reverse-chronological order; placed at their published position they can land below the fold, which is the invisibility this issue is about. Prepending plus the unseen marker looks like the better trade, but it should be settled before the code.
• Event::Feed.append: bool becomes a three way mode (replace, append, merge), which touches all five construction sites.
• The ingested cursor needs its own storage next to feed_seen, dropped on r and on a score filter change, with a reprime path for both 400 and 403 archive_horizon that cannot loop.