Skip to content

feat(watch): stream arriving mail as NDJSON over JMAP push - #65

Merged
radiosilence merged 2 commits into
mainfrom
feat/watch
Aug 28, 2026
Merged

feat(watch): stream arriving mail as NDJSON over JMAP push#65
radiosilence merged 2 commits into
mainfrom
feat/watch

Conversation

@radiosilence

@radiosilence radiosilence commented Aug 28, 2026

Copy link
Copy Markdown
Owner

Closes #64. Requested by David Celis (@davidcelis) by email — repo interaction limits are collaborators-only, so he couldn't file it himself.

What it does

fastmail watch blocks and emits one JSON object per line as mail arrives, so incoming email can drive a shell loop instead of a cron job that re-lists the inbox and diffs it by hand.

fastmail watch --mailbox inbox | while read -r line; do
  echo "$line" | jq -r '.data.subject'
done

--mailbox narrows to one folder, --full includes bodies and attachment metadata, --poll <seconds> swaps the push connection for periodic checks.

Same thing as a GraphQL subscription, when the HTTP surface is up:

curl -N http://127.0.0.1:8080/graphql/stream \\
  -H 'Content-Type: application/json' \\
  -d '{"query":"subscription { emails(mailbox: \\"inbox\\") { id subject from { email } } }"}'

Load-bearing decisions

One watcher, two front ends. jmap::ArrivalWatcher holds the cursor, the push connection, the backoff and the resync policy; fastmail watch and the emails subscription are both thin wrappers over it. They don't merely agree on semantics, they can't disagree.

Push is only a wake-up, never the source of truth. JMAP has advertised the transport all along — the session carries an eventSourceUrl (RFC 8620 §7.3) that we parsed into Session and discarded. But the Email state cursor lives in the CLI, and every notification, poll tick and reconnect runs Email/changes against it. All three paths converge on the same answer, so a dropped connection or a missed frame costs latency rather than mail — and --poll is the same code with a timer instead of a socket, not a second implementation.

Only creations are reported. Including updates would replay every flag change and folder move as an arrival, which isn't what a mail loop means by "new".

A dead credential ends the process; nothing else does. A watcher that exits on one bad response is useless in the loop it's meant to feed, so transient errors go to stderr and the loop continues. The exception is cannotCalculateChanges — the server has discarded history past our cursor, there's no way to know what was missed, and replaying the mailbox as new would be a lie. It resyncs to now and says so on stderr. stdout stays pure NDJSON in every case.

The push connection gets its own HTTP client. The shared one caps requests at 30s; a channel meant to stay open for days needs a read timeout instead, so silence reads as a dead connection rather than an idle one.

SSE rather than WebSockets for the subscription. It's a server-to-client firehose — nothing is ever sent back up the socket, and SSE reconnects on its own. It's also the same shape the CLI consumes from Fastmail, so there's one mental model for the whole path.

MCP gets nothing. Tools are request/response and a subscription never returns. The graphql tool's description now says so explicitly and points at the CLI and /graphql/stream, rather than leaving a caller to discover it by hanging.

Summaries by default — Email/get with bodies is the expensive half, and paying a document parse per arrival is the wrong default for a stream.

Verifying

Tested live against Fastmail, both paths:

  • eventSourceUrl is present and templates as expected (https://ams.api.fastmail.com/jmap/event/?types={types}&closeafter={closeafter}&ping={ping}).
  • fastmail watch --mailbox inbox → sent mail to self → one NDJSON line on stdout within seconds, stderr silent.
  • fastmail watch --mailbox inbox --poll 5 → same, via the timer path.
  • POST /graphql/stream with subscription { emails(mailbox: "inbox") {...} } → sent mail to self → arrived as an SSE data: event with the selected fields.

Automated coverage (211 tests green):

  • src/jmap/events.rs — SSE reassembly: frames split across chunks, split between \r and \n, repeated data: lines, several frames per chunk, comment-only frames (Fastmail opens with one), missing space after the colon.
  • src/jmap/mod.rsemail_changes paging via hasMoreChanges with each page matched on the state it was asked for, cannotCalculateChanges surviving the trip through parse_response intact (the watcher keys its resync off it), email_state fetching state without mail, get_email_summaries not requesting body values, and open_event_stream filling every placeholder in the URL template.
  • src/jmap/watch.rs — mailbox membership filtering, which errors are fatal, and backoff growth to the ceiling (on a paused clock, so it asserts the timing without waiting it out).

Note

GraphiQL isn't wired for subscriptions — its fetcher would need an SSE transport configured. /graphql/stream is for programmatic consumers; the IDE still does queries and mutations as before.

Closes #64.

`fastmail watch` blocks and emits one JSON object per line as mail lands,
so incoming email can drive a real-time shell loop rather than a cron job
that re-lists the inbox and diffs it by hand.

JMAP already carried the transport: the session advertises an
`eventSourceUrl` (RFC 8620 §7.3) that we parsed and discarded. Push is
treated purely as a wake-up — the `Email` state cursor lives in the CLI,
and every notification, poll tick and reconnect runs `Email/changes`
against it, so all three paths converge on the same answer and a lost
notification costs latency rather than mail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ri9XAwMqKiHLrDyTuA3np
@radiosilence radiosilence added the enhancement New feature or request label Aug 28, 2026
The watcher moves to `jmap::ArrivalWatcher` so the CLI command and the
GraphQL subscription are two front ends on one implementation rather than
two implementations that agree today. Cursor, backoff, reconnect and
resync semantics are therefore identical by construction.

Served at `/graphql/stream` over Server-Sent Events. SSE rather than
WebSockets: the subscription is a server-to-client firehose, nothing is
ever sent back up the socket, and SSE reconnects on its own.

MCP gets nothing here on purpose — tools are request/response and a
subscription never returns. The `graphql` tool's description now says so
and points at the CLI and the HTTP surface instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ri9XAwMqKiHLrDyTuA3np
@radiosilence
radiosilence merged commit 2b372ce into main Aug 28, 2026
6 checks passed
@radiosilence radiosilence mentioned this pull request Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream incoming emails for real-time processing

1 participant