0.5.203: announce-self succeeds, then the node re-composes before the HTTP response is sent — the caller sees a dropped connection (macOS)

[emooreatx]

ciris-server 0.5.203, exec'd node-only boot on macOS, CIRISAgent run #34228782947, live-qa-macos-ios → logs/ciris-macos/ciris-server.log.2026-09-08 and .boot.

Our gate logs in on :4243 and POSTs /v1/federation/announce (owner session). The node's log says the announce succeeded:

13:12:26 graph_config: wrote config entry key="net.announce_ownership" version=1
13:12:26 claim_remote: announce-self: owner-binding promoted to FEDERATION + net.announce_ownership=true (Reticulum identity announce takes effect on next boot)
13:12:26 auth::ownership: announce bundle row role="owner_key"     … federation_visible=true
13:12:26 auth::ownership: announce bundle row role="node_key"      … federation_visible=true
13:12:26 auth::ownership: announce bundle row role="owner_binding" … attestation_id="0200be1d-…"
13:12:27 node_control: SIGTERM broker installed
13:12:28 compose: resolved initial config:* snapshot …
13:12:28 compose: node already has a ROOT owner — no first-run claim PIN minted
13:12:28 compose: net.announce_ownership=true — owner binding promoted to federation scope; federation-identity-discoverable

and ciris-server.log.boot carries a boot marker at 13:12:27 — one second after the announce wrote its rows. The caller never received the response: httpx reported a bare ReadError on the POST. On Windows, Android and iOS in the same run the identical POST returned the bundle (ok -- 3/3 rows federation-visible); on macOS in run #34181104589 the same shape appeared as RemoteProtocolError: Server disconnected without sending a response.

So the sequence reads as: announce-self applies net.announce_ownership=true → the node re-composes ("takes effect on next boot", and the boot is immediate) → the in-flight HTTP response is lost in the re-compose. The write is durable and the re-composed node reports the promoted binding, so the state is right; only the response is dropped, and a client that takes a dropped response as failure (ours did, until today) reports a discoverable node as NOT discoverable.

Two questions rather than a prescription: is the immediate re-compose intended on this path, and if so should the response be flushed before it — or is a dropped response on announce-self the contract, in which case we will keep the client-side idempotent retry we are adding today and say so in our check. Either answer is fine; today neither is documented, and the difference between "announce failed" and "announce succeeded and the node restarted" is the difference between a P0 and a nit for anyone reading the gate.

[emooreatx]

Read against the node log, the agent log and the wizard log in the same artifact. The short version: the server does not re-compose on announce, and nothing on that path can drop a response; the restart one second later is the agent's own complete_setup hand-off, and the dropped read is the client side of that hand-off. The idempotent retry you added is the right client behaviour, and there is a cheaper ordering fix beside it.

What actually happened, on the clock

time (UTC)	who	what
13:12:26.865	node (embedded in the agent)	announce-self: binding promoted, net.announce_ownership=true written, bundle logged announce COMPLETE, handler returns 200
13:12:26.999	agent	complete_setup() endpoint called (the wizard's [ORDER] complete_setup begin (post-settle), 130 ms after its announce step)
13:12:27.114	agent	[RUN-WITHOUT-AI] recorded … CIRIS_RUN_WITHOUT_AI=true — the hand-off begins
13:12:27.869	node	fresh boot: tracing file sink online … fresh_subscriber=true, CIRISServer … starting, SIGTERM broker installed

So the "re-compose" is the run-without-AI hand-off, triggered by complete_setup, not by the announce. On the server side net.announce_ownership is boot-structural: the handler promotes the binding, writes the key, returns the bundle, and the value is READ at the next boot — there is no restart door on that path (request_shutdown() has exactly two callers, both under shutdown_node()).

Why the response can still be lost, and why on macOS

The read API stops through axum::serve(..).with_graceful_shutdown(..) when the host calls ciris_server.shutdown_node(): accepting stops, in-flight requests finish and are written, idle keep-alive connections are closed. A response the handler has returned cannot be dropped by that door.

Two ways a client still sees a bare ReadError / Server disconnected without sending a response, both on the client's side of the hand-off:

1. A pooled keep-alive connection reused across the restart. httpx keeps the connection from the login; if the request that hits the wire after the embedded node has gone away is the announce, macOS answers the write with RST and, unlike Linux, discards any buffered bytes with it — ReadError on the POST, nothing in the node log for that request. The announce the node log shows at 26.865 is then the wizard's (POST http://127.0.0.1:8080/v1/federation/announce in ciris_desktop_setup.*.log), not the gate's.
2. The hand-off exits without the graceful door. If the agent replaces or exits the process without first calling shutdown_node() (which blocks until :4243 is free), every open socket dies with the process and any in-flight response goes with it. I cannot see which the agent does from this side; the .boot marker one second later is consistent with either.

Either way the state is right, as you saw: the announce is durable and idempotent (promote_owner_binding_to_federation is a no-op once federation-scoped; the config write just adds a version), so retrying is correct by contract.

What I would do on your side, in order of cost

1. Order the gate after the hand-off settles: after complete_setup, poll GET :4243/health for a 200 (the boot contract — accept ⇒ serving, CIRISServer#548), open a new client, then log in and announce/verify. That removes both mechanisms.
2. Keep the idempotent retry you added — it is the right shape for a node that may restart under any client.
3. In the hand-off, call ciris_server.shutdown_node() before exec/exit if you do not already; it returns when the port is free, so the next boot cannot race the old listener either.

Nothing to change in the server for this; closing with that reading — reopen if the gate still sees a drop after step 1, with the gate's own timestamps, and I will look again.

[emooreatx]

Closing as answered: the restart is the agent's setup-complete hand-off, the server's shutdown door is graceful, and the drop is client-side (pooled connection across the restart, or a hand-off that skips shutdown_node). Reopen with gate timestamps if it persists after ordering the gate behind :4243/health.

[emooreatx]

Agreed, and thank you for reading all three logs rather than the one I quoted — the restart at 13:12:27 was our own complete_setup exec into the node, and my check had announced against the still-alive agent-hosted fold on :4243 as the process was replaced. Wrong premise on my side; the server did nothing on that path.

Fixed in CIRISAgent ab3254ad1: on a desktop no-AI leg the gate now waits for the transition — :4243 refused at least once (the fold dying), then answering again (the exec'd node) — before it announces, and the announce client carries no keep-alive pool so a socket opened to one peer is never reused against its successor. Where no transition appears within 4 s the backend did not exec (Android and iOS serve the fold in-process) and the fold is the right node to announce against. The one idempotent retry stays as belt-and-braces. Will confirm on the next five-platform run; nothing further needed here.

[emooreatx]

Per Eric: made predictable and decipherable from the node's own log — PR #569. A serve marker (<data_dir>/serving.json) is written when the read API binds and cleared only after it drains, so the next boot says in one WARN line when the previous serve was replaced, killed or crashed instead of stopped (with its pid, instance and start time, and whether that pid is still alive). The read API's stop now logs draining — in_flight=N and stopped — drained N, still_in_flight 0, took_ms, and every stop request logs its origin (shutdown_node() from the embedding host, SIGTERM, SIGINT). On the macOS run above, the headless boot at 13:12:27.87 would have opened with that WARN naming the embedded node's pid — the whole answer in the log you already collect.

[emooreatx]

Merged as fad320e (PR #569). From this build the node's own log answers the question this issue asked: a boot after a hand-off opens with WARN the previous serve on this home did NOT stop through the shutdown door … previous_pid=… pid_alive=…, a clean stop logs each teardown step by name with its duration and ends with node stopped … took_ms, and the read API's drain reports in_flight/drained. The same proof found that a clean stop never ended the process (edge's run loop, CIRISEdge#578); it is bounded now — SIGTERM to exit in ~10 s with status 143 until edge ships the listener shutdown.