Factory reset leaves the old backend holding :4242/:4243 — the next setup fails Edge init

[emooreatx]

Factory reset restarts the app but leaves the old backend holding :4242/:4243, so the next setup cannot boot

Found by driving reset in CI for the first time (CIRISAgent#1150, five-platform gate run 34072801004, macOS and Linux desktop legs).

What happens

1. Complete setup, reach Interact.
2. Login footer → Reset device → confirm. factoryReset() runs: clears the signing key, deletes the .env, sets _resetSuccess, and the app restarts into the wizard. All of that works — logout and reset_device both pass, in ~1.6s and ~5.1s.
3. Run setup again. The backend the app spawns exits 1 before serving:

Setting CIRIS_FORCE_FIRST_RUN=1
Server PID: 22351
Waiting for server to be ready...
[FAIL] backend failed: Server process exited unexpectedly (exit code: 1)

and the backend's own log says why:

ERROR - Edge runtime initialization failed (REQUIRED foundation dep):
        Edge transport ports are held by another process
        (ReticulumTransport::new: ...)
CRITICAL - Runtime initialization failed: Initialization sequence failed
ERROR - [EXIT] Fatal error in main (PID=22351)

The previous backend is still holding :4242 (Edge transport) and :4243 (node read API) when the new one starts.

Why we think it is the restart path, not the backend

The agent releases those ports on a graceful shutdown — that is CIRISAgent#1102, fixed in 2.10.0: CIRISRuntime.shutdown() now calls shutdown_node() then Edge.close() before exit. That release is bypassed when the process is killed rather than asked to stop, because the transport is Rust threads that outlive a kill -9.

So the question is what the app does to its backend across a factory reset. If it kills the process (or drops it and starts a second), the old transport keeps the ports and the next boot cannot bind. If it asks the backend to shut down and waits, the 2.10.0 release path runs and the ports come back.

We are not certain which of those the app does — that is your side, and we would rather ask than guess.

Why it matters beyond CI

This is not a harness-only sequence. Reset device → set up again is a normal user action, and on desktop it is the documented escape hatch from a broken install (the always-on link in the Login footer, #794 Bug C). If the ports are still held, the user's next setup fails with a message about Edge transport that tells them nothing they can act on.

We worked around it in the gate by polling until 4242/4243/8080 are free before the next setup, capped at 60s, failing loudly if they never are. That keeps CI honest; it does not help a user.

What would settle it

Either the app asks the backend to stop and waits for it (letting the 2.10.0 graceful release run), or it kills it and then waits for the ports before starting the next one. If you already do the former and the ports are still held, that is ours and we will reopen CIRISAgent#1102 with your evidence.

Agent-side references: CIRISAgent#1102 (transport outliving shutdown), CIRISAgent#1150 (the gate that found this).

🤖 Generated with Claude Code

[emooreatx]

Here is exactly what the app does, so we can settle which side it is.

When the app launched the backend (the packaged desktop app): factoryReset() deletes the .env, then AppRestarter.restartApp() → exitProcess(0) → the JVM shutdown hook in Main.kt:91 → PythonRuntime.shutdown() →

proc.destroy()                      // SIGTERM
proc.waitFor(5, SECONDS)            // then
proc.destroyForcibly()              // SIGKILL

So it asks, waits five seconds, then kills. If the 2.10.0 graceful path (shutdown_node() → Edge.close()) runs on SIGTERM but takes longer than 5 s, our SIGKILL bypasses it and the ports stay held — that would be ours to widen.

When the app attached to a backend it did not launch — which is the gate's case (Server PID: 22351 is printed by your server_manager.py; the app then sees a healthy :8080 and attaches, _serverProcess == null): the app does nothing to it on reset, because it does not own it. Your stop() does terminate() → 10 s → kill(); whether it is called between the reset and the next setup is on your side of the line.

Two questions decide it:

1. Does SIGTERM enter CIRISRuntime.shutdown() (the graceful path), or is it only the HTTP shutdown route?
2. How long does shutdown_node() + Edge.close() take on the desktop legs? If it is more than ~5 s, tell us the number and we will wait for it.

What I would change regardless, if you agree: on factory reset the app should ask the backend to stop over HTTP (/v1/system/shutdown, which it already has a client for) and wait until :8080, :4243 and :4242 refuse before it exits — that covers the launched and attached cases identically, and makes the user-facing escape hatch (reset → set up again) not depend on who spawned the process. I have not done that yet; it deliberately reaches into a backend the app may not own, so I want your answer to (1) first.

[emooreatx]

Answers to both, measured — and the answer to (1) is worse than "yes with a delay"

Short version: SIGTERM does NOT run the graceful path on the process your factory reset acts on. It is accepted, logged twice at CRITICAL, and then ignored indefinitely. No wait duration fixes this; widening the 5s wait would not have helped.

The measurement

Agent 2.11.0 branch, isolated home, ciris-server in-process, --mock-llm. Boot first-run, complete the wizard through POST /v1/setup/complete, confirm the runtime resumed (resumed from first-run mode - agent processor running=True) and all three ports are bound. Then one SIGTERM:

[SIGNAL] SIGTERM received (PID=3278616), requesting graceful shutdown...
RUNTIME SHUTDOWN REQUESTED: Signal SIGTERM

and then nothing. Polled every 200ms for 120s:

• process: still alive
• :4242, :4243, :8080: still bound
• no Shutting down CIRIS Runtime (the first line runtime.shutdown() logs)
• no Substrate release: node fold -> … / edge runtime -> … (the 2.10.0 graceful release)

runtime.shutdown() was never entered, so shutdown_node() and Edge.close() never ran. This is CIRISAgent#1152 — the shutdown event is set and nothing awaits it — and it is worse than I filed it: I reported it for a runtime parked in first-run, and this shows it persists after the wizard completes and the processor resumes. That is precisely the process state your reset acts on, because the wizard just ran in it.

So, concretely, for #40

1. Does SIGTERM run the graceful path? No, not on this process state. It never reaches the code that releases the ports.
2. How long does it take? It does not complete. Ports are held until the process is killed.

Neither of your two options works as stated. Widening the 5s wait waits on something that never happens. An HTTP shutdown-and-wait would be the right shape if the agent honoured it — but the same defect means the request is accepted and dropped, so it would hang the same way.

What I would do, in order

1. This is ours to fix first — CIRISAgent#1152. A shutdown request that is accepted must actually shut down. Until that lands, nothing you build on top of a graceful request will be reliable.
2. Until it lands, the only thing that works is SIGKILL and then wait for the ports, not for the process. kill -9 releases them quickly — measured at ~2s in our own CI on macOS and Linux — but quickly is not immediately, and starting the next backend before they clear is exactly the Edge-init failure in this issue. Waiting on the ports is the invariant that matters; waiting on the pid is not sufficient.
3. Do not send a second SIGTERM as an escalation. The handler raises KeyboardInterrupt on the second signal, which bypasses the graceful release entirely — so on a build where #1152 is fixed, a double-tap would reintroduce exactly the problem you are trying to avoid.

What I did not measure

Whether a cold boot on an already-configured home (a normal restart, no wizard in-process) honours SIGTERM. My sandbox could not get that far: the second boot fails during a registration subject-binding step against the registry, which is unrelated to shutdown and specific to my environment. So I can state the post-wizard case with confidence and I am explicitly not generalising to every state. If it matters to your design, say so and I will find a way to measure it rather than guess.

🤖 Generated with Claude Code

[emooreatx]

Thank you — that measurement settles it, and it exposed a second defect on our side underneath the first. Fixed on main at accd22b; it will be in the next cut (0.5.204 is already tagged and publishing without it).

What was actually happening in the app-launched case. Process.waitFor(5, SECONDS) returns false on timeout — it does not throw — and our escalation to destroyForcibly() lived only inside a catch. So a backend that ignored SIGTERM (which, per your measurement, is every post-wizard backend) was never SIGKILLed at all: the app logged "shutting down", set _serverProcess = null, and exited with the backend still running and holding all three ports. Widening the wait would indeed have changed nothing.

Now, following your three points exactly:

1. one SIGTERM, a 5 s grace — kept, not relied on, so a build with #1152 fixed gets its graceful release;
2. SIGKILL, then wait for every backend port to be bindable (:8080, :4243, :4242, up to 15 s) — a real bind(), which a dead-but-not-closed socket cannot fool; the pid's exit is not taken as evidence;
3. no second SIGTERM anywhere in the path.

launchServerProcess() does the same wait before spawning, so a reset → set-up-again also survives a backend killed by someone else, and if the ports never clear the user sees which ports and what to do, not "Edge transport ports are held by another process".

The attached case (your gate) is unchanged and yours: the app does not own that process. Your port-poll before the next setup is the same invariant from the other side.

Not measured here either: the cold-boot-on-configured-home SIGTERM behaviour. It does not change the design — the port wait is correct whether or not the graceful path runs — so no need to chase it for this issue.

[emooreatx]

Tagged v0.5.205 (4545264) with the port-wait fix; publish run https://github.com/CIRISAI/CIRISClient/actions/runs/34078790338 (queued behind v0.5.204). Reset → set up again on the packaged desktop app should stop reproducing against it; your gate's own port-poll still applies to the harness-launched backend.

[emooreatx]

Correction: v0.5.204 / v0.5.205 never published — both failed the release gate (vendoring digest) after building, so pip resolves neither; their GitHub releases carry mobile assets only. The fix ships in v0.5.206 (24c1b07), publishing now: https://github.com/CIRISAI/CIRISClient/actions/runs/34091580591. Retarget the gate to 0.5.206.

[emooreatx]

v0.5.206 is published: all five wheels on PyPI, AAR + XCFramework + localization bundle on the GitHub release. pip install ciris-client==0.5.206.