Auth-session broker: nothing verifies Core→broker reachability; mint failures surface as "target login failed"

[guohai]

Observed

A session-injected job fails on the agent with:

[Daemon] Job execution error: target login failed: fetch failed
[Daemon] Failing job 31071: target login failed: fetch failed

What actually happened (not what the message says)

Tracing the chain:

1. ensureSession (server/auth-session.ts:152) called routeToBroker("auth-session") and got a target — so a broker row was heartbeating within BROKER_OFFLINE_THRESHOLD_SECONDS and Core had its in-memory mint secret (broker-registry.ts:60-70 requires both). Otherwise the error would read no live auth-session broker registered.
2. mintViaBroker (broker-registry.ts:85) then did fetch(target.url + "/mint") — and that call threw rather than returning a response. fetch failed is undici's transport-level error: DNS failure, connection refused, TLS failure, host unreachable.
3. markWebSessionFailed stored that text; /api/eval-agent/jobs/:id/session returned it; the daemon prefixed it (vox-agentd.ts:546).

So this is not a credential/token problem. A wrong or expired mint secret would have produced a response — broker mint failed: 401 via broker-registry.ts:90 — not fetch failed. Core simply could not reach the broker's advertised URL.

Likely causes: the broker container died within the offline threshold (still "live" in the registry), or it is heartbeating fine but its BROKER_ADVERTISE_URL is no longer reachable from Core (hostname/port/container-network change after a redeploy).

The gap

Heartbeats prove the wrong direction. The broker heartbeats to Core, which proves broker→Core works. The mint needs Core→broker. Nothing validates that direction, so a stale or wrong BROKER_ADVERTISE_URL registers as perfectly healthy, shows green on the Brokers page, and only fails at mint time — once per job, with a generic transport error.

Suggested:

• Probe the advertised URL from Core at registration (reject/flag a broker Core can't reach), and periodically thereafter — surface the result on the broker row and the Brokers page.
• Distinguish the failure classes in the stored error: unreachable broker vs. broker rejected us (401) vs. the target site actually refused the login. Right now all three read as "target login failed".

Message wording

target login failed: <reason> (vox_eval_agentd/vox-agentd.ts:541,546) reads as "the target website rejected our login" when the real cause was "Core couldn't reach its own broker". Worth splitting the prefix by failure class.

Generated with SMT smt@agora.io

[guohai]

Root cause found — BROKER_ADVERTISE_URL uses the app UUID, not the container's network alias

Confirmed against the live Coolify instance (read-only queries):

Fact	Value
Broker app status	running:healthy
Broker custom_network_aliases	vox-session-broker
Broker BROKER_ADVERTISE_URL	http://tte9qzeqxt47w9afimmsrfy8:8200 (the Coolify app UUID)
Broker container log, in full	[Broker] Auth session broker listening on :8200 — one line

The broker logs [Broker] Mint request for platform X on every mint (auth-session-broker.ts:141). Its entire log since startup is the single listen line, so not one mint request has ever reached it. Meanwhile it registered successfully (the entrypoint process.exit(1)s on registration failure, and the container is up) and heartbeats cleanly (no heartbeat error lines).

So: broker→Core works perfectly; Core→broker has never worked once. Docker DNS resolves container names and network aliases — not Coolify application UUIDs — so Core's fetch("http://tte9qzeqxt47w9afimmsrfy8:8200/mint") fails resolution, which undici surfaces as TypeError: fetch failed, stored verbatim and shown to the user as target login failed: fetch failed.

Why it survived the cutover

The 2026-08-22 cutover verified exactly two things: the broker's own healthcheck (GET localhost:8200/health, from inside its own container) and that it self-registered (broker→Core). Neither exercises Core→broker. The advertise URL was set to a plausible-looking guess and never validated — which is precisely the gap this issue was opened about. Job #31071 appears to be the first real session-injected eval in production, so this is the first time the path was exercised at all.

Proposed fix

1. Set BROKER_ADVERTISE_URL=http://vox-session-broker:8200 (the actual network alias) on the broker app and redeploy so it re-registers with the corrected URL.
2. Verify the two containers actually share a Docker network first — both currently show connect_to_docker_network: null. If they're on separate per-app networks, no hostname will resolve and they need attaching (Coolify's predefined-network setting). This is worth checking before assuming the alias alone fixes it.
3. Then the reachability probe proposed above, so this class of misconfiguration fails loudly at registration instead of silently at first mint.

Generated with SMT smt@agora.io

[guohai]

Network topology verified — shared network confirmed, only the hostname is wrong

Checked before proposing any change, since a shared network is a precondition for the fix to work at all:

	Core (hww4ogkks804o80c4wwkwksc)	Broker (tte9qzeqxt47w9afimmsrfy8)
Server	Volumetric (tsc0ss44g0ggo8oogscw04gs)	Volumetric (same)
Destination uuid	ps0sw884w4sgs8cc0gwoc4og	ps0sw884w4sgs8cc0gwoc4og (same)
Docker network	coolify	coolify (same)
custom_network_aliases	0.0.0.0 ⚠️	vox-session-broker
Exposed port	5000	8200

Both containers already share the coolify network, so no network attachment is required — the earlier caveat about connect_to_docker_network: null is a red herring (that field is the additional predefined-network toggle; the shared network comes from the destination).

That isolates the defect to the hostname alone. Docker's embedded DNS resolves container names and network aliases; Coolify names containers <uuid>-<timestamp>, and the only alias configured on the broker is vox-session-broker. The bare app UUID in BROKER_ADVERTISE_URL matches neither, so resolution fails and undici raises TypeError: fetch failed.

This is consistent with the empirical evidence: the broker binds 0.0.0.0:8200 (auth-session-broker.ts:210) and exposes 8200, so had the name resolved the connection would have succeeded — yet zero mint requests have ever been logged.

Fix (single change, no topology work)

BROKER_ADVERTISE_URL=http://vox-session-broker:8200

on the broker app, then redeploy so it re-registers with the corrected URL.

Separate config smell

Core's custom_network_aliases is literally 0.0.0.0 — a bind address typed into an aliases field. Harmless for this path (Core is the client here), but it means Core's container carries a network alias named 0.0.0.0. Worth clearing.

Generated with SMT smt@agora.io

[guohai]

Correction to my own evidence, and the precise mechanism

I wrote above that "not one mint request has ever reached it," resting on the broker's empty log. That claim was overstated and I want it corrected on the record.

createBrokerServer (auth-session-broker.ts:120-142) authenticates and validates before it logs:

/health            → 200, no log
not POST /mint     → 404, no log
missing/bad Bearer → 401, no log       ← a rejected request leaves NO trace
bad body           → 400, no log
                   → console.log('[Broker] Mint request for platform X')

So log silence rules out successfully authenticated mints — not all inbound requests. A request that arrived and was rejected would look identical in the log.

The Core-side error is what actually discriminates, and it still holds. mintViaBroker (broker-registry.ts:85-91) throws broker mint failed: ${res.status} for any non-ok response. Core recorded fetch failed — undici's error for a fetch that threw, meaning no response was ever received. Had a request arrived and been rejected, Core would have stored broker mint failed: 401, not fetch failed.

So: no TCP/HTTP exchange ever completed. The failure is at connect/resolve time, before the broker's HTTP layer. The empty log is corroborating, not load-bearing.

Why the UUID hostname can't work

Docker's embedded DNS on a user-defined network answers for exactly three things: the container name, the container ID, and explicit network aliases. Coolify 4.1.2 names application containers <uuid>-<timestamp>, and the only alias configured on this app is vox-session-broker. The bare string tte9qzeqxt47w9afimmsrfy8 is none of the three.

Two shapes remain consistent with all the evidence — (a) the name never resolves, or (b) it resolves to something that isn't the live broker (e.g. a stale container from an older deploy). Both surface identically as fetch failed, and both are fixed by the same change. I could not distinguish them directly: Coolify 4.1.2 exposes no command-execution endpoint (/servers/{uuid}/execute and friends all 404) and SSH to the host is key-restricted, so the DNS lookup itself is not observable from here. (a) is the more likely of the two, since Docker drops DNS entries for stopped containers.

Why it was never caught

The value is a filled-in placeholder. The cutover instructions read: BROKER_ADVERTISE_URL — internal-only URL Core calls back to mint, e.g. http://<broker-internal-host>:8200, and the app UUID was substituted for <broker-internal-host> — it looks like a hostname, but Coolify never publishes it as one. Verification at the time covered the broker's own healthcheck (localhost:8200/health, inside its own container) and successful self-registration (broker→Core). Neither exercises Core→broker, so nothing ever tested the substituted value.

Generated with SMT smt@agora.io

[guohai]

Applied, verified — plus a nuance that qualifies the earlier narrative

Direct proof of the mechanism (no longer inference)

Obtained shell access to the host and measured it from inside the Core container, before changing anything:

broker aliases on network `coolify`:
  [tte9qzeqxt47w9afimmsrfy8-153415109167, vox-session-broker]      ← bare UUID absent

from Core:  getent hosts tte9qzeqxt47w9afimmsrfy8   → NXDOMAIN
from Core:  getent hosts vox-session-broker         → fd8c:789b:5ed0::d

from Core:  fetch http://tte9qzeqxt47w9afimmsrfy8:8200/health → FAIL  fetch failed  EAI_AGAIN
from Core:  fetch http://vox-session-broker:8200/health       → OK    {"status":"ok"}

EAI_AGAIN is a DNS resolution failure — the exact fetch failed seen in the job error. The UUID hostname is definitively unresolvable; the alias works. (The alias resolves to IPv6 while the broker binds 0.0.0.0; undici's happy-eyeballs handles it — verified by the successful fetch, not assumed.)

Change applied

BROKER_ADVERTISE_URL=http://vox-session-broker:8200 on both the runtime and mirrored preview env rows, then redeployed (p12bivxdbflf212vc44c4kx8). Broker came back running:healthy, its env now advertises the alias, Core→advertised URL returns {"status":"ok"}, and heartbeats are advancing (20s age on a 60s interval).

The nuance — I can't fully reconstruct the history

Dumping the whole brokers table afterwards shows two rows, not one:

id	url	state	created	last_seen
1	tte9qzeqxt47w9afimmsrfy8:8200	offline	Aug 22 00:xx	Aug 22 04:xx
2	vox-session-broker:8200	idle	Aug 22 04:xx	now

Row 2 — with the correct alias URL — has existed since Aug 22 and is the one heartbeating. That undercuts a clean "the advertised URL was wrong the whole time" story: the registry held a good row alongside the bad one. routeToBrokerWith picks the freshest candidate that also has a cached mint secret, so which row a given mint used depends on Core's in-memory cache at that moment — state I destroyed by redeploying, and cannot now recover.

So, stated honestly: the unresolvable URL is proven as a real defect and is now fixed, but I cannot prove from surviving evidence that it was the specific row used by job #31071. The definitive confirmation is re-running a session-injected job and seeing [Broker] Mint request for platform … appear in the broker log — which has never happened to date.

Two follow-ups this surfaced

1. Silent heartbeat failures. heartbeat() (auth-session-broker.ts:185-190) does .catch(() => null) on the fetch and then if (!res || !res.ok) return; — a failing heartbeat produces no log line at all. The console.error only covers exceptions outside that path. A broker can silently stop heartbeating with zero evidence, which is precisely what makes this class of problem hard to diagnose.
2. Stale rows accumulate. Registration appears to create a new row per distinct advertised URL rather than superseding the old one, leaving offline rows in the registry indefinitely. Worth reaping, or at least surfacing on the Brokers page.

Generated with SMT smt@agora.io

[guohai]

Update from the job 31072 investigation (2026-08-30).

Core → broker reachability is fixed and confirmed working. With BROKER_ADVERTISE_URL=http://vox-session-broker:8200, Core reached the broker and the broker logged its first-ever mint request. The job progressed to 202 minting instead of failing instantly.

The mint then failed for a different reason, and the broker reported it uninformatively. Core saw:

[SessionBroker] Mint failed for platform agora: broker mint failed: 502
[Broker] Mint failed for platform agora: aeval exited 1: 2026-08-30 17:49:51.109 | INFO | Artifacts saved to: output/mint/20260830_174839_2119

That INFO line is aeval's last line, printed after the diagnosis. The actual cause was two lines earlier, only visible inside the container:

ERROR | Error waiting for URL pattern: https://conversational-ai.agora.io/, current URL: https://sso2.agora.io/en/login?...
ERROR | Step 1 failed: platform.setup - Timeout 60000ms exceeded.

i.e. the login was rejected and the browser stayed on the SSO page. This is the same wrong-line-wins defect fixed for the daemon in #128 — mintWithAeval used stderr.trim().split("\n").pop().

Fixed on fix/broker-mint-error-summary: reuse summarizeAevalFailure (prefers loguru ERROR lines), keep scrubCredentials after it (the summarizer has a 4-char redaction floor, scrubCredentials has none), and add the broker stage's missing COPY vox_eval_agentd/aeval-output.ts — verified with a real --target broker image build.

Still open in this issue: the misleading "target login failed" wording on the daemon side, silent heartbeat failures, and stale registry rows.