Attachment fetch failures degrade silently — panel proceeds with empty content, personas answer blind, 0% failure rate

[claude-dataviking]

Summary

When a type: url attachment fails to fetch, SynthPanel logs a WARNING and proceeds, sending the question to the model with no attachment content. Personas then answer blind (or hallucinate), the run reports a 0% failure rate, and there is no run-level error — so a run can look successful while the entire premise (react to this page) silently didn't happen.

Repro A — SSRF/loopback perimeter block (silent)

Instrument with attachments: {pg: {type: url, url: "http://localhost:4321/", fetch_mode: markdown}}:

synth_panel.fetch.lower WARNING URLBlock lowering: perimeter denied http://localhost:4321/: no safe address for 'localhost': loopback

The run continues; the panelist receives empty attachment content. (The SSRF block on loopback is reasonable as a policy; silently continuing with empty content is the bug. It also means a local preview server can't be used as an attachment source at all — worth documenting.)

Repro B — controlled comparison on a public URL

Two identical 30-persona × 3-model runs over the same page content:

• type: url (markdown fetch of the live site): 48/90 panelists explicitly said they couldn't see the page; 0/90 referenced real on-page content; failure rate reported as 0%.
• type: html (same page text embedded inline): 0/90 blind; 81/90 referenced real content.

Same content, opposite outcomes — the URL fetch delivered nothing usable and the run never flagged it.

Expected

A failed/blocked/empty attachment fetch should be a hard error by default (or at minimum mark the affected responses as failed and count them in the failure rate), not a silent warning that lets the model answer without the content. Ideally:

1. Fail the run (or the affected question) when a referenced attachment yields no content, with a clear message naming the URL + reason (perimeter-denied / HTTP status / timeout).
2. Provide an opt-out flag (e.g. --allow-empty-attachments) for callers who genuinely want best-effort.
3. Record per-attachment fetch status in the saved result so it's auditable.
4. Document that loopback/private addresses are SSRF-blocked, so local preview servers won't work as URL sources (use inline html/document instead).

Environment

synthpanel 1.5.6, Python 3.12. Observed against both http://localhost:4321 (loopback block) and a live Cloudflare-fronted site (fetch returned no usable content).