feat: serve markdown to agents, and apply the _headers file sites ship - #23
Conversation
Static site generators emit the markdown source of a page as a sibling of its directory index — `about.md` next to `about/index.html`. The file is already on disk; the server just never looked for it. Now a request whose Accept header names text/markdown is served that sibling. Without this an agent can only discover the markdown by reading the <link rel="alternate"> inside the HTML document it was trying not to download. Measured on a real zop.dev build, the same page is 15-96x smaller as markdown (a changelog entry: 145,354 bytes of HTML vs 1,512). Only explicit media types count. A browser sends `*/*;q=0.8`, which matches text/markdown by the letter of RFC 9110, so matching wildcards would serve raw source to every human visitor; q-values are honoured, so a client that ranks markdown below HTML still gets HTML. The lookup falls through untouched when the client did not ask or the .md is absent, so nothing an existing deployment serves today changes. Vary: Accept is set on every response — without it a CDN can hand an agent's markdown to the next browser that asks for the same URL. .md is registered with the mime package explicitly: Go's built-in table has no entry for it and a scratch base image has no /etc/mime.types, so http.ServeFile would otherwise sniff the file and label it text/plain. Verified end to end against a 5,933-page build: agents get markdown, browsers get HTML, pages without a .md still return 200, and direct .md requests are unaffected. Handler and negotiation are at 100% statement coverage.
Lint fixes for the CI gate:
- drop the init() (gochecknoinits) — the .md content type is now set
explicitly in the handler, which also covers directly-requested .md
rather than relying on process-global MIME registration;
- split Accept parsing into parseAcceptEntry so markdownPreferred drops
back under the cyclomatic limit;
- "honoured" -> "honored" (misspell, US locale).
Also answers a 404 in markdown when the client asked for markdown. An
HTML error shell is unusable to such a client and is not small: the 404
page of a real site measured 144,188 bytes, sent in reply to a request
it could not parse. The markdown reply is 138 bytes and names
/sitemap.xml and /llms.txt so the reader can recover on its own.
The request path is deliberately not echoed into that body — gosec
flagged it as an injection sink (G705), and the caller already knows the
URL it asked for.
Verified no behaviour change for anyone else: base and patched binaries
served the same 5,933-page build and were compared over 1,505
request/response pairs (301 URLs x 5 non-markdown Accept variants,
including a browser string and `text/html, text/markdown;q=0.1`).
Status, Content-Type and body SHA-256 matched on every one. The only
deltas are the added Vary: Accept and markdown for clients that asked.
golangci-lint output is identical to origin/main — the diff introduces
no new findings.
aryanmehrotra
changed the title
`_headers` is the Netlify / Cloudflare Pages convention: a file at the root of the published directory listing path patterns and the response headers to send for them. Static site generators emit it expecting the host to honour it, and a host that ignores it fails in the worst way — the file looks authoritative in the repo while nothing it declares has ever reached a browser. https://zop.dev, served by this project, has shipped a 2,437-byte _headers for months. Measured against the live site, not one of its rules was in effect: X-Frame-Options: DENY absent X-Content-Type-Options: nosniff absent Referrer-Policy absent Permissions-Policy absent Cache-Control on /_astro/* (immutable) absent entirely That last one matters twice over: content-hashed bundles that could be cached for a year were being served with no caching directive at all. Rules are parsed once at startup. A published directory without a _headers file yields no rules, so an existing deployment is byte-for-byte unchanged. All matching rules contribute in file order, so a later specific block overrides an earlier catch-all, matching upstream precedence. Malformed lines are skipped rather than failing the file: one bad rule should not cost a site every other header it declares. Headers are applied before anything writes, so they cover hits, misses and the SPA fallback alike — a 404 that leaks framing protection is as exploitable as a 200 that does. Note for operators: if a reverse proxy sits in front of this server, it may set some of these itself. Strict-Transport-Security is the usual one (ingress-nginx sends max-age=15724800, no preload, with replace semantics), and where it does it will keep winning; the other five headers rarely have a proxy counterpart and take effect immediately. Verified against a real 5,933-page build: 22 rules load, and the expected headers appear on pages, hashed assets, the root and robots.txt. Compared 891 request/response pairs against origin/main — zero body differences, zero unintended differences. Handler and parser at 100%/96% statement coverage; golangci-lint output identical to origin/main.
Found by running the actual distroless image rather than a native binary. `GetOrDefault` only falls back when a key is ABSENT. The shipped configs/.env sets STATIC_DIR_PATH= and DEFAULT_EXTENSION= with empty values, so a deployment that supplies STATIC_DIR_PATH through the environment got "" instead — every path lookup silently rooted at the process working directory. In that shape the server still served pages but loaded zero _headers rules, which is exactly the kind of half-working state that never gets noticed. The default shape (./static, where a Dockerfile typically copies the published directory) was unaffected and loaded all 22 rules. This makes the other shapes behave the same. The startup line now names the resolved directory alongside the count. "0 rules" is normal for a site with no _headers file and indistinguishable from a misrooted path unless the path is on the line too. Verified in the real gcr.io/distroless/static-debian12 image against a 5,978-page build, 16/16: correct Content-Type for html/css/svg/txt/md with no /etc/mime.types present, all _headers rules applied including on 404s, markdown negotiation, Vary, and a 138-byte markdown 404.
37040a2 to
7444624
Compare
…long Three header changes reached responses that cannot benefit from them. Each was found by diffing the real binaries against origin/main rather than by reading the diff — the earlier comparison covered status, Content-Type and body only, so nothing had ever checked Vary or Cache-Control. Vary: Accept was sent on every response, including the hashed bundles under /_astro/ that the same _headers file marks immutable. Only an extensionless route can resolve to a .md sibling, so everything else was keying caches on a header that cannot change what they return — the cost landing precisely on the responses this server most wants cached. It is now gated on the same condition resolveFilePath uses to negotiate, and negotiable() is the one definition of that condition so the two cannot drift apart. A miss still varies whatever the path looks like, because a miss is answered in markdown whenever the client asked for it — including for extensions that never negotiate on a hit. Without that a cache can hand an agent the HTML shell it stored for a browser. A site's Cache-Control reached its 404s. `/*.html` with max-age=300 meant a file merely not propagated yet during a deploy was pinned into every cache downstream for the rule's lifetime. Cache directives are now dropped on a miss. The security headers still apply — a 404 that leaks framing protection is as exploitable as a 200 that does — and the SPA fallback keeps its caching, since that is a real route being served, not a miss. Directly requested .md files were relabelled text/plain -> text/markdown. Bodies were identical, which is how it read as a no-op, but browsers render text/plain inline and download text/markdown: every existing .md link on a site would have turned into a download prompt. The explicit type is now set only for a negotiated response, which is the case that needs it — Go's MIME table has no .md entry and distroless has no /etc/mime.types. Verified against base and patched binaries on the same build across 13 path x Accept combinations, comparing status, Content-Type, Vary, Cache-Control, X-Frame-Options, Content-Length and body SHA-256. The only changed or removed field in the whole matrix is the one intended negotiation; every other delta is a header the site's own _headers file declares. Six mutations go red: Vary unconditional on hits, no Vary on a miss, the Cache-Control strip removed, the markdown Content-Type unscoped, and negotiable() ignoring either the extension or the root. golangci-lint output is identical to origin/main (4 findings, all pre-existing).
CI caught what a macOS run could not. The previous commit asserted that a directly requested .md keeps a non-markdown Content-Type — true on macOS and in the distroless image that ships, false on the Ubuntu runner, because most Linux distributions map .md in /etc/mime.types and http.ServeFile answers text/markdown there before this server does anything. The assertion pinned the host's MIME table rather than any behaviour of ours, so it passed locally and failed in CI. The scoping itself was right and is unchanged. What changes is how it is checked: the condition moves into labelAsMarkdown, covered by a table that runs identically everywhere, and the end-to-end test now compares a direct .md against mime.TypeByExtension — the same lookup ServeFile makes — instead of against a hardcoded type. Where the platform has no entry that is text/plain, which is the production case and the one the scoping exists for; where it has one, the test agrees with it. A negotiated response is asserted to carry the explicit type on every platform, since that is the case that must not depend on the base image having a MIME table at all. Verified by running the suite under golang:1.26 with media-types installed, so /etc/mime.types really did contain `text/markdown md markdown` — the exact shape that failed. All six mutations still go red; golangci-lint remains identical to origin/main.
.well-known is handed to the next handler untouched, so that an ACME challenge is not given an extension or swallowed by the SPA fallback. That early return also skipped the _headers rules, so a site declaring X-Frame-Options for `/*` got it everywhere except there. A `/*` block means the whole site, and a path this server delegates is still a path it answered for. The rules are now applied before the branch, which is where they belonged: they depend only on the request path, not on anything the resolution step produces. Delegation itself is unchanged — the path reaches the next handler exactly as before, with no rewriting. That raises the question the miss path already answered: the site's Cache-Control must not attach to a response that is not a page the site publishes. Here the status is chosen by the delegate, so it cannot be decided up front, and the directives are withdrawn on the way out instead — a delegated 200 is a real file and keeps its caching, a delegated 404 does not. Both paths now call withdrawCacheDirectives, so the rule has one definition and one rationale rather than two that can drift. The writer is wrapped only for the delegated paths. Wrapping the main serving path would hide net/http's io.ReaderFrom from http.ServeFile and cost every static file its sendfile fast path; ACME challenges are small and rare enough not to be worth a special case to keep fast. Verified against the real binary with GoFr's own chain as the delegate: on main an ACME challenge file comes back with no security headers at all, and with this change it carries X-Frame-Options while still returning the token body intact, while an absent .well-known path 404s with the security headers and without Cache-Control. Four mutations go red: skipping the rules for .well-known, dropping the wrapper, scrubbing on every status rather than errors only, and making withdrawCacheDirectives a no-op. golangci-lint reports fewer findings than origin/main rather than more (1 vs 4). The three that went are gosec G703 taint-analysis hits on http.ServeFile, whose call sites this commit does not touch — gosec's taint walk is sensitive to unrelated edits in the same function.
ReviewSolid, careful work. Core logic — Accept negotiation, 1. [Medium — docs] README not updated for two user-facing featuresThe diff touches only
The README is the repo's sole doc surface and already carries a behavior contract. Shipping 2. [Medium — testing]
|
…d for real
Four items from review, all confirmed against the code before acting.
The empty-config test was vacuous, and the mutation narrative made that worse
rather than better. It re-implemented the fallback in its own body:
got := tt.value
if got == "" { got = tt.fallback }
assert.Equal(t, tt.want, got)
which passes by construction and never touches main.go. Deleting both guards
from main() left the suite green — the one bug that commit fixes had no test at
all. The resolution now lives in resolveOrDefault, taking the narrow config
interface it needs, and is table-tested through the real function against a fake
whose keys are present-but-empty, which is the distinction that matters and the
one configs/.env actually ships. Removing the guard now fails.
The SPA fallback could serve a negotiable route without Vary: Accept. With
foo.md on disk and no HTML page beside it, markdown clients take the hit path
and browsers land on the shell, so one URL yields two bodies while only one of
them said it varies — a shared cache could hand the shell to an agent.
Reproduced against a running server before fixing.
Vary is now advertised through one helper, which keeps Add over Set: a site's
_headers may declare its own Vary and Set would discard it, while repeated field
lines are combined by caches, so a declared `Vary: Accept-Encoding` plus ours
reads as `Accept-Encoding, Accept`. What the helper adds is idempotence, so a
site that already named Accept does not end up with `Accept, Accept`.
The README documented neither content negotiation nor _headers, though it is the
repo's only doc surface and already carries a behaviour contract. Shipping
_headers support undocumented is the same failure this PR argues against for
_headers itself. Both are now described, including the Vary scoping, the
Cache-Control-on-miss rule, `.well-known`, and the empty-value config fallback.
Every claim in those sections was checked against a running server rather than
written from memory: pattern anchoring both ways, no Vary on assets, a
site-declared Vary surviving, Cache-Control withdrawn from a miss while
X-Frame-Options is not, the legacy text/x-markdown spelling, and the startup log.
Mutation coverage extended to all of it: resolveOrDefault ignoring empty, the
SPA fallback dropping Vary or advertising it unconditionally, and the
idempotence guard removed. The unconditional case is only observable for a root
with no index.html, so that shape is pinned explicitly rather than left as an
unverified assertion. Suite green under golang:1.26 with media-types installed;
golangci-lint unchanged at 1 finding against origin/main's 4.
Re-review — 75f4c8aAll four findings addressed, and the fixes are correct. Verified locally: 1. Docs (medium) — resolved. New "Content Negotiation" and "Response Headers ( 2. Config test (medium) — resolved, and properly boundary-pinning now. The tautology is gone: resolution moved into 3. SPA-fallback Vary (low/edge) — resolved. 4. Vary Add-vs-Set (nit) — resolved better than suggested. Rather than LGTM. 👍 |
PiyushSingh-ZS
left a comment
PiyushSingh-ZS
left a comment
There was a problem hiding this comment.
Reviewed commit-by-commit; all four earlier review findings addressed correctly in 75f4c8a. Verified locally: build, go test ./..., and golangci-lint all clean, and the config-guard fix is mutation-tested (goes red when reverted). LGTM.
Note: CI is currently red on the head commit (both jobs failed in ~2–3s, which reads as a setup/runner failure rather than a code failure given the suite passes locally) — worth a re-run before merge.
2 participants
Two independent fixes to what this server sends, plus a config-resolution bug found by running the real image. All additive and gated on something the published directory opts into, so a deployment that does neither is byte-for-byte unchanged. Reviewable commit by commit.
1. Serve markdown to agents (
Accept: text/markdown)Static site generators already emit
about.mdnext toabout/index.html. The file is on disk — the server just never looked for it. No build changes needed in any consuming site.Without this, an agent can only discover the markdown by reading the
<link rel="alternate">inside the HTML document it was trying not to download. It pays full price first, which defeats the point.Measured on a real build of zop.dev, which this server hosts:
/changelog/v1-33-0/docs/zopnight/introduction/learn/1-hop-adjacency/resources/blogs/agentic-ai-finops…Per Checkly's Feb 2026 survey, Claude Code, Cursor and OpenCode send this header today. On zop.dev, AI agents are ~30% of external requests — roughly 1:1 with human browsers.
Misses are answered in markdown too. An HTML error shell is unusable to such a client and not small: a real site's 404 page measured 144,188 bytes, sent in reply to a request the client could not parse. The markdown reply is 138 bytes and names
/sitemap.xmland/llms.txtso the reader can recover. The request path is deliberately not echoed into it —gosecflagged that as an injection sink (G705).Safety
*/*;q=0.8, which matchestext/markdownby the letter of RFC 9110. Matching wildcards would serve source to every visitor on the internet. Pinned by tests using verbatim Chrome and SafariAcceptstrings.text/html, text/markdown;q=0.1still gets HTML..mdis absent, or the URL has an extension.Vary: Accept, scoped to what can actually vary — without it a CDN can hand an agent's markdown to the next browser. It is set on the extensionless routes that can resolve to a.md, and on every miss (a miss answers in markdown whenever asked, whatever the path looks like). It is not set on assets: a hashed bundle can never negotiate, so keying caches onAcceptthere would fragment them for nothing — on precisely the responses_headersmarksimmutable..mdentry and distroless has no/etc/mime.types, soServeFilewould snifftext/plain. A directly requested.mdis left alone: browsers rendertext/plaininline but downloadtext/markdown, so relabeling it would turn a site's existing.mdlinks into download prompts. Worth knowing this differs by platform — most Linux distributions map.md, so a dev box already servestext/markdownwhile the distroless image that ships does not. The decision is unit-tested rather than asserted through a served header, which would only pin the host's MIME table.2. Apply the
_headersfile the published directory shipsThe Netlify / Cloudflare Pages convention. Generators emit it expecting the host to honor it; a host that ignores it fails silently while the file looks authoritative in the repo.
zop.dev has shipped a 2,437-byte
_headersfor months. Measured against the live site, not one rule was in effect:X-Frame-Options: DENYX-Content-Type-Options: nosniffReferrer-PolicyPermissions-PolicyCache-Controlon/_astro/*(immutable)Content-hashed bundles that could be cached for a year were served with no caching directive at all.
Parsed once at startup. No file → no rules → unchanged, so a published directory that ships none behaves exactly as it does today. All matching rules contribute in file order so a later block overrides an earlier catch-all. Malformed lines are skipped rather than failing the file. Applied before anything writes, so they cover hits, misses and the SPA fallback — a 404 that leaks framing protection is as exploitable as a 200 that does.
With one exception:
Cache-Controlis dropped on a miss. A site's cache directives describe the pages it publishes, not the ones it doesn't have. A/*.htmlblock withmax-age=300would otherwise pin a file that is merely un-propagated mid-deploy into every cache downstream for the rule's full lifetime. The SPA fallback keeps its caching — that is a real route being served, not a miss.Delegated paths count as the site too.
/.well-known/is handed to the next handler untouched, so an ACME challenge is never given an extension or swallowed by the SPA fallback. That early return also skipped the header rules, so a site declaringX-Frame-Optionsfor/*got it everywhere except there. Rules are now applied before the branch — they depend only on the request path — and delegation itself is unchanged, verified by a test that pins the path arriving unrewritten. Since the delegate picks its own status, the cache directives are withdrawn on the way out rather than up front:Cache-Control200— a real ACME/.well-knownfile404— absentThe response writer is wrapped only on that path: doing it on the main serving path would hide net/http's
io.ReaderFromfromhttp.ServeFileand cost every static file its sendfile fast path.Operator note: a reverse proxy in front of this server may set some of these itself.
Strict-Transport-Securityis the usual one (ingress-nginx sendsmax-age=15724800, nopreload, with replace semantics) — where that's the case it will most likely keep winning, so apreloaddeclared in this file is unlikely to suddenly go live. The other five rarely have a proxy counterpart.3. Empty config values fell back to nothing
Found by running the actual distroless image rather than a native binary.
GetOrDefaultonly falls back when a key is absent. The shippedconfigs/.envsetsSTATIC_DIR_PATH=andDEFAULT_EXTENSION=empty, so a deployment supplyingSTATIC_DIR_PATHvia the environment got""— every lookup silently rooted at the process working directory:./staticSTATIC_DIR_PATHSTATIC_DIR_PATHThe default shape was unaffected — but by luck, not design. The startup line now names the resolved directory, since "0 rules" is normal for a site without the file and indistinguishable from a misrooted path otherwise.
Testing
go test -racegreen. 20Accept-parsing cases;_headerssuite built on a verbatim excerpt of the real zop.dev file.*/*as markdown; ignoring q-values; negotiating with no.md; reverting the markdown 404; not applying header rules; applying only the first matching rule; dropping pattern anchoring;Varyunconditional on hits;Varydropped on a miss; theCache-Controlwithdrawal removed; the markdown Content-Type unscoped;negotiable()ignoring either the extension or the root;.well-knownskipping the rules; the delegated writer unwrapped; scrubbing on every status instead of errors only;withdrawCacheDirectivesas a no-op.golang:1.26withmedia-typesinstalled, so/etc/mime.typesreally containedtext/markdown md markdown— the platform shape that behaves differently from a dev Mac and from the distroless image.golangci-lint: 1 finding, against 4 onorigin/main— nothing new. The three that dropped aregosecG703 taint hits onhttp.ServeFile, whose call sites are untouched; gosec's taint walk is sensitive to unrelated edits in the same function.Verified in the real image
gcr.io/distroless/static-debian12, built from this Dockerfile, serving a real 5,978-page build — 16/16: correct Content-Type for html/css/svg/txt/md with no/etc/mime.types; every_headersrule applied including on 404s; markdown negotiation;Vary; 138-byte markdown 404; pages without a.mdstill 200.Regression evidence
Base and patched binaries run side by side on the same build, diffing status, Content-Type, Vary, Cache-Control, X-Frame-Options, Content-Length and body SHA-256 across path ×
Acceptcombinations — a browser string,*/*,text/html, text/markdown;q=0.1,text/markdown, and no header.Across the whole matrix the only changed-or-removed field is the one intended negotiation (
/about+Accept: text/markdown). Every other delta is an addition of a header the site's own_headersfile declares:main/,/index.html,/style.css,/robots.txt,/readme.md+ Cache-Control,+ X-Frame-Options/_astro/app.*.js+ Cache-Control: …immutable,+ X-Frame-Options— noVary/about,/about/+ Cache-Control,+ X-Frame-Options,+ Vary: Accept/missing+ X-Frame-Options,+ Vary: Accept— noCache-Control/.well-known/acme-challenge/<tok>+ Cache-Control,+ X-Frame-Options— onmainthis response carried no headers at all; body unchanged/about+text/markdown+ Vary,+ Content-Type: text/markdown/readme.mdno longer appears as a Content-Type change; it is byte- and header-identical tomainapart from the_headersadditions.An earlier run before the
_headerscommit compared 1,505 pairs across 5 non-markdownAcceptvariants with 0 mismatches — though note that run compared status + Content-Type + body only, which is why theVaryandCache-Controlscoping above needed the wider diff to surface.Rollout
The markdown half is inert until the site being served actually emits
.mdsiblings — a build that doesn't is byte-for-byte unchanged, so this can land ahead of any generator work. The_headersand config halves are independent and take effect as soon as a directory ships a_headersfile.