fix(transport): stop caching a soft HTTP/1.1 fallback as the host's protocol

[burruplambert]

Problem

In auto mode doAuto learns the protocol a host speaks and caches it per session in protocolSupport, so later requests skip the negotiation (added for #68). Two different facts were being written into that cache under the same key:

• "this host negotiated http/1.1 via ALPN": a property of the host, correct to cache;
• "the H2 attempt failed this time and the HTTP/1.1 fallback got the request through": a property of one attempt. A reset or timed-out handshake, a first dial that did not survive, anything transient.

The second was written exactly like the first, at two sites: the bottom-of-doAuto fallback, and, for presets with H3 support, the fallback inside raceH3H2 (H2 failed for a non-ALPN reason, doHTTP1 succeeded, reported back as ProtocolHTTP1 with a nil error, which doAuto then cached).

Only the first request to a host (or the first after Refresh) can hit this: once a host is known as HTTP/2, a later transient failure serves one request over HTTP/1.1 without touching the cache. That makes the failure silent and permanent rather than rare: one bad handshake on the opening request and every following request to that host goes out over HTTP/1.1, and with it a different fingerprint, with nothing that would ever re-probe. Long-lived sessions and anything that creates sessions often (each starts with an empty cache) are the most exposed. On a direct connection the initial dial in getOrCreateConn is not retried (retryDial's extra attempts apply only behind a proxy), so a single transient handshake failure is enough.

Fix

The fallback runs exactly as before and the request still succeeds; it just no longer writes the cache.

• raceH3H2 reports ProtocolAuto for a response it served through its H1 fallback, documented as "nothing learned about the host", and doAuto skips the cache write for it.
• The bottom-of-doAuto fallback no longer writes.

The next request attempts H2 again and, when that works, the host is cached as H2 like any other. ALPN downgrades and the DisableHTTP2 preset branch are cached as before, so an H1-only host still pays the failed H2 attempt only once. Forced protocols never consult the cache and are unaffected.

What each request does on the wire is unchanged. Every path returns exactly what it returned before; the diff removes two map writes and adds comments. The stream path (doStreamAuto) was already correct: doStreamHTTP2OrHTTP1 falls back to H1 only on an ALPN mismatch, so it never had a soft fallback to cache.

Precedent: the library already fixed the structurally identical "cached thing goes stale and pins a long-lived session" problem for ECH configs by making that cache self-heal. This is the protocol cache's equivalent, done by not caching what was never a fact about the host.

Tests

transport/protocol_cache_test.go, all against real in-process servers:

• A listener that drops the first N TCP accepts stands in for a transient handshake failure; the H1 fallback dials fresh, lands on connection N+1 and succeeds. TestAutoDoesNotCacheSoftH1Fallback drives both doAuto branches: a preset without H3 support (plain H2 attempt, N=1) and one with it (race branch, N=2, because the probe connects before the dial). It asserts nothing is cached after the fallback and that the second request goes out over H2 and is then cached as H2. On the previous code it fails in both branches with host cached as h1.
• TestAutoStillCachesALPNDowngrade: an http/1.1-only server is still cached as HTTP/1.1 and served from the cache next time, on both presets.
• TestAutoCachesH2: a healthy H2 host is cached as H2 on the first request, as before.

transport, session, root, pool and client (minus the two tests that hit tls.peet.ws, whose certificate has expired) pass under -race. The race-branch case takes about 6s because it is the both-probes-fail path, which waits out the H3 probe budget; that is the scenario under test, not the fix.

Docs

CHANGELOG entry under Unreleased. The auto-negotiation page (step 6) now notes that the fallback is the one case that is not cached, and why.

Two observations, deliberately not changed here

• When both H2 and its H1 fallback fail, the H3-capable path makes a second H1 attempt: raceH3H2 tries H1 itself, and doAuto's bottom fallback then tries again on the non-ALPN error. Presets without H3 support get one attempt. That is inconsistent and undocumented, but removing it changes what happens on the wire in a failure case, so it is not part of this fix. Happy to send it separately if you agree it should be one attempt in one place.
• The known-ProtocolHTTP3 branch of buffered doAuto has no fallback if the H3 request fails, while doStreamAuto falls back to H2/H1. Left as is.

Unrelated: TestSessionDo_NonReplayableBodyOn307 in the root package is a pre-existing timing flake for me, roughly one run in ten on untouched main (forced H1 against a raw listener, write: broken pipe). Mentioning it only so a red run here is not mistaken for this change.

[vercel]

@burruplambert is attempting to deploy a commit to the sardanioss' projects Team on Vercel.

A member of the Team first needs to authorize it.