One request per 32 MB gets me rate limited, and a 429 then kills the session for good

[Rasmusmart57]

What happened?

The reader makes one HTTP request per 32 MB. On a 7 GB file that is about 200
requests. On a 20 GB file about 600. My origin counts requests and answers 429
when you make too many, so a long episode eventually trips it.

Every other player I have used against the same origin opens one connection and
reads the whole file through it. One request. I have used this kind of host for
years and never seen a 429 until now.

Once the 429s start, playback keeps going for 30 to 50 seconds off the segment
cache, then goes black and stays black. If I leave the player and press play on
the same stream it starts immediately. The source was never dead, it was being
throttled.

Two things I would like.

1. Let the host set how big each request is. persistentRangeBytes is a static
   let, and windowHighWater is an AVIOReader init parameter that nothing passes, so
   there is nothing to reach from outside the engine. I patched persistentRangeBytes
   to 256 MB in my own copy, which is not something I can keep doing.

2. Do not treat a 429 as a dead source. AVIOReader.isRateLimitStatus already
   knows the difference, but it is only used inside AVIOReader. It never reaches
   handleVODReadErrorExit in HLSVideoEngine+LiveReopen.swift, which spends
   readErrorReviveGate (maxAttempts: 2). Two 429s and the session ends with "not
   readable in this session". Each of those attempts reopens from offset 0, gets
   refused there too, and takes about 35 seconds to fail. Then my host hands off to
   its other engine, which asks the same origin and gets the same 429.

I am not asking for the transport to change. I read #220 and #310 and I am not
going to argue with them. I just want a way to make fewer requests.

How I know it is the range size and not the window: consecutive pump generations
advance by exactly 33554432 bytes, every time. That is the range boundary, which
is also the first case your own comment on that constant describes.

Steps to reproduce

No clean repro, sorry. It needs an origin that counts requests, and 10 to 30
minutes of continuous 4K playback before the count gets high enough.

1. Play a 7 to 20 GB VOD file over HTTPS from an origin that rate limits by
   request count.
2. Let it run. The reader issues one ranged GET per 32 MB.
3. After a few hundred requests the origin answers 429 to everything, including
   HEAD and range probes.
4. The pump wedges, the reopen path spends both revive attempts on the refusals,
   and the session is declared not readable.

Happens on every long session here. I can run builds with extra logging on real
hardware and get you traces the same day.

AetherEngine version or commit SHA

6.25.0

Host app

Custom / my own integration

Platform

tvOS

OS version

26.6

Device / chip

Apple TV 4K 3rd gen

Playback path

Native AVPlayer

Source media (for playback bugs)

4K HEVC, Dolby Vision profile 7, DTS-HD MA, MKV remuxes. 7 GB and 20 GB. Served
over HTTPS by a commercial proxy that counts requests, 302s to a signed CDN
target, and has no suffix-range support.

Error codes / log lines

One connection per 32 MB:


[AVIOReader] pump conn start gen=91 offset=5577199155 len=32MB
[AVIOReader] pump conn start gen=92 offset=5610753587 len=32MB
[AVIOReader] pump conn start gen=93 offset=5644308019 len=32MB
[AVIOReader] pump conn start gen=94 offset=5677862451 len=32MB


Once rate limited, a reopen attempt fails like this:


[AVIOReader] pump conn start gen=9 offset=0 len=32MB
[AVIOReader] pump gen=9 rejected response status=429 at offset 0
[AVIOReader] pump reconnect exhausted at offset 0 status=429 (7 consecutive rate-limited)
[AVIOReader] slow read: 32533ms at offset=0 detour=0(0ms,0fetch) stallWaits=6(549ms,6signaled) tailWaits=0(0ms) reconnects=6 backoff=31978ms lockWait=0ms connect=3ms staleGenDropped=0b iters=13 unaccounted=0ms gen=3->9
[HLSVideoEngine] restart at idx=130: #169 suspect-dead demuxer; reopen failed (Demuxer: open failed (Operation not permitted (-1))), reusing the demuxer
[HLSVideoEngine] producer restarted at idx=130 (seek=520.85s [absolute source-PTS], restart took 33568ms; stopWait=0ms reopen=33567ms seek=0ms build=0ms)
[AVIOReader] Detour rate-limit gave up at offset 3976056735 (12 consecutive rate-limited)
[HLSSegmentProducer] demuxer.readPacket threw: Demuxer: read failed (Operation not permitted (-1))
[HLSSegmentProducer] pump finished: reason=readError(-1) packetsRead=0 packetsWritten=0 lastError=-1 elapsed=184ms cacheCount=82
[HLSVideoEngine] #169 VOD readError revive cap reached (3 failures, cap 2); giving up (source not readable in this session)


The handoff to my other engine, same origin, same answer:


[PlaybackCoordinator] Aether→MPV handoff: Source read failed (code -1) @ 556.11s
[MPV][ffmpeg] warn: https: HTTP error 429 Too Many Requests
[MPV][stream] error: Failed to open https://<origin>/.../<file>.mkv
[MPV] End file error: loading failed


Starting over does not help, the probes are refused too:


[AVIOReader] pump tail prefetch skipped: this origin declined suffix ranges earlier this session (status=429 (no suffix range support))
[AVIOReader] Range probe (bytes=0-) didn't yield a size
[AVIOReader] HEAD failed (HTTP 429)
[AVIOReader] Range probe (bytes=0-1) didn't yield a size
[AVIOReader] no probe resolved a size, streaming mode (forward-only)
[AVIOReader] size probe empty; reusing cached 20351996087 bytes to keep seekability (#112)

Anything else

Checked against main before filing. winHighWaterDefault is still 16 MB for VOD
against 64 MB for live, readErrorReviveGate is still maxAttempts: 2, and
HLSVideoEngine+LiveReopen.swift has no reference to rate limiting anywhere. No
open PRs touching any of it.

On #307, since it is still open waiting for a retest. Different origin from the
original report, so take it for what it is worth, but the 6.11.0 ladder does what
you said it would:

[AVIOReader] pump gen=6 rejected response status=429 at offset 0
[AVIOReader] pump conn ended at offset 0 status=429, reconnecting (streak=4 retryAfter=0.0s)
[AVIOReader] pump reconnect exhausted at offset 0 status=429 (7 consecutive rate-limited)

The refusal is named, the streak charges, backoff is real (31978ms across six
reconnects), and it ends the read instead of spinning. No sign of the
925-generation spiral. My origin sends no Retry-After, hence retryAfter=0.0s, and
it exhausts at 7 rather than 12 because these are rate-limit statuses on the
shorter streak.

[Rasmusmart57]

I've done some more digging and research on this and it's turned into a rabbit hole lol. I'm not 100% sure about any of it, but here's where I've got to.

First thing, I tested my own theory and it didn't really hold up? I patched my copy twice, winHighWater 16 to 64 MB and then persistentRangeBytes 32 to 256 MB. That second one should be about eight times fewer requests per file. The 429s came back unchanged, twice in the same evening. So request volume either isn't the trigger or isn't the only one.

I went and re-read my providers doc, which i only kind of skimmed through to get an understanding earlier, my fault. TorBox describe their CDN as rate limited and connection limited, both. They tell you not to use the same download link in more than one app or download manager at once, and to stick to 1 connection for big downloads. So concurrency per link is at least part of what they're counting, I guess.

So, httpMaximumConnectionsPerHost is per URLSession, and AVIOReader has four of them. persistentSession for the pump's ranges, cap 2. chunkSession for detour blocks, cap 2. probeSession for size probes and HEAD, which is on URLSessionConfiguration.default so cap 6. Plus a streamSession built per call. Those caps don't compose into anything, they're four separate pools, so a pump range and a detour block and a probe can all be open against the same signed CDN URL at the same time. Still the same on main today if im not mistaken?

Two readers running at once shows up in my earlier traces, both on persistentSession, different offsets, separate generation counters:

[AVIOReader] pump conn start gen=12 offset=2926399027 len=32MB
[AVIOReader] prefetch conn start gen=4 offset=2750276265 len=32MB

In the session that actually died the subtitle prefetcher was off, so it was the pump and detour fetches instead, which is still two pools:

[AVIOReader] pump conn start gen=9 offset=0 len=32MB
[AVIOReader] pump gen=9 rejected response status=429 at offset 0
[AVIOReader] Detour rate-limit gave up at offset 3976056735 (12 consecutive rate-limited)

It also fits two things my first theory never explained. I sometimes get a 429 not long after first opening a file, long before any real number of requests:

[AVIOReader] pump tail prefetch skipped: this origin declined suffix ranges earlier this session (status=429 (no suffix range support))

And backing out of the player and pressing play on the same stream works instantly, which makes sense if tearing the session down takes the connection count back to zero?

I've collapsed the three delegate-less sessions into one pool with the cap at 1 in my copy. They all use per-task delegates already so sharing looked free. No idea yet whether it helps, I'll report back after a few long sessions, if I remember!

One thing I can't resolve from outside: if the CDN speaks HTTP/2 then everything multiplexes over one connection anyway and the cap does nothing, which would mean they're counting requests in flight rather than TCP connections. Could go either way. And the queueing might just move the problem, pump stuck behind a slow detour instead of getting refused. If that happens I'll say so.

So the thing I'd ask for now is a bit different from what I opened with. Less a cadence knob, more one connection budget per origin that all the reader's paths share, and some way for a host to say this origin only allows one. The per-session caps look like a global limit and aren't one, which seems worth fixing whether or not my provider is the only one that notices. Sorry for the churn on this one! Thanks for reading aswell :)

[Rasmusmart57]

Watched a full episode on the branch. No black screen, ran all the way to EOF. So from where I'm sitting it's fixed.

But not for the reason either of us thought, and I think that part matters more than the result.

The 429s came back. Three times, roughly 26, 35 and 41 minutes in. Every one of them with a single request in flight:

[OriginBudget] https://<origin> answered 429; concurrency budget uncapped -> 1 (peak was 1)
[AVIOReader] slow read: 45266ms at offset=10453303361 ... origin=0inflight/1peak limit=1 refusals=7
[AVIOReader] slow read: 45440ms at offset=14272858557 ... origin=1inflight/1peak limit=1 refusals=14
[AVIOReader] slow read: 43929ms at offset=17488099403 ... origin=0inflight/1peak limit=1 refusals=28

So concurrency isn't the trigger on this origin. My guess was wrong, and so is the premise the budget is built on, at least here. It refuses a lone request, which means it's counting something else. Requests over time, or bytes. Can't tell which from outside.

What actually saved the session is the metered revive gate, which is the half of the PR I barely commented on. The pump died three times and got rebuilt at 3s, 8s and 20s, the last one back in 2.3 seconds:

[HLSVideoEngine] #377 VOD pump died mid-session on a METERED origin (readError -1); waiting 3s before rebuilding the producer at 1621.70s -> seg405 (attempt 1/4)
[HLSVideoEngine] producer restarted at idx=633 (seek=2532.36s [absolute source-PTS], restart took 2308ms; stopWait=0ms reopen=1877ms seek=430ms build=0ms)

I didn't notice any of it while watching. tcs=playing the whole way through, stall=0, forward buffer never dropped below 3.7s. The segment cache covered every gap.

The budget still earns its keep, mind you. Its request telemetry is the thing that proved my own theory wrong, which I wouldn't have got to otherwise.

One small thing, not blocking: scrubbing goes dead for about 8 seconds during the refusal windows (seek#7 nativeScrub stalled, landing watch expires). The picture keeps playing fine, it's only the scrub that gives up.

[Rasmusmart57]

Follow-up, and possibly a link worth checking: I think #380 might be describing my root cause from the other end.

Their shape is a stale pinned redirect target answering 509 forever while a fresh resolve through the source URL connects in 452 ms. Mine is 429 instead of 509, and mine also happens mid-playback rather than only after a pause, so it isn't identical. But the recovery is the same story, and it's the thing I originally opened this issue about: backing out and pressing play on the same stream works instantly.

From the run I just posted, all three failures were consecutive refusals at one offset against the pinned host, and every recovery came only when the segment layer built a fresh demuxer whose reader re-resolved:

[AVIOReader] slow read: 32513ms at offset=0 ... reconnects=6 backoff=31968ms ... refusals=9
[AVIOReader] pump conn start gen=8 offset=0 len=32MB
[AVIOReader] Cached resolved URL host=nexus-175.neur.tb-cdn.st
[AVIOReader] pump gen=8 first data after 210ms

45 seconds of refusals on the pin, then 210 ms once it re-resolved. That's the same gap as the 452 ms in #380.

The other tell is that the host changes between sessions in my traces (nexus-128 one evening, nexus-175 the next), so these are per-session signed targets that presumably expire on their own schedule.

So the budget in #384 keeps the session alive through it, which is what I retested and it works. But if #381 is right then a bounded keep-pin might stop it happening at all, at least for the refusals that are a dead pin rather than a real limit. No idea if that's most of mine or a few of them. Happy to run a build with both if that's useful.

[superuser404notfound]

Reopening: closed automatically by the merge. The retest is what closes this, and it has not happened yet.

[superuser404notfound]

No churn to apologise for. The second comment is the more useful report, and the part of it that could not be settled from outside the engine turned out to be the part that mattered.

Shipped in 6.28.0.

The measurement first, because it changes what to patch

There is now a per-origin request budget, and with it the engine can log what it is actually doing. Against a test origin with real Range support that answers 429 above one in-flight request, playing one file:

run	peak concurrency the origin saw	429s	requests served
budget learns from the refusals	2	2	3, session survives
maxConcurrentSourceRequests: 1	1	0	1

The request that caused the collision was the tail prefetch, the speculative bytes=-65536. It leaves microseconds before the first data connection, so the origin sees two requests on the very first open and it is the pump that gets refused. That is your "429 not long after first opening a file, long before any real number of requests", and it is also why the suffix probe was refused in your restart trace. It now takes a slot only if one is free, and it counts when it does. In the first run of the instrumented build the budget reported a peak of 1 for an origin that had just seen 2, entirely because of this one request.

About the HTTP/2 question

That one cannot be answered from the outside, and it is worth answering before more patching, because it may make your own patch a no-op. Over HTTP/2 URLSession multiplexes every request of a session onto one connection, so httpMaximumConnectionsPerHost bounds nothing there while the origin still counts each request. Collapsing three sessions to a single pool with cap 1 would then change nothing on the wire.

URLSessionTaskMetrics.networkProtocolName is the only place that says which case an origin is, and it was read nowhere in the tree. It now logs one line per origin:

[AVIOReader] origin transport: h2 at https://cdn.example.com:- (connection new, multiplexed, so a per-session connection cap bounds nothing)

That line from a run against the real provider is the single most useful thing the next trace can carry. If it says h2, connection caps are not the lever and request count is.

What the budget does

It counts requests rather than TCP connections, since that is what an origin metering us counts, and it is keyed on scheme+host+port rather than the full URL, so a rotating signed token shares one ceiling instead of starting a fresh one per refresh. Every path goes through it: pump ranges, detour blocks, size probes, the tail prefetch, the streaming connection, and the subtitle side reader that shares the same static pools.

Counting is unconditional, capping is not. With no limit set nothing waits and behaviour is unchanged. A limit arrives either from the host, or from the origin itself, halving from the concurrency actually reached on each refusal (an origin refused with four open has said something about four, not about one).

try await engine.load(url: url, options: LoadOptions(
    maxConcurrentSourceRequests: 1
))

At one slot the speculative parallel paths switch off rather than queue. That is deliberate and it is the answer to your last worry: queueing a detour block behind a slot the pump itself holds is exactly the "pump stuck behind a slow detour" case, so instead the detour falls back to repositioning the persistent connection, which is a path it already had. Nothing can block on a slot its own caller holds.

The 429 no longer ends the session

You were right that isRateLimitStatus never reaches the revive path, and the reason is worse than it looks: the give-up arm returns a bare -1, so the classification exists and is discarded one frame later. handleVODReadErrorExit was seeing what a genuinely dead source produces, spending both attempts inside a minute reopening from byte 0 against an origin refusing precisely that.

A metered read error now gets its own budget with a growing backoff (3 s, 8 s, 20 s, 45 s) instead of an immediate reopen, and if it does run out the terminal error carries a new kind:

if info.kind == .sourceRateLimited {
    // metered, not lost: the same request works later
}

For your setup that is the one to branch on before the MPV handoff. The handoff in your trace asked the same origin and got the same 429, which is the correct behaviour for a dead source and the wrong one here.

One thing worth checking on your side: a proxy that 302s to a signed CDN target is two origins. The refusal comes back from the CDN, so the CDN is what gets throttled, and the source URL only carries the timestamp so the classification is still findable. That is handled, but it means the transport line may name the CDN host rather than the one you loaded.

Not claimed

This bounds and reveals what the engine asks of an origin. It cannot know what your provider actually permits, and whether they meter request count, concurrency or something else is still theirs to say. aetherctl play --max-concurrent-requests N reproduces a capped origin locally if that is useful for narrowing it.

Leaving this open for the retest. The two lines worth pulling from a long session: the origin transport: line, and any slow read: line, which now ends with origin=<n>inflight/<peak>peak limit=<n>.

[superuser404notfound]

My previous comment crossed with these two and answered the one before them, so this is the reply to the retest.

The short version: you are right on both counts, and the link to #380 is the more important of the two comments. It is merged, plus a follow-up.

Your origin is very likely not counting anything

That is the part I want to push back on, gently. Your reading was "it refuses a lone request, which means it's counting something else, requests over time or bytes". There is a second explanation that fits your trace better, and it is the one #380 describes: a pinned redirect target whose session has expired refuses a lone request too, forever, while a fresh redirect through the source connects immediately. Not metering at all. A dead lease.

Your own numbers are what make that the better reading:

• All three failures were consecutive refusals at one offset against the pinned host, and every recovery came when a fresh demuxer's reader re-resolved. 45 s of refusals on the pin, then first data after 210ms. The A stale pinned redirect target answers 509 forever after a pause-resume; the keep-pin rule has no bound, the streak resets on window-served reads, and the faulted-refill pacing is erased by its own reconnect #380 reporter measured 452 ms on a different CDN after ~85 s of the same.
• Your pinned host changes between sessions (nexus-128, nexus-175), so these are per-session signed targets with their own expiry.
• The budget reported uncapped -> 1 (peak was 1) at each refusal. That is the budget learning nothing, which is what "this refusal is not about concurrency" looks like from the inside. It was built on your second hypothesis and your retest is what disproved it. That it can say so is the point of counting requests at all.
• refusals= is cumulative per origin, so your 7, 14, 28 are increments of 7, 7 and 14. The ladder gives up at 7 consecutive. Each window spent whole ladders and stopped, which is the cap ending it rather than a meter relenting.

A meter that counts requests over time would not wait 26 minutes, refuse one offset until a re-resolve, then run clean for another nine.

What is merged

#381 by @tschuegy, the fix your second comment pointed at, is on main (d35d078). The keep-pin rule was absolute: a 429/503/509 kept the pinned target no matter how long the refusals lasted, because on a connection-capped panel re-resolving spends the one request there is no room for. That grace is now bounded. Past three paced attempts the pin is dropped for one re-resolve through the source, and the fresh target's 200/206 re-pins. Two smaller mechanisms from the same report went with it: window-served reads no longer reset the reconnect streaks (draining read-ahead is not network progress, so the ladder never got past streak=1 while the runway lasted), and the faulted-refill pacing is no longer erased by the reconnect it authorises.

Reviewing that PR turned up two more branches of the same mistake, now in #386: the retained head/tail spans and the detour cache's resident-block hit were also clearing the ladders on bytes that never crossed the network. The span one matters for you specifically, because it is the branch a metered origin still reaches (a refusal caps the origin to one request, which disables the detour but not the spans).

For your session that should turn each 45 s window into roughly four attempts and a re-resolve. What it will not do is prevent the pin from dying in the first place, so expect the stall to become short rather than to disappear.

The scrub

Not a separate defect, and the 8 s is exact rather than approximate: nativeScrubLandingBudgetSeconds is 8.0. The watch is armed when the restart run drains and closes when the picture actually reaches the target; past the budget, "not landed" is the honest report, so it publishes .stalled instead of latching a landing that never happened. During a refusal window the picture genuinely cannot reach the target, so the watch expiring is the correct outcome and the refusal window is the thing to shorten. If those windows stay long after this, say so, because then the budget is the wrong shape for a metered source rather than for a lost one.

The two lines from the next run

Both pin lines were behind #if DEBUG, which is why Cached resolved URL host= shows up in your traces at all: you build the engine yourself. They are release-visible now, and the drop line names why:

[AVIOReader] Dropped resolved URL cache (rate-limited x3 through pinned URL)
[AVIOReader] Cached resolved URL host=<fresh target>

Those two, adjacent, followed by a fast first data, are the proof that the re-resolve is what heals it. If instead the fresh target refuses identically, the pin was never the problem and your origin really is metering something, which is worth knowing and would be a different fix.

The origin transport: line from #377 is still the one worth pulling too. If it says h2, your own patch collapsing the sessions to one pool with cap 1 changes nothing on the wire, and it would be good to stop paying for it.

Yes to a build with both, and thank you for retesting something that turned out to disprove its own premise. That is more useful than a clean pass.