feat: allow configuring the DATA frame budget - #942
Conversation
The connection keeps a shared budget that bounds the framing overhead of DATA frames that have been received but not yet read by the application. Undersized non-final frames consume from it, larger frames and frames released to the application replenish it, and exhausting it terminates the connection with a GOAWAY carrying ENHANCE_YOUR_CALM. Until now that budget was hard-coded to DEFAULT_DATA_FRAME_BUDGET, which leaves no way to tune the tradeoff between tolerating peers that legitimately send many small frames and reacting quickly to ones that abuse them. Expose it as `data_frame_budget` on both the client and server builders, plumbed through to `Counts`, and keep the existing constant as the default so behavior is unchanged unless configured.
|
Thanks for the PR! Sorry for the disruption. I had hoped that by releasing back to the budget when the data frame was consumed, it would just work out. We could add a config option, as you've done. But, I'd like to explore if we can make it just work without one first. Is the problem that you send a bunch of small data frames, and don't read them that quickly? Or are they sent on a much larger number of active streams? |
|
Hey @seanmonstar, thank you so much for checking this PR so quickly. The problem is that we have small data frames on large number of active streams. Of course it varies depends on the workload shape. Some streams can generate many small steps each with tiny outputs, which triggers the issue if we have many multiple invocations running concurrently on the same connection. Once I put the budget to higher value, the system had enough time to process (and replenish) the budget and I no longer hit the issue. Of course there is no "good" value to the budget, and it's totally up to our workload shape. Hence I thought a configuration option could be the best approach. |
|
Hi @seanmonstar - I'd like to confirm that I've hit this too. We run a gRPC server where we send lots of small updates. We're quite latency sensitive, so batching isn't really an option for us. Would love to see this configuration option exposed in |
|
Looking at this some more, would an increase to about 1000 (currently 100) be enough in most cases? Again, if we have to, we can add a config option, but I'd rather not have to propagate something through all the other dependencies. Especially if we eventually came up with a better budget/defense mechanism in the first place. |
|
@seanmonstar IMHO since h2 is a low level crate, it shouldn't have any assumptions about the default values. I am not sure that a budget of 250KiB would be enough for all our use cases. We personally use |
|
I mean 250k would definitely work with my local test env where I can reproduce this issue. But I am not sure about all other scenarios in production where we can have many h2 connections open at max number of stream capacity per connection (and heavily loaded server) specially that the number of frames and sizes of the data is largely unknown. |
|
@seanmonstar I agree with @muhamadazmy here. I don't think Are you sure that this hard cap is the best way to handle this? I wonder if it would be better to handle this via limiting actually-empty frames and/or backpressuring via the existing connection/stream windows. Maybe including the overhead of |
|
Also we use this through Tonic - so having |
|
OK, yea true, a config here doesn't mean it has to bubble all the way up just yet. We can adjust defaults and heuristics at a higher layer.
Actually empty frames are one case, but it's also been noticed that, for example, 1-length frames are limited by flow control, but they still 'cost' more memory than that because of overhead. We can do things to reduce that overhead, but it still is a potential abuse vector. So, I'll move ahead with providing the config. If a better way to distinguish and improve the heuristic it possible, we can discuss in a separate issue. Thanks all! |
|
Thank you so much @seanmonstar for your time and great effort ❤️ |
Problem HTTP/2 flow control limits DATA payload bytes, but not the framing overhead from excessive numbers of small frames. A peer could fragment data into many tiny frames, causing disproportionate memory usage from queued events while remaining within flow-control windows. Solution Backport the framing overhead budget and empty frame handling from master (hyperium#935, hyperium#940, hyperium#942, hyperium#945, hyperium#946) to the 0.3.x maintenance branch. Validation Ran the full test suite and added regression integration tests in stream_states.rs.
3 participants
Follow up to #935
Why
Our application protocol can (and will) send many small frames over the h2
connection. This was not an issue until the latest h2 update (PR #935) which
penalize small frames.
Fix
Until now the data-frame-budget was hard-coded to DEFAULT_DATA_FRAME_BUDGET, which
leaves no way to tune the budget value.