fix: harden CAR streaming and truncation - #11409
Merged
Merged
Conversation
ls, dag get and dag export each hand their work to a goroutine and read the result back over a channel or pipe. Decoding and encoding there runs whatever codec a block's CID names, so it runs third-party code, and a panic on a detached goroutine ends the daemon rather than the command. Each now recovers, logs, and reports through the channel it already uses. In dag export the recover is registered after the existing cleanup defer so it runs first, while errCh is still open.
Picks up the CAR streaming and object parsing work from ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches for now; swap for the tagged releases before merging.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
gammazero
approved these changes
Jul 27, 2026
lidel
force-pushed
the
harden-block-decoding
branch
2 times, most recently
from
July 27, 2026 13:57
f16762e to
86730de
Compare
go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch.
lidel
force-pushed
the
harden-block-decoding
branch
from
July 27, 2026 14:17
86730de to
0ab73ab
Compare
lidel
added a commit
that referenced
this pull request
Jul 27, 2026
* fix: recover from panics in detached goroutines ls, dag get and dag export each hand their work to a goroutine and read the result back over a channel or pipe. Decoding and encoding there runs whatever codec a block's CID names, so it runs third-party code, and a panic on a detached goroutine ends the daemon rather than the command. Each now recovers, logs, and reports through the channel it already uses. In dag export the recover is registered after the existing cleanup defer so it runs first, while errCh is still open. * chore: update boxo, go-ipld-git and go-unixfsnode Picks up the CAR streaming and object parsing work from ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches for now; swap for the tagged releases before merging. * chore: update boxo, go-ipld-git and go-unixfsnode go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch. (cherry picked from commit f9baf8f)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A CAR response that stopped partway through looked exactly like a complete one. By the time anything goes wrong the status line and headers are long gone, and
X-Stream-Erroris set once the body is already streaming, so it rarely reaches the client and never survives a reverse proxy or CDN. Debugging a short response meant guessing which hop dropped it. Nothing bounded how deep a CAR walk goes either, andls,dag getanddag exporteach hand their work to a goroutine and read the result back over a channel, where a panic ends the daemon rather than the command.Fix
Dependency updates carry the rest: ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches, so this needs their tagged releases before it can merge.