Keep the upload destination file open across chunks - #94
Merged
Conversation
uploadChunk() was opening, writing, and closing the destination file on every single chunk - real per-chunk overhead (open/close syscalls, plus whatever flush happens on close), especially costly against the FAT-formatted USB target this writes to. For a ~1.2GB image at 3MB chunks, that's ~400 chunks each paying this cost. uploadMagicChunk already gets this right - it opens the file handle once (lazily, on the first chunk) and reuses it for the rest. This brings the regular upload path in line with that: uploadStart opens the file once and stores the handle on state.File, uploadChunk just writes to it, and uploadFinish (uploadCancel already did) closes it. Also fixes a pre-existing bug found along the way: the base64 decode error in uploadChunk was captured but never actually checked. One piece of a larger investigation into #75/#61 (slow/stalling uploads) - this addresses the server-side per-chunk overhead specifically; the client-side serialized, non-pipelined chunk loop and the base64/JSON wire overhead are separate, larger changes not included here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drives upload_start -> ~22 upload_chunk calls -> upload_finish through the real handlers (not mocked), and checks the resulting file on disk matches the original payload byte-for-byte. Exercises the state.File handle kept open across chunks, so this would actually catch truncation, overwriting, or interleaving bugs - a build-only check can't. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
One piece of the investigation into #75/#61 (slow/stalling WiFi uploads).
uploadChunk()was opening, writing, and closing the destination file on every single chunk:For a ~1.2GB image at 3MB chunks, that's ~400 chunks each paying open/close syscall overhead plus whatever flush happens on close - especially costly against the FAT-formatted USB target this writes to.
uploadMagicChunkalready gets this right - it opens the file handle once (lazily, on the first chunk) and reuses it for the rest.Fix
Brings the regular upload path in line with the magic-upload path:
uploadStartopens the file once and stores the handle onstate.File,uploadChunkjust writes to it,uploadFinishcloses it (uploadCancelalready did this correctly).Also fixes a real pre-existing bug found along the way: the base64 decode error in
uploadChunkwas captured but never actually checked.Not included: the client-side serialized, non-pipelined chunk loop and the base64/JSON wire overhead (~33% inflation) are separate, larger changes - this PR is just the server-side per-chunk overhead.
Test plan
go buildpassesTestUploadChunkRoundTrip: drivesupload_start-> ~22 realupload_chunkcalls ->upload_finishthrough the actual handlers (not mocked), verifies the resulting file matches the original payload byte-for-byte - passesgo test ./...passes cleanNote:
go test ./...on this branch alone still hits the pre-existing, unrelatedlastLine()crash from #93 (not yet merged) - confirmed my new test passes in isolation, and passes as part of a clean full run once #93 is applied.🤖 Generated with Claude Code