Skip to content

Keep the upload destination file open across chunks - #94

Merged
eliasbakken merged 2 commits into
mainfrom
fix/upload-chunk-file-handle
Aug 10, 2026
Merged

Keep the upload destination file open across chunks#94
eliasbakken merged 2 commits into
mainfrom
fix/upload-chunk-file-handle

Conversation

@eliasbakken

Copy link
Copy Markdown
Contributor

Problem

One piece of the investigation into #75/#61 (slow/stalling WiFi uploads). uploadChunk() was opening, writing, and closing the destination file on every single chunk:

f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0644)
f.Write(decoded)
f.Close()

For a ~1.2GB image at 3MB chunks, that's ~400 chunks each paying open/close syscall overhead plus whatever flush happens on close - especially costly against the FAT-formatted USB target this writes to.

uploadMagicChunk already gets this right - it opens the file handle once (lazily, on the first chunk) and reuses it for the rest.

Fix

Brings the regular upload path in line with the magic-upload path: uploadStart opens the file once and stores the handle on state.File, uploadChunk just writes to it, uploadFinish closes it (uploadCancel already did this correctly).

Also fixes a real pre-existing bug found along the way: the base64 decode error in uploadChunk was captured but never actually checked.

Not included: the client-side serialized, non-pipelined chunk loop and the base64/JSON wire overhead (~33% inflation) are separate, larger changes - this PR is just the server-side per-chunk overhead.

Test plan

  • go build passes
  • New test TestUploadChunkRoundTrip: drives upload_start -> ~22 real upload_chunk calls -> upload_finish through the actual handlers (not mocked), verifies the resulting file matches the original payload byte-for-byte - passes
  • Verified locally with Fix lastLine() killing the whole process via log.Fatal #93's fix cherry-picked in: full go test ./... passes clean

Note: go test ./... on this branch alone still hits the pre-existing, unrelated lastLine() crash from #93 (not yet merged) - confirmed my new test passes in isolation, and passes as part of a clean full run once #93 is applied.

🤖 Generated with Claude Code

eliasbakken and others added 2 commits August 9, 2026 21:42
uploadChunk() was opening, writing, and closing the destination file
on every single chunk - real per-chunk overhead (open/close syscalls,
plus whatever flush happens on close), especially costly against the
FAT-formatted USB target this writes to. For a ~1.2GB image at 3MB
chunks, that's ~400 chunks each paying this cost.

uploadMagicChunk already gets this right - it opens the file handle
once (lazily, on the first chunk) and reuses it for the rest. This
brings the regular upload path in line with that: uploadStart opens
the file once and stores the handle on state.File, uploadChunk just
writes to it, and uploadFinish (uploadCancel already did) closes it.

Also fixes a pre-existing bug found along the way: the base64 decode
error in uploadChunk was captured but never actually checked.

One piece of a larger investigation into #75/#61 (slow/stalling
uploads) - this addresses the server-side per-chunk overhead
specifically; the client-side serialized, non-pipelined chunk loop
and the base64/JSON wire overhead are separate, larger changes not
included here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drives upload_start -> ~22 upload_chunk calls -> upload_finish through
the real handlers (not mocked), and checks the resulting file on disk
matches the original payload byte-for-byte. Exercises the state.File
handle kept open across chunks, so this would actually catch
truncation, overwriting, or interleaving bugs - a build-only check
can't.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@eliasbakken
eliasbakken merged commit 64f623c into main Aug 10, 2026
2 checks passed
@eliasbakken
eliasbakken deleted the fix/upload-chunk-file-handle branch August 10, 2026 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant