Background
The AWS SDK v2 transfer manager supports io.ReaderAt for multipart uploads, which allows it to read file parts concurrently at arbitrary offsets without seeking. FileReader already implements ReadAt, but UploadConfig.Reader is typed as io.Reader — so the SDK never sees the capability and the benefit is lost.
The real problem: transfer speeds are measured at the wrong place
The comment in file_reader.go is telling:
// FileReader is a wrapper struct around file read operations,
// allowing us to track rudimentary read speeds from disk
BytesRead() counts bytes read from local disk, not bytes delivered to S3. This means the transfer speed and progress shown to users reflects how fast the file is being read off disk — not how fast it's actually being uploaded over the network. For fast local storage (NVMe, RAID), disk reads will complete far ahead of the network, making reported speeds and progress inaccurate.
The AWS SDK v2 transfer manager's progress listener support (added in #3041 and #3083) provides a way to track bytes as they are sent over the network, which is what users actually care about.
Proposed changes
- Change
UploadConfig.Reader from io.Reader to io.ReaderAt so the SDK manager can use concurrent part reads
- Hook into the SDK transfer manager's progress listener to track bytes delivered to S3 rather than bytes read from disk
- Update
BytesRead() / progress reporting to reflect actual network throughput
Why this matters for users
- Transfer speed shown in the GUI and CLI will reflect actual upload speed, not disk read speed
- Large file transfers on fast local storage will show correct ETAs
Relevant files
src/cli/core/transfer-api/upload.go — UploadConfig.Reader field
src/cli/core/transfer-api/file_reader.go — BytesRead() currently measures disk reads
src/cli/core/job_manager/transfer-worker.go — consumes BytesRead() for progress updates
Background
The AWS SDK v2 transfer manager supports
io.ReaderAtfor multipart uploads, which allows it to read file parts concurrently at arbitrary offsets without seeking.FileReaderalready implementsReadAt, butUploadConfig.Readeris typed asio.Reader— so the SDK never sees the capability and the benefit is lost.The real problem: transfer speeds are measured at the wrong place
The comment in
file_reader.gois telling:BytesRead()counts bytes read from local disk, not bytes delivered to S3. This means the transfer speed and progress shown to users reflects how fast the file is being read off disk — not how fast it's actually being uploaded over the network. For fast local storage (NVMe, RAID), disk reads will complete far ahead of the network, making reported speeds and progress inaccurate.The AWS SDK v2 transfer manager's progress listener support (added in #3041 and #3083) provides a way to track bytes as they are sent over the network, which is what users actually care about.
Proposed changes
UploadConfig.Readerfromio.Readertoio.ReaderAtso the SDK manager can use concurrent part readsBytesRead()/ progress reporting to reflect actual network throughputWhy this matters for users
Relevant files
src/cli/core/transfer-api/upload.go—UploadConfig.Readerfieldsrc/cli/core/transfer-api/file_reader.go—BytesRead()currently measures disk readssrc/cli/core/job_manager/transfer-worker.go— consumesBytesRead()for progress updates