Skip to content

[Feature] Use io.ReaderAt interface for S3 uploads to improve transfer performance accuracy #4

Description

@therangerangler

Background

The AWS SDK v2 transfer manager supports io.ReaderAt for multipart uploads, which allows it to read file parts concurrently at arbitrary offsets without seeking. FileReader already implements ReadAt, but UploadConfig.Reader is typed as io.Reader — so the SDK never sees the capability and the benefit is lost.

The real problem: transfer speeds are measured at the wrong place

The comment in file_reader.go is telling:

// FileReader is a wrapper struct around file read operations,
// allowing us to track rudimentary read speeds from disk

BytesRead() counts bytes read from local disk, not bytes delivered to S3. This means the transfer speed and progress shown to users reflects how fast the file is being read off disk — not how fast it's actually being uploaded over the network. For fast local storage (NVMe, RAID), disk reads will complete far ahead of the network, making reported speeds and progress inaccurate.

The AWS SDK v2 transfer manager's progress listener support (added in #3041 and #3083) provides a way to track bytes as they are sent over the network, which is what users actually care about.

Proposed changes

  1. Change UploadConfig.Reader from io.Reader to io.ReaderAt so the SDK manager can use concurrent part reads
  2. Hook into the SDK transfer manager's progress listener to track bytes delivered to S3 rather than bytes read from disk
  3. Update BytesRead() / progress reporting to reflect actual network throughput

Why this matters for users

  • Transfer speed shown in the GUI and CLI will reflect actual upload speed, not disk read speed
  • Large file transfers on fast local storage will show correct ETAs

Relevant files

  • src/cli/core/transfer-api/upload.goUploadConfig.Reader field
  • src/cli/core/transfer-api/file_reader.goBytesRead() currently measures disk reads
  • src/cli/core/job_manager/transfer-worker.go — consumes BytesRead() for progress updates

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions