Background
File Mover Express supports pausing and resuming transfer jobs today. When a job is paused, tasks that haven't started are re-queued and will pick up on resume. However, any S3 object that was actively being uploaded at the time of the pause is discarded, its progress is lost and it starts over from the beginning when the job resumes.
This is particularly noticeable with large files. A 50GB file that was 90% uploaded when paused will restart from 0% on resume.
Current behavior
When a job is paused:
- The upload context is cancelled, stopping the active upload
- The task's BytesTransferred is reset to 0
- On resume, the task is re-queued as if it had never started
- The incomplete multipart upload is left orphaned in S3 (no AbortMultipartUpload is called)
- A brand new multipart upload is started from scratch
Desired behavior
When a job is paused and then resumed, an in-progress S3 object upload should resume from where it left off rather than restarting. If resuming from the exact byte offset is not feasible, it should at minimum resume from the last completed multipart part boundary.
What needs to happen
The core challenge is that the AWS SDK v2 manager.Uploader handles multipart uploads internally and does not expose the upload ID or completed parts. Implementing object-level resume will require:
- Tracking the multipart upload ID and completed parts per task — the Task struct currently has no fields for this
- On pause, calling AbortMultipartUpload OR storing the upload ID and completed parts so the upload can be continued later
- On resume, either continuing the existing multipart upload using UploadPart directly (bypassing the SDK manager), or at minimum aborting cleanly and restarting
- Handling the case where the multipart upload has expired in S3 (uploads expire after a configurable period)
Relevant files
pause-job.go - sets job/task status and cancels context
resume-job.go -re-queues paused tasks
transfer-worker.go - handles ErrJobPaused, resets BytesTransferred
upload.go - wraps manager.Uploader
task.go - Task struct, would need new fields for upload ID / part tracking
Notes
-
Incomplete multipart uploads currently accumulate in S3. Until this is fixed, users should configure an S3 lifecycle rule to abort incomplete multipart uploads to avoid unnecessary storage costs. See AWS docs.
-
The rename operation in rename-objects.go already has a working example of manual multipart upload handling with AbortMultipartUpload — this could serve as a reference.
Background
File Mover Express supports pausing and resuming transfer jobs today. When a job is paused, tasks that haven't started are re-queued and will pick up on resume. However, any S3 object that was actively being uploaded at the time of the pause is discarded, its progress is lost and it starts over from the beginning when the job resumes.
This is particularly noticeable with large files. A 50GB file that was 90% uploaded when paused will restart from 0% on resume.
Current behavior
When a job is paused:
Desired behavior
When a job is paused and then resumed, an in-progress S3 object upload should resume from where it left off rather than restarting. If resuming from the exact byte offset is not feasible, it should at minimum resume from the last completed multipart part boundary.
What needs to happen
The core challenge is that the AWS SDK v2 manager.Uploader handles multipart uploads internally and does not expose the upload ID or completed parts. Implementing object-level resume will require:
Relevant files
pause-job.go - sets job/task status and cancels context
resume-job.go -re-queues paused tasks
transfer-worker.go - handles ErrJobPaused, resets BytesTransferred
upload.go - wraps manager.Uploader
task.go - Task struct, would need new fields for upload ID / part tracking
Notes
Incomplete multipart uploads currently accumulate in S3. Until this is fixed, users should configure an S3 lifecycle rule to abort incomplete multipart uploads to avoid unnecessary storage costs. See AWS docs.
The rename operation in rename-objects.go already has a working example of manual multipart upload handling with AbortMultipartUpload — this could serve as a reference.