Describe the feature
Describe the feature: When a hot folder is added (or re-added after removal), the initial sync uploads every file in the directory to S3 — even if identical files already exist at the destination. This is because hot folder jobs are created with Force: true, which bypasses the object-already-exists filter.
The initial directory scan (HotFolderUploadSourceDirectory) should use Force: false so the existing size + last-modified (or checksum) comparison runs before uploading. The real-time file watcher path (hotFolderUpload) can keep Force: true since those files are known to be new from the filesystem event.
Use case
A DIT sets up a hot folder, syncs 500GB of camera footage to S3, then removes the hot folder to reconfigure it. When they re-add it pointing at the same directory, all 500GB uploads again unnecessarily. This wastes bandwidth, time, and S3 PUT request costs — especially painful on metered or slow connections.
Area: Hot Folders, Transfers / Performance
Area
Hot Folders
Proposed solution
In
hot_folder.go
, change StartHotFolderUpload to accept a force parameter instead of hardcoding Force: true. Then:
HotFolderUploadSourceDirectory (initial scan) calls it with force: false
hotFolderUpload (filesystem event driven) calls it with force: true
Alternatives considered: Persisting hot folder upload state (e.g., a local database of uploaded file hashes) across remove/re-add cycles. This would be more robust but significantly more complex than leveraging the existing S3 HeadObject check.
Alternatives considered
Persisting hot folder upload state (e.g., a local database of uploaded file hashes) across remove/re-add cycles. This would be more robust but significantly more complex than leveraging the existing S3 HeadObject check.
Acknowledgements
Describe the feature
Describe the feature: When a hot folder is added (or re-added after removal), the initial sync uploads every file in the directory to S3 — even if identical files already exist at the destination. This is because hot folder jobs are created with Force: true, which bypasses the object-already-exists filter.
The initial directory scan (HotFolderUploadSourceDirectory) should use Force: false so the existing size + last-modified (or checksum) comparison runs before uploading. The real-time file watcher path (hotFolderUpload) can keep Force: true since those files are known to be new from the filesystem event.
Use case
A DIT sets up a hot folder, syncs 500GB of camera footage to S3, then removes the hot folder to reconfigure it. When they re-add it pointing at the same directory, all 500GB uploads again unnecessarily. This wastes bandwidth, time, and S3 PUT request costs — especially painful on metered or slow connections.
Area: Hot Folders, Transfers / Performance
Area
Hot Folders
Proposed solution
In
hot_folder.go
, change StartHotFolderUpload to accept a force parameter instead of hardcoding Force: true. Then:
HotFolderUploadSourceDirectory (initial scan) calls it with force: false
hotFolderUpload (filesystem event driven) calls it with force: true
Alternatives considered: Persisting hot folder upload state (e.g., a local database of uploaded file hashes) across remove/re-add cycles. This would be more robust but significantly more complex than leveraging the existing S3 HeadObject check.
Alternatives considered
Persisting hot folder upload state (e.g., a local database of uploaded file hashes) across remove/re-add cycles. This would be more robust but significantly more complex than leveraging the existing S3 HeadObject check.
Acknowledgements