Add a data profile for generic large-data transfers

CHANGELOG.md

@@ -8,6 +8,20 @@ project uses [semantic versioning][semver].
 
 ## [Unreleased]
 
+### Added
+
+- **A `data` profile for generic large-data transfers.** The verified copy
+  engine was never camera-specific — it reads every byte once, checksums it,
+  fans it out to N destinations and reads it back — but the metadata layer
+  assumed camera originals. `--profile data` (or the shorthand `--generic`)
+  turns that layer off: no ffprobe, no thumbnails, no BRAW check, so a dataset,
+  disk image, render output or backup is copied, checksummed, verified and
+  documented (CSV, MHL, ASC MHL, PDF, HTML) with nothing depending on ffmpeg.
+  The default stays `media`, so the camera-card workflow is unchanged. The
+  profile is a first-class field on `OffloadOptions`, `Job` and saved presets,
+  and is selectable in the desktop app's Simple mode and preset editor. This is
+  a one-way verified transfer, not two-way sync — see `ROADMAP.md`.
+
 ### Fixed
 
 Found by adding CI on Linux and macOS — the suite had only ever run on Windows.

README.md

@@ -4,13 +4,18 @@
 [![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
 [![Licence](https://img.shields.io/badge/licence-MIT-green)](LICENSE)
 
-Verified media offload for camera cards, with job reports that match the layout
-of [ShotPut Pro][spp]'s `JobReport.pdf`.
-
-Copy a card to one or more destinations, checksum every byte, and produce the
-paperwork a post house expects: a PDF contact sheet with per-clip metadata, a
-CSV manifest, MHL and ASC MHL manifests for re-verification downstream, and a
-self-contained HTML page.
+Verified copy for large data transfers. Read every byte once, checksum it, fan
+it out to one or more destinations, and read it back to prove what landed —
+then produce the paperwork: a CSV manifest, MHL and ASC MHL manifests for
+re-verification downstream, and a self-contained HTML page.
+
+The flagship use is camera-card offload, with job reports that match the layout
+of [ShotPut Pro][spp]'s `JobReport.pdf` — a PDF contact sheet with per-clip
+metadata, ffprobe media details and Blackmagic RAW container checks. But that
+media layer is a profile, not the engine: `--profile data` (shorthand
+`--generic`) offloads any large one-way transfer — datasets, disk images,
+render output, backups — with the same verified copy and manifests, and nothing
+depending on ffmpeg. See [Generic data transfers](#generic-data-transfers).
 
 [spp]: https://www.imagineproducts.com/product/shotput-pro/windows
 
@@ -86,6 +91,8 @@ offloader verify D:\video\080426\A001
 | `--dest PATH` | destination root; repeat for multiple copies |
 | `--hash ALGO` | `xxh3-64` (default), `xxh3-128`, `xxh64`, `xxh64be`, `md5`, `sha1`, `sha256`, `c4`, `none` |
 | `--verify MODE` | `source-only` (default), `full`, `none` |
+| `--profile P` | `media` (default: ffprobe, thumbnails, BRAW) or `data` (generic transfer, no media probing) |
+| `--generic` | shorthand for `--profile data` |
 | `--report FMT[,FMT]` | `pdf` (default), `csv`, `mhl`, `ascmhl`, `html` |
 | `--report-dir PATH` | override the report location |
 | `--thumbs N` | frames per clip, 0 to disable (default 4) |
@@ -143,6 +150,39 @@ the destination, at the cost of reading everything twice.
 - **HTML** — self-contained; thumbnails inlined as data URIs, light and dark
   themes, no external requests.
 
+## Generic data transfers
+
+The copy engine has never been camera-specific: it streams the source once,
+checksums it, writes N destinations in the same pass, evicts the page cache and
+reads each copy back off the platter. Everything that made this a *camera* tool
+— ffprobe metadata, contact-sheet thumbnails, the BRAW container check — sits in
+a layer above it.
+
+`--profile data` (or `--generic`) switches that layer off:
+
+```sh
+offloader offload \
+  --source /mnt/instrument/run_1440 \
+  --dest /archive/2026/run_1440 \
+  --dest /nas/cold/run_1440 \
+  --generic \
+  --verify full \
+  --hash sha256 \
+  --report csv,ascmhl
+```
+
+Nothing is treated as a clip, ffmpeg is never invoked, and the run does not need
+it installed. What you still get is the whole point of the tool: every byte
+read once and fanned out, both copies verified off disk, a checksum manifest
+beside each one, and `offloader verify` to re-check the archive months later for
+bit rot. The PDF, CSV, MHL, ASC MHL and HTML reports all render a plain file
+listing — the per-clip metadata block simply does not appear.
+
+This is a **one-way, write-once** transfer: the same model the tool has always
+assumed, now stated for any large data rather than only camera originals. It is
+deliberately not a sync tool — no two-way reconciliation, conflict resolution or
+partial-file updates. See [`ROADMAP.md`](ROADMAP.md).
+
 ## The desktop app
 
 ```sh
@@ -276,13 +316,13 @@ what makes the report layer testable without moving bytes.
 
 ```sh
 pip install -e ".[dev]"
-pytest                      # 400 tests, ~33s
+pytest                      # 409 tests, ~33s
 pytest --fuzz               # same suite, 3000 examples per property (~2 min)
 ruff check src tests
 pytest --cov=offloader --cov-report=term-missing
 ```
 
-400 tests at 83% line coverage. They cover formatting against the reference's
+409 tests at 82% line coverage. They cover formatting against the reference's
 exact strings, checksum vectors and streaming equivalence, copy/verify
 behaviour including simulated destination corruption, pause/resume/cancel
 concurrency, retry discrimination, BRAW container parsing, ffprobe parsing,

ROADMAP.md

@@ -105,9 +105,13 @@ already does the stem-matching this needs.
 
 ## Not planned
 
-- **A general-purpose file sync tool.** The design assumes camera originals:
-  written once, never modified, verified once and archived. Two-way sync,
-  conflict resolution and partial-file updates would compromise that.
+- **A general-purpose file *sync* tool.** The `data` profile generalises the
+  engine to any large *one-way* transfer — written once, never modified,
+  verified once, archived — because that assumption is exactly what makes the
+  "Verified" verdict meaningful, and it holds for a dataset or a disk image as
+  well as a camera card. What stays out is everything that *breaks* it:
+  two-way reconciliation, conflict resolution and partial-file updates. Those
+  turn a copy you can prove into a merge you have to trust.
 - **Metadata fidelity beyond timestamps** — ACLs, alternate data streams,
   junctions. robocopy does these well and they do not apply to camera media.
 - **Replacing the copy loop with robocopy.** Measured and rejected; see

pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "offloader"
 version = "0.1.0"
-description = "Verified media offload for camera cards, with ShotPut Pro-compatible reports, ASC MHL, and Blackmagic RAW support"
+description = "Verified copy for large data transfers, with camera-card offload, ShotPut Pro-compatible reports, ASC MHL, and Blackmagic RAW support"
 readme = "README.md"
 requires-python = ">=3.10"
 license = { text = "MIT" }

src/offloader/__init__.py

@@ -1,4 +1,5 @@
-"""Verified media offload with ShotPut Pro-compatible reporting."""
+"""Verified copy for large data transfers, with camera-card offload and
+ShotPut Pro-compatible reporting built in."""
 
 __version__ = "0.1.0"
 

src/offloader/cli.py

@@ -8,7 +8,7 @@
 from pathlib import Path
 
 from . import PRODUCT_NAME, __version__, engine, hashers, longpath, probe, retry, thumbs
-from .models import FileStatus, Job, VerificationMode
+from .models import FileStatus, Job, Profile, VerificationMode
 from .reports import WRITERS
 from .util import format_elapsed, format_size
 
@@ -122,9 +122,12 @@ def _summarize(job: Job, reports: list[Path]) -> None:
     failed = [f for f in job.files if f.status is FileStatus.FAILED]
     print()
     print(f"  {job.name}: {job.final_status}")
+    # The video count is meaningful only when media was probed; a generic
+    # data transfer never looks inside a file, so reporting "0 video" would
+    # be noise rather than information.
+    video = f"  ({job.video_files} video)" if job.profile.probes_media else ""
     print(f"  {job.total_files} files, {format_size(job.total_bytes)}"
-          f" in {format_elapsed(job.elapsed_sec)}"
-          f"  ({job.video_files} video)")
+          f" in {format_elapsed(job.elapsed_sec)}{video}")
     print(f"  Verification: {job.verification_label}")
     for destination in job.destination_roots:
         print(f"  -> {destination}")
@@ -169,6 +172,14 @@ def _common_options(parser: argparse.ArgumentParser) -> None:
     parser.add_argument("--retry-wait", type=float, default=2.0, metavar="SECONDS",
                         help="pause before the first retry, backing off after "
                              "(default: %(default)s)")
+    parser.add_argument("--profile", choices=[p.value for p in Profile],
+                        default=Profile.MEDIA.value,
+                        help="'media' (default) offloads camera cards with "
+                             "ffprobe metadata, thumbnails and the BRAW check; "
+                             "'data' is a generic large-data transfer that "
+                             "copies and verifies but skips all media probing")
+    parser.add_argument("--generic", action="store_true",
+                        help="shorthand for --profile data")
     parser.add_argument("--no-probe", action="store_true",
                         help="skip ffprobe metadata and thumbnails")
     parser.add_argument("--quiet", action="store_true", help="suppress progress")
@@ -177,7 +188,8 @@ def _common_options(parser: argparse.ArgumentParser) -> None:
 def build_parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(
         prog="offloader",
-        description=f"{PRODUCT_NAME} — verified media offload with job reports.",
+        description=f"{PRODUCT_NAME} — verified copy for large data transfers, "
+                    f"with camera-card offload and job reports built in.",
     )
     parser.add_argument("--version", action="version",
                         version=f"{PRODUCT_NAME} {__version__}")
@@ -226,6 +238,7 @@ def build_parser() -> argparse.ArgumentParser:
 
 
 def _options_from(args: argparse.Namespace, destinations: list[Path]) -> engine.OffloadOptions:
+    profile = Profile.DATA if getattr(args, "generic", False) else Profile(args.profile)
     return engine.OffloadOptions(
         destinations=destinations,
         algorithm=args.hash,
@@ -236,6 +249,7 @@ def _options_from(args: argparse.Namespace, destinations: list[Path]) -> engine.
         skip_existing=getattr(args, "skip_existing", False),
         job_name=args.name,
         extra_probe=not args.no_probe,
+        profile=profile,
         retry=retry.RetryPolicy(attempts=max(1, args.retries),
                                 delay=max(0.0, args.retry_wait)),
     )
@@ -356,6 +370,8 @@ def cmd_info(_args: argparse.Namespace) -> int:
               f" {prefix} prefix {note}")
     print(f"  checksums:   {', '.join(sorted(hashers.algorithm_keys()))}")
     print(f"  reports:     {', '.join(WRITERS)}")
+    print(f"  profiles:    {', '.join(p.value for p in Profile)} "
+          f"(--profile; 'data' skips media probing for generic transfers)")
     return 0
 
 

src/offloader/engine.py

@@ -27,6 +27,7 @@
     FileEntry,
     FileStatus,
     Job,
+    Profile,
     VerificationMode,
 )
 
@@ -176,10 +177,23 @@ class OffloadOptions:
     job_name: str | None = None
     thumbnail_dir: Path | None = None
     extra_probe: bool = True
+    #: The workflow this offload is. `Profile.DATA` is a generic large-data
+    #: transfer: no file is treated as media, so ffprobe, thumbnails and the
+    #: BRAW check are all switched off regardless of the media-only knobs above.
+    profile: Profile = Profile.MEDIA
     #: How hard to try again when a read fails for a transient-looking reason.
     #: Marginal cards and readers routinely succeed on a second attempt.
     retry: retry_mod.RetryPolicy = field(default_factory=retry_mod.RetryPolicy)
 
+    def __post_init__(self) -> None:
+        # The data profile is defined by the absence of media work, so enforce
+        # it here rather than trusting every caller to zero the media knobs.
+        # A library caller that sets only `profile=Profile.DATA` gets a clean
+        # generic transfer; the CLI and presets get the same guarantee.
+        if self.profile is Profile.DATA:
+            self.extra_probe = False
+            self.thumbnail_count = 0
+
 
 @dataclass
 class _Counters:
@@ -361,6 +375,7 @@ def run(source_root: Path, options: OffloadOptions,
         source_root=source_root,
         destination_roots=dest_roots,
         verification=options.verification,
+        profile=options.profile,
         hash_label=algorithm.label,
         started=_dt.datetime.now(),
         os_version=host.os_version,
@@ -640,6 +655,7 @@ def rescan(source_root: Path, destination_roots: Sequence[Path],
         source_root=source_root,
         destination_roots=[Path(d) for d in destination_roots] or [source_root],
         verification=options.verification,
+        profile=options.profile,
         hash_label=algorithm.label,
         started=_dt.datetime.now(),
         os_version=host.os_version,

src/offloader/gui/preset_editor.py

@@ -21,7 +21,7 @@
 )
 
 from ..hashers import ALGORITHMS
-from ..models import VerificationMode
+from ..models import Profile, VerificationMode
 from ..naming import TOKENS
 from ..presets import PRESET_COLORS, Preset
 from ..reports import WRITERS
@@ -85,11 +85,21 @@ def __init__(self, preset: Preset | None = None, parent: QWidget | None = None)
         self._verification.setCurrentIndex(
             max(0, self._verification.findData(self._source.verification.value)))
 
+        self._profile = QComboBox()
+        self._profile.addItem("Media — camera card (ffprobe, thumbnails, BRAW)",
+                              Profile.MEDIA.value)
+        self._profile.addItem("Data — any large transfer (copy and verify only)",
+                              Profile.DATA.value)
+        self._profile.setCurrentIndex(
+            max(0, self._profile.findData(self._source.profile.value)))
+        self._profile.currentIndexChanged.connect(self._on_profile_changed)
+
         self._thumbnails = QSpinBox()
         self._thumbnails.setRange(0, 8)
         self._thumbnails.setValue(self._source.thumbnail_count)
         self._thumbnails.setSuffix(" per clip")
         self._thumbnails.setSpecialValueText("Off")
+        self._thumbnails.setEnabled(self._source.profile is Profile.MEDIA)
 
         self._reports: dict[str, QCheckBox] = {}
         report_row = []
@@ -130,6 +140,7 @@ def __init__(self, preset: Preset | None = None, parent: QWidget | None = None)
         form.addRow("Colour", self._color)
         form.addRow("Destinations", self._destinations)
         form.addRow("", row(add, remove, None))
+        form.addRow("Profile", self._profile)
         form.addRow("Checksum", self._algorithm)
         form.addRow("Verification", self._verification)
         form.addRow("Thumbnails", self._thumbnails)
@@ -152,6 +163,11 @@ def __init__(self, preset: Preset | None = None, parent: QWidget | None = None)
         layout.addLayout(form)
         layout.addWidget(buttons)
 
+    def _on_profile_changed(self) -> None:
+        # A data transfer never decodes a file, so contact-sheet thumbnails do
+        # not apply — disable the control rather than let it imply otherwise.
+        self._thumbnails.setEnabled(self._profile.currentData() == Profile.MEDIA.value)
+
     def _choose_logo(self) -> None:
         path, _ = QFileDialog.getOpenFileName(
             self, "Choose a logo", "", "Images (*.png *.jpg *.jpeg *.gif)")
@@ -171,6 +187,7 @@ def _accept(self) -> None:
             destinations=self._destinations.paths(),
             algorithm=self._algorithm.currentData(),
             verification=VerificationMode(self._verification.currentData()),
+            profile=Profile(self._profile.currentData()),
             thumbnail_count=self._thumbnails.value(),
             reports=[key for key, box in self._reports.items() if box.isChecked()],
             naming_template=self._naming.text().strip() or "{card}",

src/offloader/gui/simple_mode.py

@@ -20,7 +20,7 @@
 )
 
 from ..hashers import ALGORITHMS
-from ..models import VerificationMode
+from ..models import Profile, VerificationMode
 from ..presets import Preset
 from ..reports import WRITERS
 from .preset_editor import VERIFICATION_LABELS
@@ -60,6 +60,13 @@ def __init__(self, parent: QWidget | None = None) -> None:
         self._verification.setCurrentIndex(
             max(0, self._verification.findData(VerificationMode.SOURCE_ONLY.value)))
 
+        self._profile = QComboBox()
+        self._profile.addItem("Media — camera card", Profile.MEDIA.value)
+        self._profile.addItem("Data — any large transfer", Profile.DATA.value)
+        self._profile.setCurrentIndex(
+            max(0, self._profile.findData(Profile.MEDIA.value)))
+        self._profile.currentIndexChanged.connect(self._on_profile_changed)
+
         self._thumbnails = QSpinBox()
         self._thumbnails.setRange(0, 8)
         self._thumbnails.setValue(4)
@@ -82,6 +89,7 @@ def __init__(self, parent: QWidget | None = None) -> None:
         form.setSpacing(10)
         form.setLabelAlignment(Qt.AlignRight | Qt.AlignVCenter)
         form.addRow("Job name", self._name)
+        form.addRow("Profile", self._profile)
         form.addRow("Checksum", self._algorithm)
         form.addRow("Verification", self._verification)
         form.addRow("Thumbnails", self._thumbnails)
@@ -146,12 +154,20 @@ def _overlaps(source: Path, destination: Path) -> bool:
             return False
         return source == destination or source in destination.parents
 
+    def _on_profile_changed(self) -> None:
+        # Thumbnails are contact-sheet frames from a clip — meaningless for a
+        # generic data transfer, which never decodes a file. Grey the control
+        # so the disabled state explains itself.
+        is_media = self._profile.currentData() == Profile.MEDIA.value
+        self._thumbnails.setEnabled(is_media)
+
     def build_preset(self) -> Preset:
         return Preset(
             name="Simple mode",
             destinations=self.destinations.paths(),
             algorithm=self._algorithm.currentData(),
             verification=VerificationMode(self._verification.currentData()),
+            profile=Profile(self._profile.currentData()),
             thumbnail_count=self._thumbnails.value(),
             reports=[key for key, box in self._reports.items() if box.isChecked()],
             preserve_structure=self._preserve.isChecked(),