Skip to content

Add a data profile for generic large-data transfers - #1

Merged
owenpkent merged 1 commit into
mainfrom
claude/generalize-large-data-transfer-a2llqx
Aug 9, 2026
Merged

Add a data profile for generic large-data transfers#1
owenpkent merged 1 commit into
mainfrom
claude/generalize-large-data-transfer-a2llqx

Conversation

@owenpkent

Copy link
Copy Markdown
Owner

What this changes

Generalises the tool from camera-card offload to any large one-way transfer, without disturbing the camera workflow.

The copy engine was never actually camera-specific — it reads every byte once, checksums it, fans out to N destinations, and reads each copy back off disk. Only the layer above it (ffprobe metadata, thumbnails, the BRAW container check) assumed camera originals. This change makes that layer switchable rather than rewriting the core.

New Profile { media, data }, selected with --profile data (shorthand --generic):

  • data turns the media layer off — no ffprobe, no thumbnails, no BRAW check — so a dataset, disk image, render output or backup is copied, checksummed, verified and documented (CSV / MHL / ASC MHL / PDF / HTML) with nothing depending on ffmpeg.
  • Default stays media, so the camera-card workflow is byte-for-byte unchanged.

Threaded through as a first-class field, not a CLI-only flag:

  • OffloadOptions.__post_init__ enforces the media knobs off for a data transfer, so a library caller that sets only profile=Profile.DATA still gets a clean generic copy.
  • Stamped onto Job; the CLI summary drops the "N video" line under the data profile (where it would only ever read "0 video").
  • Saved presets round-trip the profile through JSON, tolerant of a missing or garbage key; it's selectable in the desktop app's Simple mode and preset editor, where the media-only thumbnails control greys out for a data transfer.

Docs reframe the tool as verified copy for large data transfers with camera-card offload built in, and the ROADMAP.md "not a sync tool" entry now distinguishes the supported one-way verified transfer from the two-way sync (reconciliation, conflict resolution, partial-file updates) that stays out of scope.

What you verified

  • The data profile never probes. A test monkeypatches engine.probe_mod.probe to raise and runs an offload under Profile.DATA; it passes, proving ffprobe is never reached. The same job still copies and verifies (final_status == "Verified", every file VERIFIED), so switching the media layer off does not weaken the copy/verify path.
  • The media profile is unchanged. Profile.MEDIA leaves extra_probe/thumbnail_count exactly as passed; the default CLI path still resolves to Profile.MEDIA and still prints the "N video" count.
  • __post_init__ is the single source of truth. Setting profile=Profile.DATA alongside extra_probe=True, thumbnail_count=4 still yields extra_probe is False and thumbnail_count == 0, so no caller (CLI, preset, or library) can accidentally probe a data transfer.
  • Presets survive round-trips and bad input. to_dict/from_dict preserve the profile; a config with no profile key or a garbage value falls back to media rather than bricking the load.
  • End-to-end smoke test. Ran a real generic offload (--generic --verify full --report csv,mhl) over a 5 MB file plus a subfolder, then offloader verify on the resulting MHL — reports "VERIFIED — safe to erase the source". Confirmed the summary omits the video count for data and keeps it for media.

This does not make a "Verified" verdict easier to reach: the data profile removes only metadata collection, not any read, hash, or read-back step. docs/data-safety.md describes the copy/verify/delete path, which is untouched.

Checklist

  • pytest passes (409 tests, 8 skipped — added tests/test_profile.py)
  • ruff check src tests passes
  • Tests added for the behaviour changed

If this touches the copy, verification, or delete path:

  • N/A — the copy/verify/delete path is unchanged; the profile only gates metadata collection (ffprobe/thumbnails/BRAW), which sits above the engine
  • This does not make a "Verified" verdict easier to reach — no read, hash, or read-back step is altered
  • docs/data-safety.md still describes reality

If this changes documented behaviour:

  • README.md, ROADMAP.md, CHANGELOG.md and pyproject.toml updated
  • The test count in README.md is updated (400 → 409, coverage 83% → 82%)

🤖 Generated with Claude Code


Generated by Claude Code

The copy engine was never camera-specific — it reads every byte once,
checksums it, fans it out to N destinations and reads each copy back off
disk — but the layer above it assumed camera originals: ffprobe metadata,
contact-sheet thumbnails, the BRAW container check. This generalises the
tool to any large one-way transfer without touching that engine.

Add a Profile { media, data } selected with `--profile` (shorthand
`--generic`). The data profile turns the media layer off: no ffprobe, no
thumbnails, no BRAW check, so a dataset, disk image, render output or
backup is copied, verified and documented with nothing depending on
ffmpeg. The default stays `media`, so the camera-card workflow is
unchanged.

Profile is a first-class field on OffloadOptions (whose __post_init__
enforces the media knobs off for a data transfer, so a library caller
that sets only the profile still gets a clean generic copy), on Job, and
on saved presets (round-tripped through JSON, tolerant of a missing or
garbage key). It is selectable in the desktop app's Simple mode and
preset editor, where the media-only thumbnails control greys out for a
data transfer. The CLI summary drops the "N video" line under the data
profile, where it would only ever read "0 video".

Docs reframe the tool as verified copy for large data transfers with
camera-card offload built in, and the ROADMAP's "not a sync tool" entry
now distinguishes the supported one-way transfer from the two-way sync
that stays out of scope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017D2mLdX5ABymqURvDQEJsY

@owenpkent owenpkent left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran the branch's tests in a worktree — 44 passed across test_profile.py, test_presets.py and test_cli.py. This looks good to merge.

Enforcing the profile in OffloadOptions.__post_init__ rather than at the CLI is the right shape: it means the guarantee holds for a library caller and a preset, not just for --generic. The loader tolerating both a missing and a garbage profile key is the right amount of paranoia for something that reads a user-editable JSON file.

Two small things inline, neither blocking.

One cross-PR note: #2 adds companions.group() to engine.run unconditionally, so once both land a --profile data transfer will still label .xmp files and anything under a Proxy/ folder as companions of a "clip". That wants gating on profile.probes_media. The two branches also conflict textually in CHANGELOG.md, README.md (3 hunks), cli.py (_summarize — this PR adds the profile gate, #2 appends the paranoid suffix) and preset_editor.py; engine.py, models.py, presets.py and simple_mode.py auto-merge. Since #2 has a blocker and this doesn't, merging this one first and rebasing #2 seems the cheaper order.

Comment thread src/offloader/cli.py
# The video count is meaningful only when media was probed; a generic
# data transfer never looks inside a file, so reporting "0 video" would
# be noise rather than information.
video = f" ({job.video_files} video)" if job.profile.probes_media else ""

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CLI summary is gated on probes_media, but the reports aren't: pdf.py:232 and html.py:163 both emit job.video_files unconditionally, so a --profile data job still gets a Video Files: 0 row in its header block. Same reasoning as the comment above it applies — it's noise rather than information — and the README says the reports "render a plain file listing".

Comment thread src/offloader/presets.py
return f"{where} · {self.algorithm} · {verify} · {', '.join(self.reports) or 'no reports'}"
parts = [where, self.algorithm, verify, ", ".join(self.reports) or "no reports"]
if self.profile is Profile.DATA:
parts.insert(0, "data")

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inserting at 0 puts the profile ahead of the destination count, so the summary reads data · 2 destinations · sha256 · full verify. Reads a little oddly with the leading bare word — appending it, or prefixing where itself, would scan better.

@owenpkent
owenpkent merged commit 77870e0 into main Aug 9, 2026
9 checks passed
@owenpkent
owenpkent deleted the claude/generalize-large-data-transfer-a2llqx branch August 9, 2026 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants