Skip to content

feat: end a clip where both of its tracks still have content - #182

Merged
hm21 merged 4 commits into
stablefrom
feat/trim-to-common-track-end
Jul 27, 2026
Merged

feat: end a clip where both of its tracks still have content#182
hm21 merged 4 commits into
stablefrom
feat/trim-to-common-track-end

Conversation

@hm21

@hm21 hm21 commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Description

VideoSequenceBuilder clamps a clip's range to the video track's actual range — there is an explicit comment there about why that clamp is needed — and then reuses that same range for the audio insert without intersecting the audio track. When the source audio is shorter, insertTimeRange silently inserts what exists and the export ends on a stretch of missing audio. A looping player replays that stretch every single cycle as a seam.

A source asset's two tracks end apart routinely: capture and export stop the audio and video writers independently, so each track ends on its own grid.

Measurement

ffprobe over 622 published mp4s, audio track length minus video track length:

Apple-produced (Core Media, mp42/1) Android-produced (isom/131072)
exactly 0 137/362 (38 %) 6/259 (2 %)
|Δ| ≥ 40 ms 62 (17 %) 49 (19 %)
worst case −379 ms −121 ms
audio short vs. long 192 / 33 117 / 136

The two distributions differ in kind. Android is bounded and sign-balanced — packaging, mostly within one AAC access unit. Apple is bimodal (either exact or notably off) with a strong "audio short" skew, which is the signature of the missing clamp rather than of quantisation.

The change

VideoRenderData.trimToCommonTrackEnd (default false) intersects the clip range with the audio track as well, so both tracks carry content for the whole export. One value changes — clipTimeRange — and everything downstream in the clip already derives from it: the video insert, the audio insert (the same sourceRanges), clipDurationeffectiveDurationtotalDuration, the ClipInstruction, and the reverse path's AudioReverser bounds.

For a clip with 6.300 s of video and 6.213 s of audio, the export goes from video 6.300 / audio 6.213 to both at ~6.213. What is dropped is the video that never had audio under it.

It applies per clip, so in a multi-clip composition it also removes the silent hole at a clip boundary, not only the one at the end.

The bitrate-cap passthrough export needed the same treatment separately: it skips the composition entirely, so the flag would have been a silent no-op for exactly the single untrimmed clip that fast path targets — and since every qualityConfig caller sets a bitrate implicitly, that is the common case rather than a corner. passthroughExport now sets export.timeRange to the common track range instead of being disqualified from the fast path, so the export stays lossless. The trim arithmetic lives in TrackEndTrimmer, shared by both paths.

Why it is opt-in, and the 500 ms bound

Trimming shortens the export by the mismatch, which is wrong for a clip that is meant to outlast its own audio — stop motion held past a short sound, for instance. Hence the default false.

maxTrackEndMismatch (500 ms) is the second guard: a larger shortfall is treated as content rather than a track-end mismatch and is logged instead of trimmed. Without it, a 6 s stop-motion clip carrying 2 s of sound would export as 2 s. The worst real artifact in the survey was 379 ms; the one pathological file found (6.37 s of video carrying 9 frames, 0.37 s of audio) falls outside the bound and is left alone.

Related Issue: no issue in this repo — driven by divinevideo/divine-mobile#6386 (player-side counterpart in divinevideo/divine-mobile#6430).

Known gaps

  • LayeredCompositionBuilder ignores the flag. It is threaded through applyCompositionCompositionBuilderVideoSequenceBuilder and through the passthrough fast path only. A render that passes a composition: config takes the layered path and the flag does nothing. Named in the field's doc and in the changelog rather than wired, since the layered path builds its own timeline.
  • Apple only. Android's residual is dominated by AAC packaging (~1 access unit) rather than by a comparable defect, so there is no equivalent clamp to add on that side. Media3 Transformer decides the per-track output lengths there.
  • It removes video, it does not create audio. If capture stopped 87 ms early, that sound does not exist. This drops the picture above it, which is the only thing that leaves no silence.
  • The residual is not zero. The export re-encodes; the video end lands on a frame boundary and the audio end on an AAC access unit. At 44.1 kHz / 30 fps those two grids do not coincide until 17.07 s, so a 6 s clip has no exact landing point. What is left is ≤ ~23 ms of packaging instead of tens to hundreds of ms of missing content.

Verification

  • flutter analyze lib example/lib — clean
  • flutter test — 272 passed, incl. 6 new cases for the flag (default, toMap, toAsyncMap, roundtrip, missing-key default, copyWith). toAsyncMap is the one that actually reaches the native side.
  • Mockito mocks regenerated (copyWith gained a parameter)
  • swiftc -parse on every changed Swift file — clean
  • Passthrough path measured on device: 10.000 s video / 9.880 s audio source → both tracks end at 9.880 s, gap 0.0, video codec unchanged (still lossless).

The composition path is not device-verified yet — that is what keeps this a draft. What is left to check there: an export whose source audio is ~50–100 ms short ends on both tracks together; a stop-motion clip held past a short sound is untouched and logs the skip; an editor-added music track is unaffected, since AudioSequenceBuilder already clamps its pre-render with CMTimeMinimum(prerender.duration, effectivePlayDuration).

Type of Change

  • ✨ New feature (non-breaking change which adds functionality)
  • 🛠️ Bug fix (non-breaking change which fixes an issue)
  • ❌ Breaking change (fix or feature that would cause existing functionality to change)
  • 🧹 Code refactor
  • ✅ Build configuration change
  • 📝 Documentation
  • 🗑️ Chore

A source asset's audio and video tracks routinely end tens of
milliseconds apart, because capture and export stop them independently.
VideoSequenceBuilder clamped the clip range to the video track's range —
with an explicit comment about why that clamp is needed — and then
reused that same range for the audio insert without intersecting the
audio track. When the source audio is shorter, insertTimeRange silently
inserts what exists and the export ends on a stretch of missing audio.
A looping player replays that stretch every cycle as a seam.

Measured across 622 published mp4s: 17% of the Apple-produced files were
off by >=40ms, the worst by 379ms, and 192 of 225 non-zero deltas had
the audio short rather than long — the signature of the missing clamp
rather than of packaging.

The new VideoRenderData.trimToCommonTrackEnd intersects the clip range
with the audio track too, so both tracks carry content for the whole
export. It stays off by default because it shortens the export by the
mismatch, which is wrong for a clip meant to outlast its own audio.
A shortfall beyond 500ms is treated as content rather than a track-end
mismatch and left alone, so a stop-motion clip held past a short sound
cannot be swallowed; the worst real artifact measured was 379ms.
@hm21 hm21 self-assigned this Jul 27, 2026
The bitrate-cap passthrough export skips the composition entirely, so
`isPassthroughEligible` letting it through made the flag a silent no-op
for exactly the single untrimmed clip it targets — and every
`qualityConfig` caller sets a bitrate implicitly, so this was the common
case rather than a corner.

`passthroughExport` now sets `export.timeRange` to the common track range
instead of disqualifying the fast path, so the export stays lossless.
Measured on a 10.000s video / 9.880s audio source: both tracks end at
9.880s, gap 0.0, video codec unchanged.

The trim arithmetic moves to `TrackEndTrimmer` so both paths share one
implementation, and records why the two-sided `CMTimeRangeGetIntersection`
is safe here: AVFoundation folds a leading empty edit into the track's
duration rather than its start, so the clip keeps its head.

Also names the layered-path gap in the public doc instead of only the
platform caveat, asserts the flag on `toAsyncMap` (the map that actually
reaches the native side, where the previous tests only covered `toMap`),
and adds an Apple-guarded example entry.
@hm21
hm21 marked this pull request as ready for review July 27, 2026 14:22
@hm21
hm21 merged commit 6d8cad2 into stable Jul 27, 2026
1 check passed
@hm21
hm21 deleted the feat/trim-to-common-track-end branch July 27, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant