feat: end a clip where both of its tracks still have content - #182
Merged
Conversation
A source asset's audio and video tracks routinely end tens of milliseconds apart, because capture and export stop them independently. VideoSequenceBuilder clamped the clip range to the video track's range — with an explicit comment about why that clamp is needed — and then reused that same range for the audio insert without intersecting the audio track. When the source audio is shorter, insertTimeRange silently inserts what exists and the export ends on a stretch of missing audio. A looping player replays that stretch every cycle as a seam. Measured across 622 published mp4s: 17% of the Apple-produced files were off by >=40ms, the worst by 379ms, and 192 of 225 non-zero deltas had the audio short rather than long — the signature of the missing clamp rather than of packaging. The new VideoRenderData.trimToCommonTrackEnd intersects the clip range with the audio track too, so both tracks carry content for the whole export. It stays off by default because it shortens the export by the mismatch, which is wrong for a clip meant to outlast its own audio. A shortfall beyond 500ms is treated as content rather than a track-end mismatch and left alone, so a stop-motion clip held past a short sound cannot be swallowed; the worst real artifact measured was 379ms.
The bitrate-cap passthrough export skips the composition entirely, so `isPassthroughEligible` letting it through made the flag a silent no-op for exactly the single untrimmed clip it targets — and every `qualityConfig` caller sets a bitrate implicitly, so this was the common case rather than a corner. `passthroughExport` now sets `export.timeRange` to the common track range instead of disqualifying the fast path, so the export stays lossless. Measured on a 10.000s video / 9.880s audio source: both tracks end at 9.880s, gap 0.0, video codec unchanged. The trim arithmetic moves to `TrackEndTrimmer` so both paths share one implementation, and records why the two-sided `CMTimeRangeGetIntersection` is safe here: AVFoundation folds a leading empty edit into the track's duration rather than its start, so the clip keeps its head. Also names the layered-path gap in the public doc instead of only the platform caveat, asserts the flag on `toAsyncMap` (the map that actually reaches the native side, where the previous tests only covered `toMap`), and adds an Apple-guarded example entry.
7 tasks
hm21
marked this pull request as ready for review
July 27, 2026 14:22
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
VideoSequenceBuilderclamps a clip's range to the video track's actual range — there is an explicit comment there about why that clamp is needed — and then reuses that same range for the audio insert without intersecting the audio track. When the source audio is shorter,insertTimeRangesilently inserts what exists and the export ends on a stretch of missing audio. A looping player replays that stretch every single cycle as a seam.A source asset's two tracks end apart routinely: capture and export stop the audio and video writers independently, so each track ends on its own grid.
Measurement
ffprobeover 622 published mp4s, audio track length minus video track length:Core Media,mp42/1)isom/131072)The two distributions differ in kind. Android is bounded and sign-balanced — packaging, mostly within one AAC access unit. Apple is bimodal (either exact or notably off) with a strong "audio short" skew, which is the signature of the missing clamp rather than of quantisation.
The change
VideoRenderData.trimToCommonTrackEnd(defaultfalse) intersects the clip range with the audio track as well, so both tracks carry content for the whole export. One value changes —clipTimeRange— and everything downstream in the clip already derives from it: the video insert, the audio insert (the samesourceRanges),clipDuration→effectiveDuration→totalDuration, theClipInstruction, and the reverse path'sAudioReverserbounds.For a clip with 6.300 s of video and 6.213 s of audio, the export goes from
video 6.300 / audio 6.213to both at ~6.213. What is dropped is the video that never had audio under it.It applies per clip, so in a multi-clip composition it also removes the silent hole at a clip boundary, not only the one at the end.
The bitrate-cap passthrough export needed the same treatment separately: it skips the composition entirely, so the flag would have been a silent no-op for exactly the single untrimmed clip that fast path targets — and since every
qualityConfigcaller sets a bitrate implicitly, that is the common case rather than a corner.passthroughExportnow setsexport.timeRangeto the common track range instead of being disqualified from the fast path, so the export stays lossless. The trim arithmetic lives inTrackEndTrimmer, shared by both paths.Why it is opt-in, and the 500 ms bound
Trimming shortens the export by the mismatch, which is wrong for a clip that is meant to outlast its own audio — stop motion held past a short sound, for instance. Hence the default
false.maxTrackEndMismatch(500 ms) is the second guard: a larger shortfall is treated as content rather than a track-end mismatch and is logged instead of trimmed. Without it, a 6 s stop-motion clip carrying 2 s of sound would export as 2 s. The worst real artifact in the survey was 379 ms; the one pathological file found (6.37 s of video carrying 9 frames, 0.37 s of audio) falls outside the bound and is left alone.Related Issue: no issue in this repo — driven by divinevideo/divine-mobile#6386 (player-side counterpart in divinevideo/divine-mobile#6430).
Known gaps
LayeredCompositionBuilderignores the flag. It is threaded throughapplyComposition→CompositionBuilder→VideoSequenceBuilderand through the passthrough fast path only. A render that passes acomposition:config takes the layered path and the flag does nothing. Named in the field's doc and in the changelog rather than wired, since the layered path builds its own timeline.Media3 Transformerdecides the per-track output lengths there.Verification
flutter analyze lib example/lib— cleanflutter test— 272 passed, incl. 6 new cases for the flag (default,toMap,toAsyncMap, roundtrip, missing-key default,copyWith).toAsyncMapis the one that actually reaches the native side.copyWithgained a parameter)swiftc -parseon every changed Swift file — cleanThe composition path is not device-verified yet — that is what keeps this a draft. What is left to check there: an export whose source audio is ~50–100 ms short ends on both tracks together; a stop-motion clip held past a short sound is untouched and logs the skip; an editor-added music track is unaffected, since
AudioSequenceBuilderalready clamps its pre-render withCMTimeMinimum(prerender.duration, effectivePlayDuration).Type of Change