feat: chroma key (greenscreen) for the render pipeline

CHANGELOG.md

@@ -1,3 +1,10 @@
+## 2.11.0
+- **FEAT**(android, iOS, macOS): Add `ChromaKey` — green-screen removal with a soft edge and spill suppression. The keyed area becomes a `backgroundColor`, a `backgroundImage`, or (in a `VideoComposition`) the layer below. Settable on `VideoRenderData`, `VideoLayer` and `VideoSegment`, resolved per clip as segment → layer → global. H.264/HEVC carry no alpha, so on the single-track `videoSegments` path a key without a background is flattened to black.
+- **FEAT**: `ChromaKey.autoDetect(video)` measures the key color and `similarity` off the footage, for any saturated screen hue. `ChromaKey.detect` returns the raw measurement. Costs one thumbnail decode, no render.
+- **FEAT**: `ChromaKey.greenScreen()` and `ChromaKey.blueScreen()` presets. Blue separates skin better but crowds denim and blue eyes, so it keys tighter and despills more gently.
+- **FIX**(android): A `VideoComposition` layer no longer squares its own alpha. `VideoCompositionTransformation` blended against an already-transparent framebuffer, thinning any partially transparent edge.
+- **FIX**(android, iOS, macOS): Two sources resolved in the same millisecond no longer collide on one temp file in `EditorVideo.safeFilePath`.
+
 ## 2.10.0
 - **FEAT**(iOS, macOS): Add `VideoRenderData.trimToCommonTrackEnd` (default `false`). Ends each clip where both its video and audio track still have content, so an export no longer finishes on a stretch of missing audio — the seam a looping player replays every cycle. Leave it off when a clip is meant to outlast its own audio.
 

README.md

@@ -129,6 +129,7 @@ The ProVideoEditor is a Flutter widget designed for video editing within your ap
 - 🎞️ **Clip Transitions**: Add transitions between adjacent clips — `dissolve`, `fadeToBlack`, `fadeToWhite`, `slide`, `push`, and `wipe` — with configurable duration, easing curve, and direction.
 - 🧮 **Color Matrix**: Apply one or multiple 4x5 color matrices (e.g., for filters).
 - 💧 **Blur**: Add a blur effect to the video.
+- 🟩 **Chroma Key**: Remove a green (or blue, or any saturated hue) screen with a soft edge and spill suppression, and fill it with a color, an image, or — in a `VideoComposition` — the layer below. `ChromaKey.autoDetect` measures the key straight off the footage; `greenScreen()`/`blueScreen()` presets are there when you already know. Configurable globally, per `VideoLayer`, or per `VideoSegment`.
 - 📡 **Bitrate**: Cap the video bitrate. Sources already below the cap are exported losslessly over the fast path; sources above it are re-encoded down to the cap. If constant bitrate (CBR) isn't supported, it will gracefully fall back to the next available mode.
 - 🌐 **Streaming Optimization**: Optimize video for progressive playback by placing metadata (moov atom) at the start of the file.
 
@@ -160,6 +161,7 @@ The ProVideoEditor is a Flutter widget designed for video editing within your ap
 | `Multiple ColorMatrix 4x5` | ✅      | ✅  | ✅     | ❌      | ❌     | 🚫   |
 | `Cancel export task`       | ✅      | ✅  | ✅     | ❌      | ❌     | 🚫   |
 | `Blur background`          | 🧪      | 🧪  | 🧪     | ❌      | ❌     | 🚫   |
+| `Chroma Key (Greenscreen)` | ✅      | ✅  | ✅     | ❌      | ❌     | 🚫   |
 | `Custom Audio Tracks`      | ✅      | ✅  | ✅     | ❌      | ❌     | 🚫   |
 | `Merge Videos`             | ✅      | ✅  | ✅     | ❌      | ❌     | 🚫   |
 | `Stop-Motion (Images→Video)`| ✅     | ✅  | ✅     | ❌      | ❌     | 🚫   |
@@ -620,13 +622,57 @@ var task = VideoRenderData(
          ColorFilter(matrix: [ 1.0, 0.0, 0.0, 0.0, 50.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0 ]),
          ColorFilter(matrix: [ 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0 ]),
     ],
+    chromaKey: const ChromaKey(
+        // Defaults to SMPTE "chroma key green". Brightness matters as well as
+        // hue, so measure it with `ChromaKey.autoDetect` when you can.
+        color: Color(0xFF00B140),
+        similarity: 0.20,  // raise it if the screen survives the key
+        smoothness: 0.08,  // width of the soft edge
+        spill: 0.5,        // pull the green cast off the subject's edges
+        backgroundColor: Color(0xFF000000),
+    ),
 );
 
 Uint8List result = await ProVideoEditor.instance.renderVideo(task);
 
 /// Note: Blur is an experimental feature (🧪 in platform matrix)
 /// The blur effect may render differently than in Flutter's preview.
 
+/// Chroma key: the most reliable key is measured, not guessed. Paint, lighting
+/// and the camera all shift the recorded screen away from any constant, and
+/// that shift has to be paid for with a wider `similarity` — the very margin
+/// that protects the subject. On a real studio clip the SMPTE constant sat
+/// 0.10–0.20 away from the screen and needed `similarity: 0.20`; the measured
+/// color sat within 0.05 and 0.10 was plenty.
+final key = await ChromaKey.autoDetect(
+    greenScreenVideo,
+    backgroundColor: Colors.black,
+);
+
+/// When you already know the screen, the presets carry the measured values.
+/// Note that blue is not green with a different hue: it separates skin better,
+/// but denim (0.19), blue eyes (0.20) and light blue shirts (0.21) all crowd
+/// it, so `blueScreen()` keys tighter and despills more gently.
+const greenKey = ChromaKey.greenScreen(backgroundColor: Color(0xFF000000));
+const blueKey = ChromaKey.blueScreen(backgroundColor: Color(0xFF000000));
+
+/// Chroma key: to put a *video* behind the green screen, leave the key
+/// transparent and place the keyed clip on a layer above another one. In the
+/// single-track `videoSegments` path there is nothing underneath and the codec
+/// carries no alpha, so a key there needs a `backgroundColor` or
+/// `backgroundImage`.
+final keyedTask = VideoRenderData(
+    composition: VideoComposition(
+        layers: [
+            VideoLayer(clips: [VideoSegment(video: backgroundVideo)]),
+            VideoLayer(
+                clips: [VideoSegment(video: greenScreenVideo)],
+                chromaKey: const ChromaKey(),
+            ),
+        ],
+    ),
+);
+
 /// Listen progress
 StreamBuilder<ProgressModel>(
     stream: ProVideoEditor.instance.progressStreamById(task.id),

android/src/main/kotlin/ch/waio/pro_video_editor/src/features/render/RenderVideo.kt

@@ -26,6 +26,7 @@ import ch.waio.pro_video_editor.src.features.render.helpers.VideoReverser
 import ch.waio.pro_video_editor.src.features.render.helpers.ClipTransitionGeometry
 import ch.waio.pro_video_editor.src.features.render.helpers.ClipTransitionRenderer
 import ch.waio.pro_video_editor.src.features.render.helpers.MediaInfoExtractor
+import ch.waio.pro_video_editor.src.features.render.models.ChromaKeyConfig
 import ch.waio.pro_video_editor.src.features.render.models.CodecResourceExhaustedException
 import ch.waio.pro_video_editor.src.features.render.models.RenderConfig
 import ch.waio.pro_video_editor.src.features.render.models.RenderJobHandle
@@ -89,7 +90,17 @@ class RenderVideo(private val context: Context) {
         val hasBlur = config.blur != null && config.blur > 0.0
         val hasColorFilters = config.colorFilters.isNotEmpty()
 
-        return hasImageLayers || hasBlur || hasColorFilters
+        // The chroma key is an ES 2.0 SDR shader, so a 10-bit HDR source has to
+        // be transcoded down first. Checked at every level it can be set:
+        // globally, per clip, and — for the layered path — per layer and per
+        // layer clip.
+        val hasChromaKey = config.chromaKey != null ||
+                config.videoClips.any { it.chromaKey != null } ||
+                config.composition?.layers?.any { layer ->
+                    layer.chromaKey != null || layer.clips.any { it.chromaKey != null }
+                } == true
+
+        return hasImageLayers || hasBlur || hasColorFilters || hasChromaKey
     }
 
     /**
@@ -211,7 +222,16 @@ class RenderVideo(private val context: Context) {
                     // 1) Pre-transcode HEVC 10-bit clips (if needed).
                     if (needsPreTranscode) {
                         Log.d(RENDER_TAG, "Pre-transcoding HEVC 10-bit videos...")
-                        val inputPaths = workingConfig.videoClips.map { it.inputPath }
+                        // Both paths, not just the single-track one: a GPU
+                        // effect on a composition layer needs its clips
+                        // transcoded too. ChromaKeyEffect is an ES 2.0 SDR
+                        // shader and hard-fails on an HDR input, so leaving the
+                        // layered clips untouched here aborted the export.
+                        val inputPaths = (
+                            workingConfig.videoClips.map { it.inputPath } +
+                                (workingConfig.composition?.layers ?: emptyList())
+                                    .flatMap { layer -> layer.clips.map { it.inputPath } }
+                            ).distinct()
                         val transcodeMap = VideoTranscoder.transcodeClipsIfNeeded(
                             context, inputPaths
                         )
@@ -223,13 +243,22 @@ class RenderVideo(private val context: Context) {
                                 "Pre-transcoded ${transcodedFiles.size} HEVC 10-bit videos to H.264"
                             )
                         }
-                        val updatedClips = workingConfig.videoClips.map { clip ->
+                        fun retarget(clip: VideoClip): VideoClip {
                             val newPath = transcodeMap[clip.inputPath] ?: clip.inputPath
-                            if (newPath != clip.inputPath) {
+                            return if (newPath != clip.inputPath) {
                                 clip.copy(inputPath = newPath)
                             } else clip
                         }
-                        workingConfig = workingConfig.copy(videoClips = updatedClips)
+                        workingConfig = workingConfig.copy(
+                            videoClips = workingConfig.videoClips.map(::retarget),
+                            composition = workingConfig.composition?.let { composition ->
+                                composition.copy(
+                                    layers = composition.layers.map { layer ->
+                                        layer.copy(clips = layer.clips.map(::retarget))
+                                    }
+                                )
+                            }
+                        )
                     }
 
                     // 2) Pre-render reversed clips into single forward MP4 temp files.
@@ -262,6 +291,7 @@ class RenderVideo(private val context: Context) {
                         val transitionedClips = preRenderTransitions(
                             workingConfig.videoClips,
                             enableAudio = workingConfig.enableAudio,
+                            globalChromaKey = workingConfig.chromaKey,
                             shouldStop = shouldStopPolling,
                             collectPath = { paths.add(it) },
                             onProgress = { f -> transitionProgress(f.toDouble()) },
@@ -399,6 +429,7 @@ class RenderVideo(private val context: Context) {
     private fun preRenderTransitions(
         clips: List<VideoClip>,
         enableAudio: Boolean,
+        globalChromaKey: ChromaKeyConfig?,
         shouldStop: AtomicBoolean,
         collectPath: (String) -> Unit,
         onProgress: (Float) -> Unit,
@@ -497,11 +528,14 @@ class RenderVideo(private val context: Context) {
                 // Keep the outgoing clip's speed; it now ends `tailSrc` of source
                 // earlier (those frames moved into the speed-adjusted blend).
                 addClip(current.copy(endUs = curEnd - tailSrc, transition = null))
+                val blendKey = blendChromaKey(current, next, globalChromaKey, "$i")
                 result.add(
                     VideoClip(
                         inputPath = rendered.outputPath,
                         startUs = 0L,
                         endUs = rendered.durationUs.takeIf { it > 0 },
+                        chromaKey = blendKey.config,
+                        suppressChromaKey = blendKey.suppressed,
                     )
                 )
                 // Trim the incoming head in place; it keeps its own speed/transition.
@@ -587,11 +621,15 @@ class RenderVideo(private val context: Context) {
                         result[0] = first.copy(startUs = firstStart + headSrc)
                         result[lastIdx] = last.copy(endUs = lastEnd - tailSrc)
                     }
+                    val wrapKey =
+                        blendChromaKey(last, first, globalChromaKey, "loop wrap")
                     result.add(
                         VideoClip(
                             inputPath = rendered.outputPath,
                             startUs = 0L,
                             endUs = rendered.durationUs.takeIf { it > 0 },
+                            chromaKey = wrapKey.config,
+                            suppressChromaKey = wrapKey.suppressed,
                         )
                     )
                 } else {
@@ -605,6 +643,47 @@ class RenderVideo(private val context: Context) {
         return result
     }
 
+    /**
+     * The chroma key to apply to a pre-rendered overlap blend.
+     *
+     * The blend is composed from the raw sources by [ClipTransitionRenderer],
+     * which knows nothing about keying, so the key has to be re-applied to its
+     * output. That only has a defined meaning when both sides key the same way:
+     * blending a keyed clip with an unkeyed one produces mixed colors that no
+     * single key can undo. Mismatched sides are therefore left unkeyed and
+     * reported, rather than silently keyed with one side's settings.
+     */
+    private fun blendChromaKey(
+        outgoing: VideoClip,
+        incoming: VideoClip,
+        global: ChromaKeyConfig?,
+        boundary: String,
+    ): BlendChromaKey {
+        // Compare the *effective* keys. A clip that leaves its own key null
+        // still inherits the global one, so comparing the raw per-clip fields
+        // reported a mismatch for two clips that in fact key identically.
+        val outgoingKey = outgoing.chromaKey ?: global
+        val incomingKey = incoming.chromaKey ?: global
+        if (outgoingKey == incomingKey) return BlendChromaKey(outgoingKey, false)
+
+        Log.w(
+            RENDER_TAG,
+            "Chroma key: the two clips at boundary $boundary use different keys; " +
+                "the pre-rendered transition blend is emitted unkeyed"
+        )
+        // A null key alone would mean "inherit", and `VideoSequenceBuilder`
+        // would then fall back to the global key — applying to the blend the
+        // very thing this warning promises it will not. The suppress flag is
+        // what makes "unkeyed" actually mean unkeyed.
+        return BlendChromaKey(null, true)
+    }
+
+    /** The key for a pre-rendered blend, and whether it opts out of the global one. */
+    private data class BlendChromaKey(
+        val config: ChromaKeyConfig?,
+        val suppressed: Boolean,
+    )
+
     /**
      * Internal render implementation after optional pre-transcoding.
      */

android/src/main/kotlin/ch/waio/pro_video_editor/src/features/render/helpers/ApplyChromaKey.kt

@@ -0,0 +1,56 @@
+import androidx.media3.common.Effect
+import androidx.media3.common.util.UnstableApi
+import ch.waio.pro_video_editor.src.features.render.helpers.ChromaKeyEffect
+import ch.waio.pro_video_editor.src.features.render.models.ChromaKeyConfig
+import ch.waio.pro_video_editor.src.shared.logging.PluginLog as Log
+
+/**
+ * Adds a chroma key ("green screen" removal) to a clip's effect chain.
+ *
+ * Must be added **before** rotation, flip, the color LUT and blur, so the key
+ * sees the original decoded colors. Media3's LUT shader preserves alpha
+ * (`gl_FragColor.a = inputColor.a`), so a color filter after the key is safe.
+ *
+ * @param videoEffects List to add the effect to
+ * @param config The resolved key (clip ?: layer ?: global), or null for none
+ * @param flattenTransparency Whether a key without a background should be
+ *  filled with opaque black instead of left transparent. True on the
+ *  single-track path — see below.
+ */
+@UnstableApi
+fun applyChromaKey(
+    videoEffects: MutableList<Effect>,
+    config: ChromaKeyConfig?,
+    flattenTransparency: Boolean = false,
+) {
+    if (config == null) return
+
+    // H.264/HEVC carry no alpha channel, and the single-track path has no layer
+    // underneath to blend into, so "transparent" has no meaning there. Left
+    // alone the two platforms would disagree visibly: Apple's premultiplied
+    // color cube writes rgb*0 and the area comes out black, while this shader
+    // writes straight alpha and leaves the RGB untouched, so the screen would
+    // simply stay green — "nothing happened". Substituting opaque black makes
+    // both platforms produce the documented result.
+    val effective = if (flattenTransparency && config.isTransparent) {
+        config.copy(backgroundColor = OPAQUE_BLACK)
+    } else {
+        config
+    }
+
+    val background = when {
+        effective.backgroundImageData != null -> "image"
+        effective.backgroundColor != null -> "color"
+        else -> "transparent"
+    }
+    Log.d(
+        RENDER_TAG,
+        "Applying chroma key: similarity=${effective.similarity} " +
+            "smoothness=${effective.smoothness} spill=${effective.spill} " +
+            "background=$background"
+    )
+
+    videoEffects += ChromaKeyEffect(effective)
+}
+
+private const val OPAQUE_BLACK = 0xFF000000.toInt()

android/src/main/kotlin/ch/waio/pro_video_editor/src/features/render/helpers/ApplyComposition.kt

@@ -69,7 +69,8 @@ fun applyComposition(
             imageLayers = imageLayerConfigs,
             audioTracks = config.audioTracks,
             globalStartUs = config.startUs,
-            globalEndUs = config.endUs
+            globalEndUs = config.endUs,
+            globalChromaKey = config.chromaKey
         )
         return CompositionResult(
             layeredBuilder.build(),