Skip to content

[Partner Nodes] feat: add Runway Aleph2 node#14306

Merged
Kosinkadink merged 2 commits into
masterfrom
feat/api-nodes/runway-aleph2
Jun 12, 2026
Merged

[Partner Nodes] feat: add Runway Aleph2 node#14306
Kosinkadink merged 2 commits into
masterfrom
feat/api-nodes/runway-aleph2

Conversation

@bigcat88

@bigcat88 bigcat88 commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Nodes for this endpoint: https://docs.dev.runwayml.com/api/#tag/Start-generating/paths/~1v1~1video_to_video/post

API Node PR Checklist

Scope

  • Is API Node Change

Pricing & Billing

  • Need pricing update
  • No pricing update

If Need pricing update:

  • Metronome rate cards updated
  • Auto‑billing tests updated and passing

QA

  • QA done
  • QA not required

Comms

  • Informed Kosinkadink

@bigcat88 bigcat88 marked this pull request as ready for review June 12, 2026 16:44
@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds Aleph2 video-to-video editing support to the Runway node collection. It introduces three new node classes—Keyframe, Prompt Image, and Video-to-Video—that enable timeline-aware guidance images for video generation. The implementation validates constraints (prompt length, video duration, fps caps), uploads videos and guidance images, constructs Aleph2 request payloads, polls for task completion, and downloads results. Simultaneously, the PR refactors progress extraction into get_response and migrates exception handling from the removed RunwayApiError to ValueError across existing nodes.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: adding Runway Aleph2 node support. It is specific, concise, and directly reflects the primary purpose of the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The pull request description is related to the changeset, referencing the Runway Aleph2 endpoint and including a relevant API Node PR Checklist.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@comfy_api_nodes/nodes_runway.py`:
- Around line 789-815: Validate all Aleph2 inputs (keyframes and prompt_images)
before calling upload_video_to_comfyapi or upload_image_to_comfyapi: first check
keyframes is not None and enforce len(keyframes.items) <= 5 and call
_check_seconds for each KEYFRAME_MODE_SECONDS item, and similarly validate
prompt_images length and timestamps/percentages for PROMPT_IMAGE_MODE_TIMESTAMP
before any uploads; only after these validations pass, call
upload_video_to_comfyapi and then upload_image_to_comfyapi for each item to
build keyframe_models and prompt_image_models so failed/invalid requests never
trigger uploads. Ensure references to upload_video_to_comfyapi,
upload_image_to_comfyapi, keyframes, prompt_images, and _check_seconds are used
to locate and reorder the logic.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4b13f73b-2385-4493-a9e4-735d92a0265d

📥 Commits

Reviewing files that changed from the base of the PR and between d7a5527 and 370d59c.

⛔ Files ignored due to path filters (1)
  • comfy_api_nodes/apis/runway.py is excluded by !comfy_api_nodes/apis/**
📒 Files selected for processing (1)
  • comfy_api_nodes/nodes_runway.py

Comment on lines +789 to +815
video_url = await upload_video_to_comfyapi(cls, video)

keyframe_models: list[RunwayAleph2KeyframeSeconds | RunwayAleph2KeyframeAt] = []
if keyframes is not None:
if len(keyframes.items) > 5:
raise ValueError("Aleph2 supports at most 5 keyframes.")
for item in keyframes.items:
image_url = await upload_image_to_comfyapi(cls, item.image, mime_type="image/png")
if item.mode == KEYFRAME_MODE_SECONDS:
_check_seconds(item.value, "Keyframe timestamp")
keyframe_models.append(RunwayAleph2KeyframeSeconds(seconds=item.value, uri=image_url))
else:
keyframe_models.append(RunwayAleph2KeyframeAt(at=item.value, uri=image_url))

prompt_image_models: list[RunwayAleph2PromptImage] = []
if prompt_images is not None:
if len(prompt_images.items) > 5:
raise ValueError("Aleph2 supports at most 5 prompt images.")
for item in prompt_images.items:
image_url = await upload_image_to_comfyapi(cls, item.image, mime_type="image/png")
position: RunwayAleph2TimestampPosition | RunwayAleph2RelativePosition
if item.mode == PROMPT_IMAGE_MODE_TIMESTAMP:
_check_seconds(item.value, "Prompt image timestamp")
position = RunwayAleph2TimestampPosition(timestampSeconds=item.value)
else:
position = RunwayAleph2RelativePosition(positionPercentage=item.value)
prompt_image_models.append(RunwayAleph2PromptImage(position=position, uri=image_url))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Finish validating Aleph2 inputs before the first upload.

Line 789 uploads the source video, and Lines 796/808 upload guidance images, before this method rejects too many items or out-of-range timestamps. Invalid requests currently pay the full upload cost for assets that will never be used.

♻️ Suggested flow
-        video_url = await upload_video_to_comfyapi(cls, video)
-
-        keyframe_models: list[RunwayAleph2KeyframeSeconds | RunwayAleph2KeyframeAt] = []
-        if keyframes is not None:
-            if len(keyframes.items) > 5:
-                raise ValueError("Aleph2 supports at most 5 keyframes.")
+        if keyframes is not None and len(keyframes.items) > 5:
+            raise ValueError("Aleph2 supports at most 5 keyframes.")
+        if prompt_images is not None and len(prompt_images.items) > 5:
+            raise ValueError("Aleph2 supports at most 5 prompt images.")
+
+        for item in keyframes.items if keyframes is not None else []:
+            if item.mode == KEYFRAME_MODE_SECONDS:
+                _check_seconds(item.value, "Keyframe timestamp")
+        for item in prompt_images.items if prompt_images is not None else []:
+            if item.mode == PROMPT_IMAGE_MODE_TIMESTAMP:
+                _check_seconds(item.value, "Prompt image timestamp")
+
+        video_url = await upload_video_to_comfyapi(cls, video)
+
+        keyframe_models: list[RunwayAleph2KeyframeSeconds | RunwayAleph2KeyframeAt] = []
+        if keyframes is not None:
             for item in keyframes.items:
                 image_url = await upload_image_to_comfyapi(cls, item.image, mime_type="image/png")
-                if item.mode == KEYFRAME_MODE_SECONDS:
-                    _check_seconds(item.value, "Keyframe timestamp")
+                if item.mode == KEYFRAME_MODE_SECONDS:
                     keyframe_models.append(RunwayAleph2KeyframeSeconds(seconds=item.value, uri=image_url))

As per coding guidelines, comfy_api_nodes/** reviews should focus on proper error handling for API failures.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@comfy_api_nodes/nodes_runway.py` around lines 789 - 815, Validate all Aleph2
inputs (keyframes and prompt_images) before calling upload_video_to_comfyapi or
upload_image_to_comfyapi: first check keyframes is not None and enforce
len(keyframes.items) <= 5 and call _check_seconds for each KEYFRAME_MODE_SECONDS
item, and similarly validate prompt_images length and timestamps/percentages for
PROMPT_IMAGE_MODE_TIMESTAMP before any uploads; only after these validations
pass, call upload_video_to_comfyapi and then upload_image_to_comfyapi for each
item to build keyframe_models and prompt_image_models so failed/invalid requests
never trigger uploads. Ensure references to upload_video_to_comfyapi,
upload_image_to_comfyapi, keyframes, prompt_images, and _check_seconds are used
to locate and reorder the logic.

Source: Coding guidelines

@Kosinkadink Kosinkadink merged commit 28a40fb into master Jun 12, 2026
23 checks passed
@bigcat88 bigcat88 deleted the feat/api-nodes/runway-aleph2 branch June 15, 2026 08:01
POWERFULMOVES added a commit to POWERFULMOVES/PMOVES-Creator that referenced this pull request Jun 17, 2026
* feat(assets): add job_ids filter to GET /api/assets (Comfy-Org#13998)

* feat(assets): add job_ids filter to GET /api/assets

Mirrors the existing cloud `job_ids` query param on the local Python server:
clients can pass a comma-separated list (or repeated query params) of UUIDs
to filter assets by their associated job.

The `AssetReference.job_id` column already exists, so no migration is
needed — this just plumbs the filter through schema → service → query.

Marks the parameter as available in both runtimes by dropping the
`[cloud-only]` description prefix and the `x-runtime: [cloud]` tag from
the OpenAPI spec, per the OSS field-drift convention (absent runtime tag
= populated by both local and cloud).

* fix(assets): tighten job_ids — array schema, max_length, narrow except

From cursor-reviews on the parent commit:

- OpenAPI: declare job_ids as `type: array, items: string format: uuid`
  with `style: form, explode: true` so it matches the documented
  contract (and matches sibling include_tags/exclude_tags shape).
  Description now states both accepted shapes explicitly.
- Schema: cap `job_ids` at 500 entries (max_length on the Pydantic
  field) so a client can't splice an unbounded list into the IN clauses.
- Schema: drop `AttributeError` from the except — `raw` only contains
  `str` items by construction, so `uuid.UUID(<str>)` raises `ValueError`
  exclusively; the second clause was dead code.

* fix(assets): tighten job_ids validator + add schema-level tests

Aligns with the parallel hardening from draft PR Comfy-Org#13848 (now closed as
a duplicate). The validator now:

- Raises ValueError on non-string list items (was: silently dropped).
- Raises ValueError on non-string / non-list top-level values like dict
  or int (was: silently passed through to Pydantic's downstream coercion).

Adds tests-unit/assets_test/queries/test_list_assets_query.py covering
the validator end-to-end: CSV canonicalization, dedup order, default
empty, invalid UUID, non-string list item, non-string non-list value,
and the max_length=500 boundary.

* feat(prompt): enforce canonical UUID prompt_id at job creation

POST /prompt previously accepted any client-supplied prompt_id verbatim,
str()-coercing even non-strings, and minting the literal job id "None"
for an explicit JSON null. The new GET /api/assets job_ids filter matches
stored job ids as canonical UUIDs exactly, so a non-UUID id minted a job
whose assets could never be filtered.

- validate_job_id (comfy_execution/jobs.py): requires a string in the
  canonical lowercase hyphenated UUID form; raises ValueError otherwise,
  including parseable-but-non-canonical spellings (uppercase, braced, URN,
  bare hex), which would otherwise be silently rewritten and then miss
  every exact-match lookup downstream (history keys, websocket
  correlation, /interrupt, the assets job_ids filter).
- POST /prompt: absent or null prompt_id means the server mints uuid4;
  invalid means 400 invalid_prompt_id on the standard error envelope.
- openapi.yaml: document the request-side prompt_id (format uuid,
  nullable) on PromptRequest.
- tests: unit matrix for validate_job_id; integration tests against the
  booted server covering rejection, acceptance, and null handling.

---------

Co-authored-by: guill <jacob.e.segal@gmail.com>

* feat(assets): include asset id in executed WebSocket message (Comfy-Org#13862)

* feat(assets): enrich executed WS message with asset metadata

When --enable-assets is set, each file-type output entry in the
`executed` WebSocket message now includes id, name, asset_hash, size,
and mime_type — matching the shape already returned by /upload/image.

The enrichment lives in comfy_execution/asset_enrichment.py (no torch
dependency) and is called from both send sites in execution.py: freshly
executed nodes register the file inline via register_file_in_place;
cached node re-sends look up the existing AssetReference by file path
to avoid re-hashing. Errors are caught per-entry so a failure never
blocks the WS message from sending.

* fix(assets): inject only id in executed WS message per Asset Identity RFC

Per the Asset Identity RFC, the executed WebSocket payload should carry
id alone — hash is already encoded in the filename, and name/preview_url/
size belong behind GET /api/assets/{id} rather than being pushed eagerly.

Simplifies the DB lookup path: we only need ref.id, so the asset.hash
null-check is no longer required as a fallback trigger.

* fix(assets): reject path traversal when resolving output abs_path

Subfolder/filename were joined and absolutized without containment check,
so '..' segments or an absolute filename could escape the type's base
directory and register an unrelated on-disk file as an asset.

Add commonpath-based containment check; skip enrichment (warn, leave
entry unchanged) when the resolved path escapes base. Catches ValueError
from cross-drive paths on Windows.

* docs(assets): drop Asset Identity RFC reference from docstring

* docs(assets): trim docstring to what enrichment does, not what it doesn't

* test(assets): use real platform paths so containment check works on Windows

The previous test setup patched os.path.abspath to identity and used a
POSIX-style '/output' base, which collided with Windows path separators
in os.path.commonpath. Drop the abspath/join patches and use a real
tempdir-rooted base so the containment check runs against actual
platform paths.

* refactor(assets): enrich at output-processing time, not in the WS send path

Per review: enrichment lived inside the client_id-guarded send sites, so a
headless run (no websocket client) never registered assets at all, and
ui_outputs/history stored the un-enriched entries.

Now output_ui is enriched once, right after the node produces it and before
it is stored in ui_outputs — so registration happens regardless of connected
clients, and the asset id flows into history and the execution cache for
free. _send_cached_ui re-sends the stored (already-enriched) dict verbatim,
which lets the DB-lookup-by-path fallback be deleted: every enrichment is
now a fresh output, and register_file_in_place re-hashes on upsert so an
overwritten path can never carry a stale id.

* revert(assets): drop job_ids filter from GET /api/assets (Comfy-Org#14408)

The job_ids query filter added in Comfy-Org#13998 has no live consumer: the
frontend Generated tab kept sourcing from GET /jobs, and the cloud side
removed its equivalent filter from the shared asset spec. Carrying it on
the local server only re-introduces Core<->Cloud drift on the shared
contract, so remove it to match.

Removed: the job_ids field + validator on ListAssetsQuery, the IN(...)
clauses in list_references_page, the service/route passthrough, and the
filter-only tests.

Kept: the canonical-UUID prompt_id enforcement at job creation (also
landed in Comfy-Org#13998). It stands on its own -- job ids are matched verbatim
by history keys, websocket correlation, and /interrupt -- and cloud
inherits it by running core for execution, so no divergence is created.

* chore(openapi): sync shared API contract from cloud@e3c52ad (Comfy-Org#14406)

* I don't think this actually works anymore. (Comfy-Org#14403)

* ops: tolerate already force casted dynamic weight (Comfy-Org#14410)

Some custom nodes .to weights completely out of load context which
can wreak havoc if its for a model that is not active. Detect this
condition and just let it fall-through to the non-dynamic loader
straight up.

* Improve context window resizing for SCAIL2 (CORE-286) (Comfy-Org#14394)

* Fix SCAIL-2 reference mask background convention (Comfy-Org#14415)

* [Partner Nodes] fix(GPT Image): handle mismatched image sizes returned when size="auto" (Comfy-Org#14414)

Signed-off-by: bigcat88 <bigcat88@icloud.com>

* [Partner Nodes] fix(KlingTextToVideoNode): validation error for "kling-v2-master" model (Comfy-Org#14418)

Signed-off-by: bigcat88 <bigcat88@icloud.com>

* Make --enable-manager-legacy-ui imply --enable-manager (Comfy-Org#14421)

* Don't crash when using flux kv cache with split batches. (Comfy-Org#14422)

* Add Comfy-Usage-Source pass-through for API node requests (Comfy-Org#14404)

* [Partner Nodes] feat: enable Bria Replace Background node (Comfy-Org#14397)

* Fix potential dtype issue with ideogram 4. (Comfy-Org#14436)

* add --high-ram option (Comfy-Org#14437)

Add this option for users who know they have so much ram they want
to pin everything or have a pagefile that outruns their disk speed.

The removes the RAM pressure caps completely and pins behind the
primary model load forcing all models to be permanently comitted
to RAM.

* [Partner Nodes] feat: add Runway Aleph2 node (Comfy-Org#14306)

Signed-off-by: bigcat88 <bigcat88@icloud.com>

* Use comfy kitchen apply rope in omnigen2 model. (Comfy-Org#14442)

* Add 10-bit video support (Comfy-Org#14452)

Create Video gets a bit_depth option (8-bit/10-bit); the selected depth is carried by the video and applied when it gets encoded. Save Video and Video Slice now keep the source bit depth instead of always quantizing to 8-bit, so 10-bit videos stay 10-bit. 10-bit uses h264 with the yuv420p10le pixel format,so there's no new codec or container.

Signed-off-by: bigcat88 <bigcat88@icloud.com>

* Expose deploy_environment in /system_stats (Comfy-Org#14402)

* Remove the comfy python path append.

* Revert last commit. Last time I use this stupid GitHub app.

* Fix nondeterministic video decode at unaligned widths (CORE-299) (Comfy-Org#14438)

* [Partner Nodes] feat(Tripo3d): add new "Import 3D" node (Comfy-Org#14466)

Signed-off-by: bigcat88 <bigcat88@icloud.com>

* bump manager version to 4.2.2 (Comfy-Org#14471)

* This is already auto enabled by default. (Comfy-Org#14476)

* chore: update embedded docs to v0.5.4 (Comfy-Org#14478)

* fix(ruff): exclude vendored ComfyUI tree from lint (scope to PMOVES code)

* fix(ruff): resolve PMOVES-owned lint in pmoves_health

Scoped ruff (after excluding vendored ComfyUI) surfaced PMOVES-owned violations:
remove unused imports (F401), fix undefined FastAPI annotation (F821) by importing
it at module scope, and # noqa intentional demo/CLI prints (T201).

* fix(ruff): resolve PMOVES-owned lint in pmoves_announcer

Scoped ruff (after excluding vendored ComfyUI) surfaced PMOVES-owned violations:
remove unused imports (F401), fix undefined FastAPI annotation (F821) by importing
it at module scope, and # noqa intentional demo/CLI prints (T201).

* fix(ruff): resolve PMOVES-owned lint in pmoves_registry

Scoped ruff (after excluding vendored ComfyUI) surfaced PMOVES-owned violations:
remove unused imports (F401), fix undefined FastAPI annotation (F821) by importing
it at module scope, and # noqa intentional demo/CLI prints (T201).

---------

Signed-off-by: bigcat88 <bigcat88@icloud.com>
Co-authored-by: Matt Miller <mattmiller@comfy.org>
Co-authored-by: guill <jacob.e.segal@gmail.com>
Co-authored-by: Comfy Org PR Bot <snomiao+comfy-pr@gmail.com>
Co-authored-by: comfyanonymous <121283862+comfyanonymous@users.noreply.github.com>
Co-authored-by: rattus <46076784+rattus128@users.noreply.github.com>
Co-authored-by: Barish Ozbay <17261091+drozbay@users.noreply.github.com>
Co-authored-by: Jukka Seppänen <40791699+kijai@users.noreply.github.com>
Co-authored-by: Alexander Piskun <13381981+bigcat88@users.noreply.github.com>
Co-authored-by: Jedrzej Kosinski <kosinkadink1@gmail.com>
Co-authored-by: Robin Huang <robin.j.huang@gmail.com>
Co-authored-by: John Pollock <pollockjj@users.noreply.github.com>
Co-authored-by: Dr.Lt.Data <128333288+ltdrdata@users.noreply.github.com>
Co-authored-by: Daxiong (Lin) <contact@comfyui-wiki.com>
Co-authored-by: pmoves-fork-sync[bot] <pmoves-fork-sync[bot]@users.noreply.github.com>
Co-authored-by: POWERFULMOVES <142271328+POWERFULMOVES@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants