[Partner Nodes] feat: add Runway Aleph2 node#14306
Conversation
Signed-off-by: bigcat88 <bigcat88@icloud.com>
📝 WalkthroughWalkthroughThis PR adds Aleph2 video-to-video editing support to the Runway node collection. It introduces three new node classes—Keyframe, Prompt Image, and Video-to-Video—that enable timeline-aware guidance images for video generation. The implementation validates constraints (prompt length, video duration, fps caps), uploads videos and guidance images, constructs Aleph2 request payloads, polls for task completion, and downloads results. Simultaneously, the PR refactors progress extraction into get_response and migrates exception handling from the removed RunwayApiError to ValueError across existing nodes. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@comfy_api_nodes/nodes_runway.py`:
- Around line 789-815: Validate all Aleph2 inputs (keyframes and prompt_images)
before calling upload_video_to_comfyapi or upload_image_to_comfyapi: first check
keyframes is not None and enforce len(keyframes.items) <= 5 and call
_check_seconds for each KEYFRAME_MODE_SECONDS item, and similarly validate
prompt_images length and timestamps/percentages for PROMPT_IMAGE_MODE_TIMESTAMP
before any uploads; only after these validations pass, call
upload_video_to_comfyapi and then upload_image_to_comfyapi for each item to
build keyframe_models and prompt_image_models so failed/invalid requests never
trigger uploads. Ensure references to upload_video_to_comfyapi,
upload_image_to_comfyapi, keyframes, prompt_images, and _check_seconds are used
to locate and reorder the logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 4b13f73b-2385-4493-a9e4-735d92a0265d
⛔ Files ignored due to path filters (1)
comfy_api_nodes/apis/runway.pyis excluded by!comfy_api_nodes/apis/**
📒 Files selected for processing (1)
comfy_api_nodes/nodes_runway.py
| video_url = await upload_video_to_comfyapi(cls, video) | ||
|
|
||
| keyframe_models: list[RunwayAleph2KeyframeSeconds | RunwayAleph2KeyframeAt] = [] | ||
| if keyframes is not None: | ||
| if len(keyframes.items) > 5: | ||
| raise ValueError("Aleph2 supports at most 5 keyframes.") | ||
| for item in keyframes.items: | ||
| image_url = await upload_image_to_comfyapi(cls, item.image, mime_type="image/png") | ||
| if item.mode == KEYFRAME_MODE_SECONDS: | ||
| _check_seconds(item.value, "Keyframe timestamp") | ||
| keyframe_models.append(RunwayAleph2KeyframeSeconds(seconds=item.value, uri=image_url)) | ||
| else: | ||
| keyframe_models.append(RunwayAleph2KeyframeAt(at=item.value, uri=image_url)) | ||
|
|
||
| prompt_image_models: list[RunwayAleph2PromptImage] = [] | ||
| if prompt_images is not None: | ||
| if len(prompt_images.items) > 5: | ||
| raise ValueError("Aleph2 supports at most 5 prompt images.") | ||
| for item in prompt_images.items: | ||
| image_url = await upload_image_to_comfyapi(cls, item.image, mime_type="image/png") | ||
| position: RunwayAleph2TimestampPosition | RunwayAleph2RelativePosition | ||
| if item.mode == PROMPT_IMAGE_MODE_TIMESTAMP: | ||
| _check_seconds(item.value, "Prompt image timestamp") | ||
| position = RunwayAleph2TimestampPosition(timestampSeconds=item.value) | ||
| else: | ||
| position = RunwayAleph2RelativePosition(positionPercentage=item.value) | ||
| prompt_image_models.append(RunwayAleph2PromptImage(position=position, uri=image_url)) |
There was a problem hiding this comment.
Finish validating Aleph2 inputs before the first upload.
Line 789 uploads the source video, and Lines 796/808 upload guidance images, before this method rejects too many items or out-of-range timestamps. Invalid requests currently pay the full upload cost for assets that will never be used.
♻️ Suggested flow
- video_url = await upload_video_to_comfyapi(cls, video)
-
- keyframe_models: list[RunwayAleph2KeyframeSeconds | RunwayAleph2KeyframeAt] = []
- if keyframes is not None:
- if len(keyframes.items) > 5:
- raise ValueError("Aleph2 supports at most 5 keyframes.")
+ if keyframes is not None and len(keyframes.items) > 5:
+ raise ValueError("Aleph2 supports at most 5 keyframes.")
+ if prompt_images is not None and len(prompt_images.items) > 5:
+ raise ValueError("Aleph2 supports at most 5 prompt images.")
+
+ for item in keyframes.items if keyframes is not None else []:
+ if item.mode == KEYFRAME_MODE_SECONDS:
+ _check_seconds(item.value, "Keyframe timestamp")
+ for item in prompt_images.items if prompt_images is not None else []:
+ if item.mode == PROMPT_IMAGE_MODE_TIMESTAMP:
+ _check_seconds(item.value, "Prompt image timestamp")
+
+ video_url = await upload_video_to_comfyapi(cls, video)
+
+ keyframe_models: list[RunwayAleph2KeyframeSeconds | RunwayAleph2KeyframeAt] = []
+ if keyframes is not None:
for item in keyframes.items:
image_url = await upload_image_to_comfyapi(cls, item.image, mime_type="image/png")
- if item.mode == KEYFRAME_MODE_SECONDS:
- _check_seconds(item.value, "Keyframe timestamp")
+ if item.mode == KEYFRAME_MODE_SECONDS:
keyframe_models.append(RunwayAleph2KeyframeSeconds(seconds=item.value, uri=image_url))As per coding guidelines, comfy_api_nodes/** reviews should focus on proper error handling for API failures.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@comfy_api_nodes/nodes_runway.py` around lines 789 - 815, Validate all Aleph2
inputs (keyframes and prompt_images) before calling upload_video_to_comfyapi or
upload_image_to_comfyapi: first check keyframes is not None and enforce
len(keyframes.items) <= 5 and call _check_seconds for each KEYFRAME_MODE_SECONDS
item, and similarly validate prompt_images length and timestamps/percentages for
PROMPT_IMAGE_MODE_TIMESTAMP before any uploads; only after these validations
pass, call upload_video_to_comfyapi and then upload_image_to_comfyapi for each
item to build keyframe_models and prompt_image_models so failed/invalid requests
never trigger uploads. Ensure references to upload_video_to_comfyapi,
upload_image_to_comfyapi, keyframes, prompt_images, and _check_seconds are used
to locate and reorder the logic.
Source: Coding guidelines
* feat(assets): add job_ids filter to GET /api/assets (Comfy-Org#13998) * feat(assets): add job_ids filter to GET /api/assets Mirrors the existing cloud `job_ids` query param on the local Python server: clients can pass a comma-separated list (or repeated query params) of UUIDs to filter assets by their associated job. The `AssetReference.job_id` column already exists, so no migration is needed — this just plumbs the filter through schema → service → query. Marks the parameter as available in both runtimes by dropping the `[cloud-only]` description prefix and the `x-runtime: [cloud]` tag from the OpenAPI spec, per the OSS field-drift convention (absent runtime tag = populated by both local and cloud). * fix(assets): tighten job_ids — array schema, max_length, narrow except From cursor-reviews on the parent commit: - OpenAPI: declare job_ids as `type: array, items: string format: uuid` with `style: form, explode: true` so it matches the documented contract (and matches sibling include_tags/exclude_tags shape). Description now states both accepted shapes explicitly. - Schema: cap `job_ids` at 500 entries (max_length on the Pydantic field) so a client can't splice an unbounded list into the IN clauses. - Schema: drop `AttributeError` from the except — `raw` only contains `str` items by construction, so `uuid.UUID(<str>)` raises `ValueError` exclusively; the second clause was dead code. * fix(assets): tighten job_ids validator + add schema-level tests Aligns with the parallel hardening from draft PR Comfy-Org#13848 (now closed as a duplicate). The validator now: - Raises ValueError on non-string list items (was: silently dropped). - Raises ValueError on non-string / non-list top-level values like dict or int (was: silently passed through to Pydantic's downstream coercion). Adds tests-unit/assets_test/queries/test_list_assets_query.py covering the validator end-to-end: CSV canonicalization, dedup order, default empty, invalid UUID, non-string list item, non-string non-list value, and the max_length=500 boundary. * feat(prompt): enforce canonical UUID prompt_id at job creation POST /prompt previously accepted any client-supplied prompt_id verbatim, str()-coercing even non-strings, and minting the literal job id "None" for an explicit JSON null. The new GET /api/assets job_ids filter matches stored job ids as canonical UUIDs exactly, so a non-UUID id minted a job whose assets could never be filtered. - validate_job_id (comfy_execution/jobs.py): requires a string in the canonical lowercase hyphenated UUID form; raises ValueError otherwise, including parseable-but-non-canonical spellings (uppercase, braced, URN, bare hex), which would otherwise be silently rewritten and then miss every exact-match lookup downstream (history keys, websocket correlation, /interrupt, the assets job_ids filter). - POST /prompt: absent or null prompt_id means the server mints uuid4; invalid means 400 invalid_prompt_id on the standard error envelope. - openapi.yaml: document the request-side prompt_id (format uuid, nullable) on PromptRequest. - tests: unit matrix for validate_job_id; integration tests against the booted server covering rejection, acceptance, and null handling. --------- Co-authored-by: guill <jacob.e.segal@gmail.com> * feat(assets): include asset id in executed WebSocket message (Comfy-Org#13862) * feat(assets): enrich executed WS message with asset metadata When --enable-assets is set, each file-type output entry in the `executed` WebSocket message now includes id, name, asset_hash, size, and mime_type — matching the shape already returned by /upload/image. The enrichment lives in comfy_execution/asset_enrichment.py (no torch dependency) and is called from both send sites in execution.py: freshly executed nodes register the file inline via register_file_in_place; cached node re-sends look up the existing AssetReference by file path to avoid re-hashing. Errors are caught per-entry so a failure never blocks the WS message from sending. * fix(assets): inject only id in executed WS message per Asset Identity RFC Per the Asset Identity RFC, the executed WebSocket payload should carry id alone — hash is already encoded in the filename, and name/preview_url/ size belong behind GET /api/assets/{id} rather than being pushed eagerly. Simplifies the DB lookup path: we only need ref.id, so the asset.hash null-check is no longer required as a fallback trigger. * fix(assets): reject path traversal when resolving output abs_path Subfolder/filename were joined and absolutized without containment check, so '..' segments or an absolute filename could escape the type's base directory and register an unrelated on-disk file as an asset. Add commonpath-based containment check; skip enrichment (warn, leave entry unchanged) when the resolved path escapes base. Catches ValueError from cross-drive paths on Windows. * docs(assets): drop Asset Identity RFC reference from docstring * docs(assets): trim docstring to what enrichment does, not what it doesn't * test(assets): use real platform paths so containment check works on Windows The previous test setup patched os.path.abspath to identity and used a POSIX-style '/output' base, which collided with Windows path separators in os.path.commonpath. Drop the abspath/join patches and use a real tempdir-rooted base so the containment check runs against actual platform paths. * refactor(assets): enrich at output-processing time, not in the WS send path Per review: enrichment lived inside the client_id-guarded send sites, so a headless run (no websocket client) never registered assets at all, and ui_outputs/history stored the un-enriched entries. Now output_ui is enriched once, right after the node produces it and before it is stored in ui_outputs — so registration happens regardless of connected clients, and the asset id flows into history and the execution cache for free. _send_cached_ui re-sends the stored (already-enriched) dict verbatim, which lets the DB-lookup-by-path fallback be deleted: every enrichment is now a fresh output, and register_file_in_place re-hashes on upsert so an overwritten path can never carry a stale id. * revert(assets): drop job_ids filter from GET /api/assets (Comfy-Org#14408) The job_ids query filter added in Comfy-Org#13998 has no live consumer: the frontend Generated tab kept sourcing from GET /jobs, and the cloud side removed its equivalent filter from the shared asset spec. Carrying it on the local server only re-introduces Core<->Cloud drift on the shared contract, so remove it to match. Removed: the job_ids field + validator on ListAssetsQuery, the IN(...) clauses in list_references_page, the service/route passthrough, and the filter-only tests. Kept: the canonical-UUID prompt_id enforcement at job creation (also landed in Comfy-Org#13998). It stands on its own -- job ids are matched verbatim by history keys, websocket correlation, and /interrupt -- and cloud inherits it by running core for execution, so no divergence is created. * chore(openapi): sync shared API contract from cloud@e3c52ad (Comfy-Org#14406) * I don't think this actually works anymore. (Comfy-Org#14403) * ops: tolerate already force casted dynamic weight (Comfy-Org#14410) Some custom nodes .to weights completely out of load context which can wreak havoc if its for a model that is not active. Detect this condition and just let it fall-through to the non-dynamic loader straight up. * Improve context window resizing for SCAIL2 (CORE-286) (Comfy-Org#14394) * Fix SCAIL-2 reference mask background convention (Comfy-Org#14415) * [Partner Nodes] fix(GPT Image): handle mismatched image sizes returned when size="auto" (Comfy-Org#14414) Signed-off-by: bigcat88 <bigcat88@icloud.com> * [Partner Nodes] fix(KlingTextToVideoNode): validation error for "kling-v2-master" model (Comfy-Org#14418) Signed-off-by: bigcat88 <bigcat88@icloud.com> * Make --enable-manager-legacy-ui imply --enable-manager (Comfy-Org#14421) * Don't crash when using flux kv cache with split batches. (Comfy-Org#14422) * Add Comfy-Usage-Source pass-through for API node requests (Comfy-Org#14404) * [Partner Nodes] feat: enable Bria Replace Background node (Comfy-Org#14397) * Fix potential dtype issue with ideogram 4. (Comfy-Org#14436) * add --high-ram option (Comfy-Org#14437) Add this option for users who know they have so much ram they want to pin everything or have a pagefile that outruns their disk speed. The removes the RAM pressure caps completely and pins behind the primary model load forcing all models to be permanently comitted to RAM. * [Partner Nodes] feat: add Runway Aleph2 node (Comfy-Org#14306) Signed-off-by: bigcat88 <bigcat88@icloud.com> * Use comfy kitchen apply rope in omnigen2 model. (Comfy-Org#14442) * Add 10-bit video support (Comfy-Org#14452) Create Video gets a bit_depth option (8-bit/10-bit); the selected depth is carried by the video and applied when it gets encoded. Save Video and Video Slice now keep the source bit depth instead of always quantizing to 8-bit, so 10-bit videos stay 10-bit. 10-bit uses h264 with the yuv420p10le pixel format,so there's no new codec or container. Signed-off-by: bigcat88 <bigcat88@icloud.com> * Expose deploy_environment in /system_stats (Comfy-Org#14402) * Remove the comfy python path append. * Revert last commit. Last time I use this stupid GitHub app. * Fix nondeterministic video decode at unaligned widths (CORE-299) (Comfy-Org#14438) * [Partner Nodes] feat(Tripo3d): add new "Import 3D" node (Comfy-Org#14466) Signed-off-by: bigcat88 <bigcat88@icloud.com> * bump manager version to 4.2.2 (Comfy-Org#14471) * This is already auto enabled by default. (Comfy-Org#14476) * chore: update embedded docs to v0.5.4 (Comfy-Org#14478) * fix(ruff): exclude vendored ComfyUI tree from lint (scope to PMOVES code) * fix(ruff): resolve PMOVES-owned lint in pmoves_health Scoped ruff (after excluding vendored ComfyUI) surfaced PMOVES-owned violations: remove unused imports (F401), fix undefined FastAPI annotation (F821) by importing it at module scope, and # noqa intentional demo/CLI prints (T201). * fix(ruff): resolve PMOVES-owned lint in pmoves_announcer Scoped ruff (after excluding vendored ComfyUI) surfaced PMOVES-owned violations: remove unused imports (F401), fix undefined FastAPI annotation (F821) by importing it at module scope, and # noqa intentional demo/CLI prints (T201). * fix(ruff): resolve PMOVES-owned lint in pmoves_registry Scoped ruff (after excluding vendored ComfyUI) surfaced PMOVES-owned violations: remove unused imports (F401), fix undefined FastAPI annotation (F821) by importing it at module scope, and # noqa intentional demo/CLI prints (T201). --------- Signed-off-by: bigcat88 <bigcat88@icloud.com> Co-authored-by: Matt Miller <mattmiller@comfy.org> Co-authored-by: guill <jacob.e.segal@gmail.com> Co-authored-by: Comfy Org PR Bot <snomiao+comfy-pr@gmail.com> Co-authored-by: comfyanonymous <121283862+comfyanonymous@users.noreply.github.com> Co-authored-by: rattus <46076784+rattus128@users.noreply.github.com> Co-authored-by: Barish Ozbay <17261091+drozbay@users.noreply.github.com> Co-authored-by: Jukka Seppänen <40791699+kijai@users.noreply.github.com> Co-authored-by: Alexander Piskun <13381981+bigcat88@users.noreply.github.com> Co-authored-by: Jedrzej Kosinski <kosinkadink1@gmail.com> Co-authored-by: Robin Huang <robin.j.huang@gmail.com> Co-authored-by: John Pollock <pollockjj@users.noreply.github.com> Co-authored-by: Dr.Lt.Data <128333288+ltdrdata@users.noreply.github.com> Co-authored-by: Daxiong (Lin) <contact@comfyui-wiki.com> Co-authored-by: pmoves-fork-sync[bot] <pmoves-fork-sync[bot]@users.noreply.github.com> Co-authored-by: POWERFULMOVES <142271328+POWERFULMOVES@users.noreply.github.com>
Nodes for this endpoint: https://docs.dev.runwayml.com/api/#tag/Start-generating/paths/~1v1~1video_to_video/post
API Node PR Checklist
Scope
Pricing & Billing
If Need pricing update:
QA
Comms