AI Scene Composer

English | 简体中文

AI Scene Composer is a lightweight 3D blocking tool for AI image generation.

It solves a practical problem: when you generate images with text prompts alone, character positions, camera angle, body poses, and small structural details in the scene are often hard to control. This project lets you first build a simple 3D reference image, then send that reference to an image model so the final render keeps the spatial layout you actually intended.

The core workflow is simple:

1. Place colored proxy characters, props, and rooms in a lightweight 3D editor.
2. Set the camera and composition like a storyboard or virtual set.
3. Export a clean blocking image.
4. Use that image as the visual reference for Google Gemini / Nano Banana-style image generation.

Status: alpha. The web editor, local scene save/load, model import, and the resource library workflow are usable, and AI generation runs end-to-end with your own Gemini / Nano Banana API key (BYOK). Cloud persistence (Cloud Storage) and accounts (Firebase Auth/Firestore) are not wired into the client yet.

Why This Exists

Text-to-image models are powerful, but text is a weak control surface for geometry.

Prompts like "the child sits on the left, the tall woman stands in the middle with arms raised, two people walk on the right, wide cinematic shot" can still produce random spacing, swapped subjects, wrong poses, or an unexpected camera.

AI Scene Composer moves the hard spatial work into a simple 3D layer:

• Character positions are controlled by the layout, not only by words.
• Pose and scale are visible before generation.
• Camera framing is chosen explicitly.
• The generated image can focus on style, lighting, detail, and realism.

Example

The 3D blocking image is intentionally simple. It is not meant to be beautiful; it is a precise composition guide.

3D blocking reference	Generated image

The same idea can be used with a different prompt/style direction:

3D blocking reference	Generated image

The important part is not that the proxy figures look realistic. The important part is that the final image preserves the intended left/right arrangement, relative scale, pose direction, and camera framing much more reliably than a prompt-only workflow.

Features

• 3D virtual set editor: place proxy actors, props, and rooms.
• Color-coded characters: use simple identity colors that can be referenced in prompts.
• Shot system (storyboard): capture the current framing as a shot (camera position + aspect + thumbnail), recall/switch shots, and adjust FOV via lens presets.
• Indoor & outdoor environments: enclosed room shells plus an outdoor engine (ground, gradient sky, and lighting that switches automatically indoor↔outdoor).
• Scene presets with posed characters: one-click setups (palace audience, office, kitchen, café, clinic, in-car, street, park, …) that load with actors already placed and posed.
• AI generation (BYOK): send the blocking image + prompt to Gemini / Nano Banana with your own API key; choose the model, style, and output resolution.
• Settings panel: your API key stays in the browser (never logged); pick model and export size.
• Tri-lingual UI: English, Simplified Chinese, and Traditional Chinese.
• Clean export: generate a composition image without UI controls or selection helpers.
• Local scene files: save and reload JSON scene snapshots.
• glTF/GLB import: import user models at runtime and store them in scene snapshots as self-contained assets.
• Model splitting: split imported glTF models by top-level parts for separate selection and movement.
• File-based resource library: add assets by adding models/<id>/meta.json. Current catalog: 77 models · 19 poses · 16 scene presets.
• Open-source ready workflow: license, contribution guide, CI, linting, formatting, tests, and asset credits.

What It Is Not

• It is not a full 3D modeling tool.
• It is not a character rigging or animation system.
• It is not tied to only one image model; the editor exports a reference image that can be used with any model that accepts image references.
• It is not production SaaS yet; Firebase persistence, storage, queueing, and billing are planned but not complete.

Quick Start

Requirements:

• Node.js 20 or newer
• pnpm 9.15.3 through Corepack

corepack enable
corepack prepare pnpm@9.15.3 --activate
pnpm install
pnpm web

Open the Expo web URL printed by the dev server.

Useful Commands

pnpm gen                                  # regenerate resource catalog and credits
pnpm web                                  # start the web editor
pnpm api                                  # start the local API server (Gemini generation)
pnpm build                                # generate catalog and build/typecheck packages
pnpm build:web                            # export the Expo web build
pnpm lint                                 # ESLint
pnpm test                                 # Node test runner via tsx
pnpm typecheck                            # TypeScript checks through Turbo
pnpm format:check                         # Prettier check
pnpm check                                # local equivalent of the main CI checks
pnpm --filter @asc/resource-library check # resource catalog consistency checks

Repository Layout

apps/client                 Expo web editor: 3D scene, shots, settings, import/export UI
packages/resource-library   File-based model catalog and contribution surface
packages/shared-types       Shared TypeScript contracts for scene/generation data
services/api                Cloud Run-style Node/TypeScript API for Gemini image generation
docs/                       Architecture notes, Firebase plan, asset pipeline, roadmap

Resource Library

The resource library is designed for community contribution:

packages/resource-library/models/<asset-id>/meta.json
packages/resource-library/models/<asset-id>/model.glb

One model equals one folder. Run pnpm gen after changing model metadata. The generated catalog and license credits must stay committed.

See packages/resource-library/README.md and CONTRIBUTING.md for details.

Architecture

3D editor (Expo web + React Three Fiber)
        |
        | clean blocking reference image
        v
API service (Node / TypeScript, Cloud Run style)
        |
        | image + prompt
        v
Image generation model

The editor is intentionally thin and local-first. The server owns model API calls and credentials. The long-term plan is to add Firebase Auth, Firestore scene persistence, Cloud Storage for generated assets, and optional queueing for generation jobs.

Current Limitations

• AI generation runs end-to-end with your own API key, but results are returned inline (data URL); Cloud Storage persistence and accounts (Firebase Auth/Firestore) are not wired into the client yet.
• Scene JSON files can become large when they embed imported user models.
• Imported user assets are local scene assets; do not submit them to the repository unless redistribution rights are clear.
• Native iOS/Android packaging is not the current target. Web and iPad Safari come first.

Roadmap

• Persist scenes and generated images through Firebase (Auth + Firestore + Cloud Storage).
• Add storyboard/batch export across saved shots.
• Add stronger schema validation for scene files and imported assets.
• Expand the CC0/redistributable resource library.
• Add richer prompt templates for realistic and anime styles.
• Improve visual QA and regression testing for exported blocking images.

Contributing

The easiest way to contribute is to add clean, redistributable assets to the resource library.

Please read CONTRIBUTING.md before opening a pull request. Asset licensing is taken seriously: prefer CC0 assets, include attribution where required, and never submit assets with unclear redistribution rights.

Licensing

Code is licensed under the Apache License 2.0. See LICENSE and NOTICE.

3D assets may have their own licenses. Asset-specific metadata lives in packages/resource-library/models/*/meta.json, and generated credits live in packages/resource-library/CREDITS.md.

Do not assume a model asset is covered by the repository code license.

Security

Please report suspected vulnerabilities privately. See SECURITY.md.

Documentation

• Project structure
• Development plan
• Phase 1 tasks
• Asset library
• Firebase architecture
• Deployment
• Open-source release guide