diff --git a/docs/plans/platform/009_ima2_image_engine.md b/docs/plans/platform/009_ima2_image_engine.md index 30570a4..31d550c 100644 --- a/docs/plans/platform/009_ima2_image_engine.md +++ b/docs/plans/platform/009_ima2_image_engine.md @@ -1,15 +1,13 @@ # LINA image engine: ima2-gen -Date: 2026-09-07. Status: adoption approved by the user; integration pending refactor. +Date: 2026-09-07. Status: first image slice implemented and qualified through ima2 OAuth / gpt-5.6-luna. Parent: [Installable LINA refactor preparation](008_refactor_preparation.md). Upstream: [ima2-gen](https://github.com/lidge-jun/ima2-gen). ## Decision Adopt ima2-gen as LINA's image-generation engine and ecosystem integration. -Implement the adapter after the ongoing refactor establishes the owning modules. -This records an accepted product decision, not a completed runtime integration. -Attach the integration to the Lina-owned tool/job boundary. +The adapter attaches to Lina-owned tools in `lina-runtime`, registered by the Codex session host. Source verification and authorized external generation qualification are separate acceptance steps. LINA owns user intent, conversation and agent association, job tracking, and result delivery. ima2 owns visual generation, its provider connections, and its studio. @@ -74,4 +72,233 @@ The upstream [CLI documentation](https://github.com/lidge-jun/ima2-gen/blob/main and README for setup, discovery and generation interfaces; these are implementation leads, not pinned compatibility or end-to-end evidence. -Provider integration remains planned; this document does not authorize a generation call or service change. +External qualification for the selected route is recorded below. Additional generation calls and service changes remain separately authorized. + + +## Local implementation + +The adapter is pinned to **ima2-gen 3.14.0**, source revision +[`36aa6fcea62f753d858d20c4823e04922951c954`](https://github.com/lidge-jun/ima2-gen/tree/36aa6fcea62f753d858d20c4823e04922951c954). +Other versions fail with `UNSUPPORTED_VERSION` until their contracts are qualified. +The runtime remains external: Lina does not install, launch, stop, or update it. + +| Owner | Responsibility | +| --- | --- | +| `packages/lina-runtime/src/images/client*.ts` | Discovery, health/catalog parsing, explicit selection, submit/read/cancel, bounded result download | +| `packages/lina-runtime/src/images/store.ts` | Session-bound job identity, state and provenance under the existing app lifetime lease | +| `packages/lina-runtime/src/images/jobs.ts` | Reconciliation, cancellation intent, managed attachment import, durable completion delivery | +| `packages/lina-runtime/src/images/tools.ts` | `lina_image_models`, `generate`, `edit`, `jobs`, `read`, `cancel` tools using the existing Codex authorization boundary | +| `packages/lina-runtime/src/session-app.ts` | Registration, per-session storage and lifecycle; existing Codex notice writer handles conversation delivery | +| `packages/lina-core/src/attachments/store.ts` | Immutable bytes, MIME/container checks, hash and ownership; stable artifact IDs support crash-safe import replay | +| `packages/lina-web/client/markdown*.ts` | Same-session managed image previews inside existing assistant messages | + +Existing attachment, control and notice owners were located before adding this adapter. +The Codex task manager owns native coding sessions and cannot represent ima2 jobs; +its existing notice delivery is reusable, while image execution needs separate state. +No new engine SDK or package dependency is introduced. UI redesign, world logic, +video, batches, masks and Studio workflows remain outside this change. + +### Connection and use + +Start and configure ima2 separately using its own setup interface. Lina discovers +its actual address from `~/.ima2/server.json` (`backend.url`), or the owner can set +`LINA_IMA2_URL` to an explicit HTTP(S) origin. `LINA_IMA2_SERVER_FILE` overrides the +local discovery path. Credentials embedded in URLs, redirects and arbitrary result +URLs are rejected. File-based discovery is restricted to loopback; an explicit origin +may be remote. This slice does not forward cookies or bearer/LAN tokens, so a remote +service must already permit these requests. Protected services return an access +error; Lina does not weaken their authentication. + +In the existing conversation, ask Lina to inspect image connections. The +`lina_image_models` tool returns the setup URL, live catalog and operation support, +with service ownership marked external and prior verified generation records listed +separately. Choose an exact provider/model, request one image, then refer to that +image for a follow-up edit. The tool supplies the saved attachment ID and its checked +bytes as the edit reference. No provider or billing route changes automatically. + +Catalog entries are read from ima2, including unavailable lanes. Adapter operation +support also checks the pinned generation route: an unknown lane must never fall +through to ima2's default provider. Lanes whose explicit selection or reference +contract is unsupported remain visible but cannot generate/edit through this slice. + +Both creation and reference editing use `POST /api/generate` with `async: true`, +`n: 1`, explicit provider/model, and the same durable UUID for request and idempotency +identity. This intentionally uses generation with an image reference: upstream's +synchronous `/api/edit` does not supply the same recovery contract. + +### State, ownership and recovery + +The current conversation binding fixes the agent, session, workspace and journal. +Image records live in its `images/jobs.json`, independently of Codex work-session +state. Each record retains the original chat request/call, provider/model, selected +endpoint/version, source artifact ID and resulting attachment metadata. + +The adapter records submission intent before its first generation request. A lost +response stays uncertain; restart and `lina_image_read` only reconcile the same ID +using `/api/inflight?includeTerminal=1` and exact request-filtered history. Expired +idempotency/terminal records never authorize another generation. A changed endpoint +or runtime version cannot inherit an old request. One unresolved image job blocks a +new generation in that conversation until its outcome is resolved. If ima2 has lost +both the terminal and history records, the job remains uncertain and requires +operator investigation in ima2. This slice provides no discard/force-retry bypass. +Existing attachments remain readable; restoring upstream evidence permits normal +reconciliation to continue. + +Queued, running and post-processing states remain visible. Cancellation intent is +saved immediately, including while a preceding status read or submit is blocked. +It survives restart and is retried only against an observed +active job. A cancellation acknowledgement is not proof that a provider stopped or +refunded the request. A completed result racing cancellation is retained. + +Results must be one valid PNG or JPEG within the existing **2 MiB** attachment limit. +The generated filename is resolved only under the selected ima2 origin; Lina verifies +its media type, container, dimensions and hash before import. Larger or unsupported +outputs become a terminal import failure, retain the upstream filename/request +identity, and are not displayed. They do not block a later explicitly requested +generation. Transport failures during import remain uncertain. Stable artifact IDs retain +identical bytes across replay and reject conflicts without overwriting files. + +The existing Codex notice journal records `image_` before delivering a message. +An active conversation defers that notice until idle. Restart recognizes the same +marker even if the local delivery receipt was not saved. The journal stores the +full message, not just a sent flag: the conversation snapshot replays it after a +crash before browser delivery. A delivery failure is recorded separately from the +image result, retried later, and does not prevent application shutdown. Only matching-session +attachment preview URLs render as images; external URLs, data URLs and foreign-session +references stay text. Original images remain available after editing. + +### Verification and external-call gate + +Local verification uses synthetic temporary state and local fake dependencies, never +production conversations, model credentials or an existing daemon. Contract fixtures +come from the pinned source, rather than an invented provider schema. Tests cover +submission uncertainty, wrong model/authentication, outage, result validation, +reference ownership, concurrent read/cancel, cancellation recovery, artifact replay, +notice deferral and restart. Independent review also reproduced and closed a +shutdown failure after a rejected notice, the oversized-output admission block, +and cancellation intent lost behind a pending read. Each has a failing-then-passing +regression. The original affected package suites remain required. + +Run the affected suites and source gates: + +```sh +bun test packages/lina-core/test packages/lina-runtime/test packages/lina-codex/test packages/lina-web/test +bun run typecheck +bun run lint +bun run ci:validate +bun run ci:build +``` + +Before external qualification, obtain approval for a named provider/model and exactly +one generation plus one reference edit. Use a separate temporary Lina state root and +workspace, an owned test conversation, and the configured external ima2 runtime. +Record the exact runtime/provider/model, two upstream request IDs, attachment hashes, +observed browser output, and restart/deduplication results. A passing fake HTTP test, +model catalog, or source build must never be reported as a real provider call. + +Local browser harness: + +```sh +bun packages/lina-web/scripts/qa-image-engine.ts +``` + +Open the printed URL and send `generate`, then `edit`. This uses the real Lina +Codex adapter with synthetic RPC, a local fake ima2 server, isolated state and +320×200 blue/green PNG fixtures. `generate fail` exercises failure. No external +provider call occurs. The harness omits the unrelated fleet/task API; browser +acceptance stubs that sidebar to an empty task list. Browser checks confirmed both +images loaded, the original attachment stayed unchanged, refresh restored both, +and a 390px viewport had no horizontal overflow. A separate integration test +restarts the actual app and reuses its Codex notice journal and attachment files. + +Local verification on `codex/image-engine`, based on `dev` at `e41ce13`: + +| Check | Result | +| --- | --- | +| `bun test` | Exit 0; 1,429 passed, 0 failed; 6,485 assertions across 196 files | +| `bun run typecheck` | Exit 0; root and browser TypeScript | +| `bun run lint` | Exit 0; 5 existing warnings, no errors | +| `bun run ci:validate` | Exit 0; workflow, setup, forms and contribution links | +| `bun run ci:build` | Exit 0; runtime assets and read-only CLI smoke | +| Independent code review | PASS after four reproduced findings were repaired | +| Browser | Synthetic generation/edit, image bytes loaded, reload restored both; desktop and mobile checked | + +These local results were collected before PR publication. They do not certify a +merge or deployment. +The isolated fake Codex driver and fake ima2 server prove adapter wiring and +persistence, not real model reasoning or provider generation quality. The separately +approved provider qualification is recorded below. + +UI evidence: [desktop](009_ima2_image_engine_desktop.png) and +[390px mobile](009_ima2_image_engine_mobile.png). These screenshots show the local +synthetic harness described above, not external generation. The blue and green +rectangles are deterministic fixtures created by the included QA script; their +only inputs are dimensions and color values. The screenshots contain Lina's UI and +synthetic test messages, with no third-party image inputs or private conversation +data, and are contributed under the repository's Apache-2.0 license. + +External generation: **PASS for OAuth / gpt-5.6-luna: one creation and one reference edit.** +Push, merge and deployment remain separately authorized actions. + + +### ChatGPT connection qualification (2026-09-07) + +After the owner authorized account connection, ima2-gen 3.14.0 was installed in a +separate local runtime directory. The npm package reports the same `gitHead` as the +pinned source. Its offline installation doctor passed all eight checks. + +The supported `ima2 login` path recognized the existing file-backed Codex ChatGPT +session. No new browser login or manual credential copy was needed. ima2's OAuth +proxy loaded the account's model list from the authenticated Codex endpoint, then +`/api/oauth/status` returned `ready`. The Lina `Ima2Client` successfully parsed the +actual ima2 catalog. A controlled restart retained the configuration and returned +to the same ready connection. + +The configured image catalog and the account's model list are different surfaces; +`oauth/gpt-5.6-luna` is present in both. Connection qualification itself sent no +image-generation request; the later approved generation is recorded separately +below. Automatic startup at login/reboot and deployment into the user's running +Lina installation were not configured. + + +### Authorized live image qualification (2026-09-07) + +The owner approved exactly one creation and one reference edit through +`oauth/gpt-5.6-luna`. Both requests succeeded against the separately installed +ima2-gen 3.14.0 runtime. Each upstream history entry reports 29.9 seconds. + +| Operation | Upstream request ID | Result | +| --- | --- | --- | +| Create | `da6bbe17-b55c-4f56-94a3-a70b0df30530` | Blue robot with an orange hat; PNG, 1254×1254, 898,621 bytes | +| Reference edit | `8dcde191-3084-4bb8-9c82-54595b56e402` | Same robot with a purple hat; PNG, 1254×1254, 946,544 bytes | + +The edit transmitted the original managed bytes. Its reference SHA-256 equals +`2a20f93835c57013a16af88119ac88e959cb6c85cd7c1842c6aed6aa111958b6`, the stored original +hash. The edited result hash is +`6c535dabc0478ecfe902a7dbb6759d603417de2663cb1f46e99c95f4f936c376`. The original file +remained unchanged. Actual image dimensions were decoded from the returned PNGs; +ima2's history reports the nominal `1024x1024` request size, which is not a byte-level +dimension guarantee on this route. + +The real Lina Codex adapter, image tools, attachment store, native notice journal, +HTTP preview and existing web renderer handled the results. The conversation RPC +was a deterministic test driver, so this proves the image integration path, not a +real conversation model's autonomous tool selection. External image requests were +real and constrained before dispatch to the approved provider/model, exact prompts, +one image per request and two requests total. No extra generation or provider +fallback occurred. + +Both images were observed in the same originating test conversation. Restarting +Lina restored the same session, two attachment hashes and exactly two completion +notices. After the first upstream terminal snapshot expired, the adapter also +recovered its completed result from the exact request-filtered history without +resubmission. Subsequent browser reload and a 390px mobile viewport displayed the +saved images. Browser proof waits for image decoding and painting, not only network +completion; the unrelated task sidebar was stubbed empty because this isolated +harness has no fleet API. The test app was stopped and its artifacts retained. + +Failure, cancellation and uncertain-request negatives remain covered by local +fixtures. No extra paid call was made to induce those failures. Other providers, +models, mask/video/batch operations, and production deployment remain unqualified. +The local qualification receipt and original/edited image files are retained in +`.codexclaw/evidence/image-engine-live-20260907/` (gitignored). diff --git a/docs/plans/platform/009_ima2_image_engine_desktop.png b/docs/plans/platform/009_ima2_image_engine_desktop.png new file mode 100644 index 0000000..03d3237 Binary files /dev/null and b/docs/plans/platform/009_ima2_image_engine_desktop.png differ diff --git a/docs/plans/platform/009_ima2_image_engine_mobile.png b/docs/plans/platform/009_ima2_image_engine_mobile.png new file mode 100644 index 0000000..97ddf9f Binary files /dev/null and b/docs/plans/platform/009_ima2_image_engine_mobile.png differ diff --git a/packages/lina-core/src/attachments/store.ts b/packages/lina-core/src/attachments/store.ts index 4f6f602..fb6a368 100644 --- a/packages/lina-core/src/attachments/store.ts +++ b/packages/lina-core/src/attachments/store.ts @@ -65,18 +65,42 @@ export class AttachmentStore { return isDeepStrictEqual(this.binding, binding); } - put(name: string, bytes: Uint8Array): AttachmentMetadata { + /** A caller-owned stable ID makes a generated artifact import replayable. */ + put( + name: string, + bytes: Uint8Array, + id: string = randomUUID(), + ): AttachmentMetadata { this.assertOpen(); + validateId(id); const safeName = validateName(name); const mime = inspectContent(safeName, bytes); - if (this.records.size + this.orphanCount >= ATTACHMENT_MAX_FILES) + const previous = this.records.get(id); + if (previous) { + if (previous.name !== safeName || previous.sha256 !== hash(bytes)) + throw new AttachmentError("invalid-request", "Attachment ID conflict"); + return this.get(id); + } + const target = join(this.filesDirectory, id); + const retained = lstatSync(target, { throwIfNoEntry: false }); + if (retained) { + checkedRegular(target); + if (hash(readRegular(target)) !== hash(bytes)) + throw new AttachmentError("invalid-request", "Attachment ID conflict"); + } + if ( + !retained && + this.records.size + this.orphanCount >= ATTACHMENT_MAX_FILES + ) throw new AttachmentError("quota", "Attachment file quota is exhausted"); - if (this.totalBytes + bytes.byteLength > ATTACHMENT_MAX_TOTAL_BYTES) + if ( + !retained && + this.totalBytes + bytes.byteLength > ATTACHMENT_MAX_TOTAL_BYTES + ) throw new AttachmentError( "quota", "Attachment storage quota is exhausted", ); - const id = randomUUID(); const metadata: AttachmentMetadata = { id, name: safeName, @@ -85,15 +109,17 @@ export class AttachmentStore { sha256: hash(bytes), }; const temporary = join(this.filesDirectory, `.upload-${randomUUID()}.tmp`); - const target = join(this.filesDirectory, id); try { - writeExclusive(temporary, bytes); - renameSync(temporary, target); - fsyncDirectory(this.filesDirectory); + if (!retained) { + writeExclusive(temporary, bytes); + renameSync(temporary, target); + fsyncDirectory(this.filesDirectory); + } const next = [...this.records.values(), metadata]; this.writeManifest(next); this.records.set(id, metadata); - this.totalBytes += metadata.size; + if (retained) this.orphanCount--; + else this.totalBytes += metadata.size; return { ...metadata }; } catch (error) { this.poisoned = true; diff --git a/packages/lina-core/test/attachments.test.ts b/packages/lina-core/test/attachments.test.ts index b4cd4a0..d3fa8f5 100644 --- a/packages/lina-core/test/attachments.test.ts +++ b/packages/lina-core/test/attachments.test.ts @@ -38,6 +38,25 @@ function fixture() { return { dir, binding }; } +test("a stable artifact ID imports once across restart and rejects conflicting bytes", () => { + const { dir, binding } = fixture(); + const id = "11111111-1111-4111-8111-111111111111"; + const bytes = new TextEncoder().encode("immutable result"); + const first = new AttachmentStore(dir, binding); + const receipt = first.put("result.txt", bytes, id); + expect(receipt.id).toBe(id); + first.close(); + const second = new AttachmentStore(dir, binding); + expect(second.put("result.txt", bytes, id)).toEqual(receipt); + expect(readdirSync(join(dir, "attachments/files"))).toEqual([id]); + expect(() => + second.put("result.txt", new TextEncoder().encode("changed"), id), + ).toThrow("conflict"); + expect(() => second.put("different.txt", bytes, id)).toThrow("conflict"); + expect(second.bytes(id)).toEqual(bytes); + second.close(); +}); + test("stores immutable typed bytes and reads UTF-8 in surrogate-safe pages", () => { const { dir, binding } = fixture(); const store = new AttachmentStore(dir, binding); @@ -193,3 +212,24 @@ test("retains unreferenced crash files without adopting them or blocking committ ); reopened.close(); }); + +test("stable import recovers its own pre-manifest crash file without overwriting a conflict", () => { + const { dir, binding } = fixture(); + const store = new AttachmentStore(dir, binding); + const id = "22222222-2222-4222-8222-222222222222"; + const bytes = new TextEncoder().encode("retained result"); + writeFileSync(join(dir, "attachments/files", id), bytes); + store.close(); + const recovered = new AttachmentStore(dir, binding); + expect(() => + recovered.put("result.txt", new TextEncoder().encode("different"), id), + ).toThrow("conflict"); + expect(readFileSync(join(dir, "attachments/files", id))).toEqual( + Buffer.from(bytes), + ); + expect(recovered.put("result.txt", bytes, id).id).toBe(id); + recovered.close(); + const reopened = new AttachmentStore(dir, binding); + expect(reopened.bytes(id)).toEqual(bytes); + reopened.close(); +}); diff --git a/packages/lina-runtime/src/fleet/codex-fleet.ts b/packages/lina-runtime/src/fleet/codex-fleet.ts index 32eb0f3..ec4c9d6 100644 --- a/packages/lina-runtime/src/fleet/codex-fleet.ts +++ b/packages/lina-runtime/src/fleet/codex-fleet.ts @@ -214,6 +214,14 @@ async function startUnlocked( ].filter(existsSync); const app = await startPersistentApp({ ...appOptions, + imageEngine: { + ...(env["LINA_IMA2_URL"] + ? { baseUrl: env["LINA_IMA2_URL"] } + : {}), + serverFile: + env["LINA_IMA2_SERVER_FILE"] ?? + join(home, ".ima2", "server.json"), + }, engine: createCodexEngine({ services: hub.createContextServices( getSettings, diff --git a/packages/lina-runtime/src/images/client-contract.ts b/packages/lina-runtime/src/images/client-contract.ts new file mode 100644 index 0000000..a6bba72 --- /dev/null +++ b/packages/lina-runtime/src/images/client-contract.ts @@ -0,0 +1,324 @@ +import { z } from "zod"; +import { + ATTACHMENT_MAX_BYTES, + inspectContent, +} from "../../../lina-core/src/attachments/validation.ts"; +import { + Ima2Error, + type Ima2ImageMime, + type Ima2Job, + type Ima2Lane, + type Ima2Result, +} from "./client-types.ts"; + +export const IMA2_VERSION = "3.14.0"; +export const IMAGE_MAX_BYTES = ATTACHMENT_MAX_BYTES; +export const requestIdSchema = z.string().regex(/^[A-Za-z0-9._:-]{1,128}$/); +const identifier = z.string().min(1).max(255); +const filenameSchema = z + .string() + .min(5) + .max(255) + .regex(/\.(?:png|jpe?g)$/i) + .refine( + (value) => + !value.startsWith(".") && + !/[\\/%?#:]/u.test(value) && + Buffer.byteLength(value, "utf8") <= 255 && + [...value].every((character) => { + const code = character.codePointAt(0) ?? 0; + return code > 0x1f && !(code >= 0x7f && code <= 0x9f); + }), + ); +export const resultSchema = z.strictObject({ + requestId: requestIdSchema, + filename: filenameSchema, +}); +export const submitSchema = z.strictObject({ + requestId: requestIdSchema, + provider: identifier, + model: identifier, + prompt: z.string().min(1).max(32_000), + reference: z + .strictObject({ + bytes: z.instanceof(Uint8Array), + mime: z.enum(["image/png", "image/jpeg"]), + }) + .optional(), +}); +export const healthSchema = z.object({ + ok: z.literal(true), + version: z.string(), +}); +export const cancelSchema = z.object({ + requestId: requestIdSchema, + active: z.boolean(), + aborted: z.boolean(), +}); +export const acceptedSchema = z.object({ + requestId: requestIdSchema, + async: z.literal(true), +}); +export const replaySchema = z.object({ + requestId: requestIdSchema, + filename: filenameSchema, + provider: identifier, + model: identifier, + idempotentReplay: z.literal(true), +}); + +// Known /api/generate dispatch branches at 36aa6fc. Unknown providers fall +// through to oauth upstream. agy ignores rawModel, so cannot honor selection. +const GENERATE_PROVIDERS = new Set([ + "oauth", + "api", + "grok", + "grok-api", + "gemini-api", + "atlascloud", + "minimax", + "nai", + "comfy", +]); +const modelSchema = z.object({ + id: identifier, + label: z.string().max(255), + capabilities: z.object({ inputRoles: z.array(z.string()) }).optional(), +}); +const laneSchema = z.object({ + status: z.enum(["ready", "locked", "disconnected", "key-missing"]), + models: z.object({ image: z.array(modelSchema) }), + surfaces: z + .object({ + generate: z + .object({ supported: z.boolean(), references: z.boolean() }) + .optional(), + }) + .optional(), +}); +const catalogSchema = z.object({ + ok: z.literal(true), + lanes: z.record(identifier, laneSchema), +}); + +export function parse( + schema: z.ZodType, + value: unknown, + code: + | "INVALID_INPUT" + | "INVALID_RESULT" + | "INVALID_RESPONSE" = "INVALID_RESPONSE", +): T { + const result = schema.safeParse(value); + if (!result.success) + throw new Ima2Error(code, "ima2 data did not match the required contract"); + return result.data; +} + +export function catalogLanes(value: unknown): Ima2Lane[] { + return Object.entries(parse(catalogSchema, value).lanes).map( + ([provider, lane]) => { + const surface = lane.surfaces?.generate; + const generate = + GENERATE_PROVIDERS.has(provider) && surface?.supported === true; + return { + provider, + status: lane.status, + models: lane.models.image.map((model) => ({ + id: model.id, + label: model.label, + generate, + edit: + generate && + surface?.references === true && + model.capabilities?.inputRoles.includes("image_references") === + true, + })), + }; + }, + ); +} + +export function imageMime( + bytes: Uint8Array, + mime: string, + maxBytes: number, +): Ima2ImageMime { + if (bytes.byteLength > maxBytes) + throw new Ima2Error( + "BODY_TOO_LARGE", + "Image exceeds the configured byte limit", + ); + if (mime !== "image/png" && mime !== "image/jpeg") + throw new Ima2Error( + "INVALID_IMAGE", + "Only PNG and JPEG images are supported", + ); + try { + // Reuse managed attachment container, checksum, and dimension validation. + inspectContent(mime === "image/png" ? "image.png" : "image.jpg", bytes); + } catch { + throw new Ima2Error( + "INVALID_IMAGE", + "Image bytes do not match a supported image container", + ); + } + return mime; +} + +const jobRowSchema = z.object({ + requestId: requestIdSchema, + kind: z.string(), + phase: z.string(), +}); +const terminalSchema = jobRowSchema.extend({ + status: z.string(), + errorCode: z.string().optional(), + meta: z.object({ + filenames: z.array(z.string()).optional(), + imageCount: z.number().optional(), + }), +}); +const inflightSchema = z.object({ + jobs: z.array(z.unknown()), + terminalJobs: z.array(z.unknown()), +}); +const historySchema = z.object({ + items: z.array(z.object({ requestId: z.string(), filename: z.string() })), + nextCursor: z.unknown().optional(), +}); +const identitySchema = z.object({ requestId: z.string() }); + +function matching(rows: unknown[], requestId: string): unknown[] { + return rows.filter( + (row) => parse(identitySchema, row).requestId === requestId, + ); +} + +function completed(requestId: string, filenames: string[]): Ima2Job { + if (filenames.length !== 1) + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 must return exactly one image", + ); + const result: Ima2Result = parse(resultSchema, { + requestId, + filename: filenames[0], + }); + return { requestId, state: "completed", result }; +} + +function failed( + requestId: string, + state: "failed" | "cancelled" | "timed_out", +): Ima2Job { + const messages = { + failed: { code: "GENERATION_FAILED", message: "ima2 generation failed" }, + cancelled: { + code: "GENERATION_CANCELED", + message: + "ima2 marked the job cancelled; provider completion is not guaranteed", + }, + timed_out: { + code: "JOB_TRACKING_TIMEOUT", + message: "ima2 tracking expired; provider completion is unknown", + }, + }; + return { requestId, state, error: messages[state] }; +} + +export function inflightJob( + value: unknown, + requestId: string, +): Ima2Job | undefined { + const data = parse(inflightSchema, value); + const active = matching(data.jobs, requestId); + const terminal = matching(data.terminalJobs, requestId); + if (active.length + terminal.length > 1) + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 returned conflicting job identities", + ); + if (terminal.length === 1) { + const row = parse(terminalSchema, terminal[0]); + // abortJob creates this tombstone even when no generation is known. + // It is not cancellation evidence; let read() check saved history. + if (row.kind === "unknown" && row.status === "canceled") return undefined; + if (row.kind !== "classic") + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 job is not a classic image request", + ); + if (row.errorCode === "JOB_TRACKING_TIMEOUT") + return failed(requestId, "timed_out"); + if (["done", "completed", "complete"].includes(row.status)) { + if (row.meta.imageCount !== undefined && row.meta.imageCount !== 1) + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 returned an unexpected image count", + ); + return completed(requestId, row.meta.filenames ?? []); + } + if (["canceled", "cancelled"].includes(row.status)) + return failed(requestId, "cancelled"); + if (["error", "failed"].includes(row.status)) + return failed(requestId, "failed"); + return { requestId, state: "unknown" }; + } + if (active.length === 0) return undefined; + const row = parse(jobRowSchema, active[0]); + if (row.kind !== "classic") + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 job is not a classic image request", + ); + if ( + [ + "queued", + "validating", + "preparing", + "planning", + "provider-queued", + ].includes(row.phase) + ) + return { requestId, state: "queued" }; + if ( + [ + "streaming", + "partial", + "uploading", + "provider-running", + "provider-poll", + "polling", + "progress", + "submitted", + ].includes(row.phase) + ) + return { requestId, state: "running" }; + if ( + ["decoding", "downloading", "media-processing", "persisting"].includes( + row.phase, + ) + ) + return { requestId, state: "post_processing" }; + return { requestId, state: "unknown" }; +} + +export function historyJob(value: unknown, requestId: string): Ima2Job { + const data = parse(historySchema, value); + if (data.items.some((item) => item.requestId !== requestId)) + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 history returned a different request identity", + ); + if (data.items.length === 0) return { requestId, state: "unknown" }; + if (data.nextCursor) + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 history returned more than one image", + ); + return completed( + requestId, + data.items.map((item) => item.filename), + ); +} diff --git a/packages/lina-runtime/src/images/client-http.ts b/packages/lina-runtime/src/images/client-http.ts new file mode 100644 index 0000000..768a685 --- /dev/null +++ b/packages/lina-runtime/src/images/client-http.ts @@ -0,0 +1,294 @@ +import { open } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { z } from "zod"; +import { parse } from "./client-contract.ts"; +import { type Ima2ClientOptions, Ima2Error } from "./client-types.ts"; + +const DEFAULT_TIMEOUT_MS = 15_000; +const MAX_JSON_BYTES = 4 * 1024 * 1024; +const MAX_DISCOVERY_BYTES = 64 * 1024; +const discoverySchema = z.object({ backend: z.object({ url: z.string() }) }); + +function origin(value: string): string { + try { + if (!/^https?:\/\//i.test(value) || /[\s\\?#@]/.test(value)) + throw new Error(); + const url = new URL(value); + if (url.pathname !== "/" || url.username || url.password) throw new Error(); + return url.origin; + } catch { + throw new Ima2Error( + "INVALID_SERVER_URL", + "ima2 server must be an HTTP(S) origin without credentials or path", + ); + } +} + +async function discover(serverFile: string): Promise { + try { + const file = await open(serverFile, "r"); + let value: unknown; + try { + const stat = await file.stat(); + if (!stat.isFile() || stat.size > MAX_DISCOVERY_BYTES) throw new Error(); + const buffer = Buffer.alloc(MAX_DISCOVERY_BYTES + 1); + const { bytesRead } = await file.read(buffer, 0, buffer.length, 0); + if (bytesRead > MAX_DISCOVERY_BYTES) throw new Error(); + value = JSON.parse(buffer.subarray(0, bytesRead).toString("utf8")); + } finally { + await file.close(); + } + const base = origin(parse(discoverySchema, value).backend.url); + if (!["localhost", "127.0.0.1", "[::1]"].includes(new URL(base).hostname)) + throw new Error(); + return base; + } catch { + throw new Ima2Error( + "DISCOVERY_UNAVAILABLE", + "No usable local ima2 server discovery; configure a server origin or start ima2 separately", + ); + } +} + +function aborted(signal: AbortSignal): Ima2Error { + return new Ima2Error( + signal.reason instanceof Ima2Error && signal.reason.code === "TIMEOUT" + ? "TIMEOUT" + : "ABORTED", + "ima2 request stopped before a response was confirmed", + ); +} + +async function untilAbort( + work: Promise, + signal: AbortSignal, +): Promise { + let onAbort = () => {}; + const stop = new Promise((_, reject) => { + onAbort = () => reject(aborted(signal)); + if (signal.aborted) onAbort(); + else signal.addEventListener("abort", onAbort, { once: true }); + }); + try { + return await Promise.race([work, stop]); + } finally { + signal.removeEventListener("abort", onAbort); + } +} + +async function boundedBody( + response: Response, + limit: number, + signal: AbortSignal, +): Promise { + const length = response.headers.get("content-length"); + if (length !== null && (!/^\d+$/.test(length) || Number(length) > limit)) + throw new Ima2Error( + "BODY_TOO_LARGE", + "ima2 response exceeds the byte limit", + ); + if (!response.body) + throw new Ima2Error("INVALID_RESPONSE", "ima2 response body is missing"); + const reader = response.body.getReader(); + const chunks: Uint8Array[] = []; + let total = 0; + try { + for (;;) { + const { done, value } = await untilAbort(reader.read(), signal); + if (done) break; + total += value.byteLength; + if (total > limit) + throw new Ima2Error( + "BODY_TOO_LARGE", + "ima2 response exceeds the byte limit", + ); + chunks.push(value); + } + } finally { + void reader.cancel().catch(() => {}); // Cleanup cannot replace the safe boundary error. + reader.releaseLock(); + } + const bytes = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { + bytes.set(chunk, offset); + offset += chunk.byteLength; + } + return bytes; +} + +function httpError(status: number): Ima2Error { + // A 409 may refer to an existing paid generation, not a rejected job. + const outcome = status < 500 && status !== 409 ? "rejected" : "unknown"; + if (status === 401 || status === 403) + return new Ima2Error( + "ACCESS_DENIED", + "ima2 access denied; configure access in ima2", + outcome, + status, + ); + if (status === 409) + return new Ima2Error( + "CONFLICT", + "ima2 request identity is already in use or conflicts with a prior request", + outcome, + status, + ); + if (status === 429) + return new Ima2Error( + "RATE_LIMITED", + "ima2 rejected the request due to a rate or capacity limit", + outcome, + status, + ); + return new Ima2Error( + "HTTP_ERROR", + "ima2 returned an unsuccessful HTTP response", + outcome, + status, + ); +} + +export class Ima2Http { + readonly #options: Ima2ClientOptions; + readonly #timeoutMs: number; + #origin: Promise | undefined; + + constructor(options: Ima2ClientOptions) { + this.#options = { ...options }; + this.#timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS; + if (!Number.isSafeInteger(this.#timeoutMs) || this.#timeoutMs < 1) + throw new Ima2Error( + "INVALID_INPUT", + "ima2 timeout must be a positive integer", + ); + } + + async baseUrl(signal?: AbortSignal): Promise { + if (signal?.aborted) throw aborted(signal); + if (!this.#origin) { + const selected = this.#options.baseUrl; + const pending = + selected === undefined + ? discover( + this.#options.serverFile ?? + join(homedir(), ".ima2", "server.json"), + ) + : Promise.resolve().then(() => origin(selected)); + this.#origin = pending; + void pending.catch(() => { + if (this.#origin === pending) this.#origin = undefined; + }); + } + return signal ? untilAbort(this.#origin, signal) : this.#origin; + } + + async json( + path: string, + init: RequestInit = {}, + ): Promise<{ status: number; value: unknown }> { + const response = await this.request(path, init, MAX_JSON_BYTES, true); + try { + return { + status: response.status, + value: JSON.parse( + new TextDecoder("utf-8", { fatal: true }).decode(response.bytes), + ) as unknown, + }; + } catch { + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 returned malformed JSON", + init.method === "POST" || init.method === "DELETE" + ? "unknown" + : "rejected", + ); + } + } + + async request( + path: string, + init: RequestInit, + limit: number, + json = false, + ): Promise<{ bytes: Uint8Array; mime: string; status: number }> { + const controller = new AbortController(); + const parent = init.signal; + const onAbort = () => controller.abort(); + if (parent?.aborted) throw aborted(parent); + parent?.addEventListener("abort", onAbort, { once: true }); + const timer = setTimeout( + () => + controller.abort(new Ima2Error("TIMEOUT", "ima2 request timed out")), + this.#timeoutMs, + ); + let sent = false; + let response: Response | undefined; + try { + const base = await this.baseUrl(controller.signal); + const target = new URL(path, base); + if (!path.startsWith("/") || target.origin !== base) + throw new Ima2Error( + "INVALID_SERVER_URL", + "ima2 request must stay on the selected origin", + ); + if (controller.signal.aborted) throw aborted(controller.signal); + sent = true; + response = await untilAbort( + (this.#options.fetch ?? fetch)(target.href, { + ...init, + signal: controller.signal, + redirect: "manual", + credentials: "omit", + }), + controller.signal, + ); + if ( + response.redirected || + (response.status >= 300 && response.status < 400) + ) + throw new Ima2Error( + "REDIRECT_REFUSED", + "ima2 redirects are not allowed", + ); + if (!response.ok) throw httpError(response.status); + const mime = + response.headers + .get("content-type") + ?.split(";")[0] + ?.trim() + .toLowerCase() ?? ""; + if (json && mime !== "application/json") + throw new Ima2Error("INVALID_RESPONSE", "ima2 response was not JSON"); + return { + bytes: await boundedBody(response, limit, controller.signal), + mime, + status: response.status, + }; + } catch (error) { + const safe = controller.signal.aborted + ? aborted(controller.signal) + : error instanceof Ima2Error + ? error + : new Ima2Error("NETWORK_ERROR", "ima2 transport failed"); + const mutation = init.method === "POST" || init.method === "DELETE"; + const definiteRejection = + safe.status !== undefined && + safe.status >= 400 && + safe.status < 500 && + safe.status !== 409; + throw new Ima2Error( + safe.code, + safe.message, + mutation && sent && !definiteRejection ? "unknown" : safe.outcome, + safe.status, + ); + } finally { + clearTimeout(timer); + parent?.removeEventListener("abort", onAbort); + if (response?.body && !response.body.locked) + void response.body.cancel().catch(() => {}); + } + } +} diff --git a/packages/lina-runtime/src/images/client-types.ts b/packages/lina-runtime/src/images/client-types.ts new file mode 100644 index 0000000..eb14f9e --- /dev/null +++ b/packages/lina-runtime/src/images/client-types.ts @@ -0,0 +1,95 @@ +export type Ima2ImageMime = "image/png" | "image/jpeg"; +export type Ima2Result = { requestId: string; filename: string }; +export type Ima2Failure = { code: string; message: string }; +export type Ima2Job = { + requestId: string; + state: + | "queued" + | "running" + | "post_processing" + | "completed" + | "failed" + | "cancelled" + | "timed_out" + | "unknown"; + result?: Ima2Result; + error?: Ima2Failure; +}; +/** Registry acknowledgement, not a guarantee that the provider stopped or refunded. */ +export type Ima2Cancellation = { + requestId: string; + active: boolean; + aborted: boolean; +}; +export type Ima2SubmitInput = { + requestId: string; + provider: string; + model: string; + prompt: string; + reference?: { bytes: Uint8Array; mime: Ima2ImageMime }; +}; +export type Ima2ClientOptions = { + baseUrl?: string; + serverFile?: string; + fetch?: (url: string, init: RequestInit) => Promise; + timeoutMs?: number; + /** May lower, but never raise, the managed attachment limit of 2 MiB. */ + maxImageBytes?: number; +}; +export type Ima2Lane = { + provider: string; + status: "ready" | "locked" | "disconnected" | "key-missing"; + models: { id: string; label: string; generate: boolean; edit: boolean }[]; +}; +export type Ima2Connection = { + /** Canonical origin; fixed for this client instance after discovery. */ + baseUrl: string; + version: string; + /** Catalog readiness only; does not prove a successful provider call. */ + ready: boolean; + lanes: Ima2Lane[]; +}; +export interface Ima2ClientPort { + connect(signal?: AbortSignal): Promise; + submit(input: Ima2SubmitInput, signal?: AbortSignal): Promise; + read(requestId: string, signal?: AbortSignal): Promise; + cancel(requestId: string, signal?: AbortSignal): Promise; + download( + result: Ima2Result, + signal?: AbortSignal, + ): Promise<{ bytes: Uint8Array; mime: Ima2ImageMime }>; +} + +export type Ima2ErrorCode = + | "INVALID_INPUT" + | "INVALID_SERVER_URL" + | "DISCOVERY_UNAVAILABLE" + | "UNSUPPORTED_VERSION" + | "INVALID_RESPONSE" + | "LANE_UNAVAILABLE" + | "MODEL_UNAVAILABLE" + | "UNSUPPORTED_OPERATION" + | "INVALID_IMAGE" + | "INVALID_RESULT" + | "BODY_TOO_LARGE" + | "ACCESS_DENIED" + | "HTTP_ERROR" + | "CONFLICT" + | "RATE_LIMITED" + | "REDIRECT_REFUSED" + | "NETWORK_ERROR" + | "ABORTED" + | "TIMEOUT"; + +/** No upstream body, URL, prompt, credential, or raw exception is retained. */ +export class Ima2Error extends Error { + constructor( + readonly code: Ima2ErrorCode, + message: string, + readonly outcome: "rejected" | "unknown" = "rejected", + readonly status?: number, + ) { + super(message); + this.name = "Ima2Error"; + } +} diff --git a/packages/lina-runtime/src/images/client.ts b/packages/lina-runtime/src/images/client.ts new file mode 100644 index 0000000..3b18792 --- /dev/null +++ b/packages/lina-runtime/src/images/client.ts @@ -0,0 +1,242 @@ +import { + acceptedSchema, + cancelSchema, + catalogLanes, + healthSchema, + historyJob, + IMA2_VERSION, + IMAGE_MAX_BYTES, + imageMime, + inflightJob, + parse, + replaySchema, + requestIdSchema, + resultSchema, + submitSchema, +} from "./client-contract.ts"; +import { Ima2Http } from "./client-http.ts"; +import { + type Ima2Cancellation, + type Ima2ClientOptions, + type Ima2ClientPort, + type Ima2Connection, + Ima2Error, + type Ima2ImageMime, + type Ima2Job, + type Ima2Result, + type Ima2SubmitInput, +} from "./client-types.ts"; + +export type { + Ima2Cancellation, + Ima2ClientOptions, + Ima2ClientPort, + Ima2Connection, + Ima2ErrorCode, + Ima2Failure, + Ima2ImageMime, + Ima2Job, + Ima2Lane, + Ima2Result, + Ima2SubmitInput, +} from "./client-types.ts"; +export { Ima2Error } from "./client-types.ts"; + +/** + * HTTP contracts verified against ima2-gen 36aa6fcea62f753d858d20c4823e04922951c954. + * No engine lifecycle, polling loop, automatic retry, or provider fallback. + * Persist baseUrl + requestId outside this adapter before submitting. Upstream + * idempotency expires after 24h; absent records must never cause blind replay. + */ +export class Ima2Client implements Ima2ClientPort { + readonly #http: Ima2Http; + readonly #maxImageBytes: number; + #connection: Ima2Connection | undefined; + + constructor(options: Ima2ClientOptions = {}) { + this.#http = new Ima2Http(options); + this.#maxImageBytes = options.maxImageBytes ?? IMAGE_MAX_BYTES; + if ( + !Number.isSafeInteger(this.#maxImageBytes) || + this.#maxImageBytes < 1 || + this.#maxImageBytes > IMAGE_MAX_BYTES + ) + throw new Ima2Error( + "INVALID_INPUT", + "Image byte limit must be between 1 and 2 MiB", + ); + } + + async connect(signal?: AbortSignal): Promise { + this.#connection = undefined; + const baseUrl = await this.#http.baseUrl(signal); + const init = signal ? { signal } : {}; + const health = parse( + healthSchema, + (await this.#http.json("/api/health", init)).value, + ); + if (health.version !== IMA2_VERSION) + throw new Ima2Error( + "UNSUPPORTED_VERSION", + "This adapter requires ima2 3.14.0; qualify other versions before use", + ); + const lanes = catalogLanes( + (await this.#http.json("/api/models", init)).value, + ); + this.#connection = { + baseUrl, + version: health.version, + lanes, + ready: lanes.some( + (lane) => + lane.status === "ready" && + lane.models.some((model) => model.generate), + ), + }; + return structuredClone(this.#connection); + } + + async submit(input: Ima2SubmitInput, signal?: AbortSignal): Promise { + const value = parse(submitSchema, input, "INVALID_INPUT"); + const reference = value.reference; + if (reference) + imageMime(reference.bytes, reference.mime, this.#maxImageBytes); + const references = reference + ? [ + `data:${reference.mime};base64,${Buffer.from(reference.bytes).toString("base64")}`, + ] + : []; + const connection = this.#connection ?? (await this.connect(signal)); + const lane = connection.lanes.find( + (lane) => lane.provider === value.provider, + ); + if (lane?.status !== "ready") + throw new Ima2Error( + "LANE_UNAVAILABLE", + "Selected ima2 provider is unavailable; configure it in ima2", + ); + const model = lane.models.find((model) => model.id === value.model); + if (!model) + throw new Ima2Error( + "MODEL_UNAVAILABLE", + "Selected image model is absent from the ima2 catalog", + ); + if (!model.generate || (reference && !model.edit)) + throw new Ima2Error( + "UNSUPPORTED_OPERATION", + "Selected ima2 model does not support this adapter operation", + ); + const body = { + requestId: value.requestId, + provider: value.provider, + model: value.model, + prompt: value.prompt, + async: true, + n: 1, + references, + format: "png", + }; + const response = await this.#http.json("/api/generate", { + method: "POST", + headers: { + "Content-Type": "application/json", + "Idempotency-Key": value.requestId, + "X-Request-Id": value.requestId, + }, + body: JSON.stringify(body), + ...(signal ? { signal } : {}), + }); + try { + if (response.status === 202) { + const accepted = parse(acceptedSchema, response.value); + if (accepted.requestId !== value.requestId) throw new Error(); + return { requestId: value.requestId, state: "queued" }; + } + if (response.status !== 200) throw new Error(); + const replay = parse(replaySchema, response.value); + if ( + replay.requestId !== value.requestId || + replay.provider !== value.provider || + replay.model !== value.model + ) + throw new Error(); + return { + requestId: value.requestId, + state: "completed", + result: { requestId: value.requestId, filename: replay.filename }, + }; + } catch { + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 submission response did not confirm the requested identity and output", + "unknown", + ); + } + } + + async read(requestId: string, signal?: AbortSignal): Promise { + parse(requestIdSchema, requestId, "INVALID_INPUT"); + if (!this.#connection) await this.connect(signal); + const init = signal ? { signal } : {}; + const job = inflightJob( + (await this.#http.json("/api/inflight?includeTerminal=1", init)).value, + requestId, + ); + if (job) return job; + // Terminal snapshots expire, and unknown cancellation tombstones cannot + // prove a job's outcome. History retains identity with the saved file; + // no other job's "latest" is used. + return historyJob( + ( + await this.#http.json( + `/api/history?requestId=${encodeURIComponent(requestId)}&limit=2`, + init, + ) + ).value, + requestId, + ); + } + + async cancel( + requestId: string, + signal?: AbortSignal, + ): Promise { + parse(requestIdSchema, requestId, "INVALID_INPUT"); + if (!this.#connection) await this.connect(signal); + const response = await this.#http.json( + `/api/inflight/${encodeURIComponent(requestId)}`, + { method: "DELETE", ...(signal ? { signal } : {}) }, + ); + try { + const ack = parse(cancelSchema, response.value); + if (ack.requestId !== requestId) throw new Error(); + return ack; + } catch { + throw new Ima2Error( + "INVALID_RESPONSE", + "ima2 cancellation acknowledgement did not match the request", + "unknown", + ); + } + } + + async download( + result: Ima2Result, + signal?: AbortSignal, + ): Promise<{ bytes: Uint8Array; mime: Ima2ImageMime }> { + const value = parse(resultSchema, result, "INVALID_RESULT"); + if (!this.#connection) await this.connect(signal); + const response = await this.#http.request( + `/generated/${encodeURIComponent(value.filename)}`, + signal ? { signal } : {}, + this.#maxImageBytes, + ); + const mime = imageMime(response.bytes, response.mime, this.#maxImageBytes); + if ((/\.png$/i.test(value.filename) ? "image/png" : "image/jpeg") !== mime) + throw new Ima2Error( + "INVALID_IMAGE", + "ima2 image media type does not match its filename", + ); + return { bytes: response.bytes, mime }; + } +} diff --git a/packages/lina-runtime/src/images/jobs.ts b/packages/lina-runtime/src/images/jobs.ts new file mode 100644 index 0000000..db82141 --- /dev/null +++ b/packages/lina-runtime/src/images/jobs.ts @@ -0,0 +1,433 @@ +import { setTimeout as delay } from "node:timers/promises"; +import { + AttachmentError, + type AttachmentStore, +} from "../../../lina-core/src/attachments/store.ts"; +import { isImageMime } from "../../../lina-core/src/attachments/types.ts"; +import type { SessionPort } from "../sdk-port.ts"; +import type { Ima2Client } from "./client.ts"; +import { Ima2Error } from "./client-types.ts"; +import { + type ImageJob, + type ImageJobInput, + type ImageJobStore, + terminalImageState, +} from "./store.ts"; + +export type ImageClient = Pick< + Ima2Client, + "connect" | "submit" | "read" | "cancel" | "download" +>; +type Options = { + store: ImageJobStore; + attachments: AttachmentStore; + client: ImageClient; + notify: SessionPort["appendNotice"]; +}; +const POLL_MS = 1500; +const WAIT_MS = 5 * 60 * 1000; +const ERROR_GUIDANCE: Record = { + DISCOVERY_UNAVAILABLE: + "이미지 엔진이 연결되지 않았습니다. ima2를 시작하거나 연결 주소를 설정해주세요.", + ACCESS_DENIED: "ima2에서 인증 정보를 확인해주세요.", + LANE_UNAVAILABLE: + "선택한 이미지 서비스가 준비되지 않았습니다. ima2 연결 설정을 확인해주세요.", + MODEL_UNAVAILABLE: + "선택한 이미지 모델을 찾지 못했습니다. 이미지 모델 목록을 확인해주세요.", + UNSUPPORTED_OPERATION: "선택한 모델은 이 이미지 작업을 지원하지 않습니다.", + UNSUPPORTED_VERSION: "연결된 ima2 버전이 지원 범위와 다릅니다.", + BODY_TOO_LARGE: + "이미지가 첨부 크기 한도를 넘었습니다. 최대 2 MiB까지 저장할 수 있습니다.", + INVALID_IMAGE: + "이미지 파일을 검증하지 못했습니다. 결과를 표시하지 않았습니다.", +}; +function safeError(error: unknown): string { + if (error instanceof AttachmentError) + return `ATTACHMENT_${error.code.toUpperCase().replaceAll("-", "_")}: 이미지 결과를 저장하거나 검증하지 못했습니다.`; + // Upstream response bodies may contain credentials. Persist only our own code. + const code = + error && + typeof error === "object" && + "code" in error && + typeof error.code === "string" && + /^[A-Z_]{1,64}$/.test(error.code) + ? error.code + : "IMAGE_CONNECTION_FAILED"; + return `${code}: ${ERROR_GUIDANCE[code] ?? "이미지 연결 또는 작업 상태를 확인해주세요."}`; +} +export class ImageJobs { + private readonly active = new Map>(); + private readonly shutdown = new AbortController(); + private poll: Promise | undefined; + private notices: Promise | undefined; + private recoveryError: string | null = null; + constructor(private readonly options: Options) { + if (!options.attachments.isBoundTo(options.store.binding)) + throw Error("Image attachment binding differs"); + } + list() { + return this.options.store.list(); + } + get(id: string) { + return this.options.store.get(id); + } + async connect(signal?: AbortSignal) { + const connection = await this.options.client.connect(signal); + const verified = this.list() + .filter((j) => j.state === "completed") + .map((j) => ({ + provider: j.provider, + model: j.model, + version: j.runtimeVersion, + endpoint: j.endpoint, + completedAt: j.updatedAt, + })); + return { + ...connection, + ownership: "external" as const, + setupUrl: connection.baseUrl, + generationVerified: verified, + recoveryError: this.recoveryError, + }; + } + async start(input: ImageJobInput, signal?: AbortSignal): Promise { + this.shutdown.signal.throwIfAborted(); + signal?.throwIfAborted(); + const previous = this.list().find( + (j) => j.requestId === input.requestId && j.callId === input.callId, + ); + if (previous) return this.options.store.create(input); + if (this.list().some((j) => !terminalImageState(j.state))) + throw Error( + "이미지 작업이 아직 끝나지 않았습니다. 기존 작업을 확인해주세요.", + ); + let reference: + | { bytes: Uint8Array; mime: "image/png" | "image/jpeg" } + | undefined; + if (input.sourceArtifactId) { + const meta = this.options.attachments.get(input.sourceArtifactId); + if (!isImageMime(meta.mime)) + throw Error("편집할 이미지 첨부가 아닙니다."); + reference = { + bytes: this.options.attachments.bytes(meta.id), + mime: meta.mime, + }; + } + const created = this.options.store.create(input); + return this.exclusive(created.id, async () => { + let dispatched = false; + let admitted = false; + try { + const connection = await this.options.client.connect( + this.signal(signal), + ); + this.signal(signal).throwIfAborted(); + if (this.get(created.id).cancelRequested) { + return this.options.store.update(created.id, { + state: "cancelled", + error: null, + }); + } + this.options.store.update(created.id, { + endpoint: connection.baseUrl, + runtimeVersion: connection.version, + state: "submitting", + }); + dispatched = true; + const result = await this.options.client.submit( + { + requestId: created.id, + provider: input.provider, + model: input.model, + prompt: input.prompt, + ...(reference ? { reference } : {}), + }, + this.signal(signal), + ); + admitted = true; + await this.apply(created.id, result); + } catch (error) { + const rejected = + !admitted && + error instanceof Ima2Error && + error.outcome === "rejected"; + this.options.store.update(created.id, { + state: + dispatched && !rejected + ? this.get(created.id).cancelRequested + ? "cancelling" + : "uncertain" + : signal?.aborted + ? "cancelled" + : "failed", + error: safeError(error), + }); + } + return this.get(created.id); + }); + } + async reconcile(id: string): Promise { + return this.exclusive(id, async () => { + const job = this.get(id); + if (terminalImageState(job.state)) { + await this.flushNotices(); + return this.get(id); + } + try { + if ( + job.state === "prepared" || + (job.cancelRequested && job.endpoint === null) + ) { + this.options.store.update(id, { + state: job.cancelRequested ? "cancelled" : "failed", + error: + "생성 요청 전에 실행이 중단되었습니다. 새 요청으로 시작해주세요.", + }); + } else { + const connection = await this.options.client.connect( + this.shutdown.signal, + ); + if ( + connection.baseUrl !== job.endpoint || + connection.version !== job.runtimeVersion + ) + throw Error("Image runtime changed"); + let result = await this.options.client.read(id, this.shutdown.signal); + if ( + this.get(id).cancelRequested && + ["queued", "running", "post_processing"].includes(result.state) + ) { + await this.options.client.cancel(id, this.shutdown.signal); + result = await this.options.client.read(id, this.shutdown.signal); + } + await this.apply(id, result); + } + } catch (error) { + if (!terminalImageState(this.get(id).state)) + this.options.store.update(id, { + state: this.get(id).cancelRequested ? "cancelling" : "uncertain", + error: safeError(error), + }); + } + await this.flushNotices(); + return this.get(id); + }); + } + async cancel(id: string): Promise { + // Serialize the intent behind submission without coalescing it with a read. + const current = this.get(id); + if (terminalImageState(current.state)) return current; + this.options.store.update(id, { + cancelRequested: true, + state: "cancelling", + error: null, + }); + return this.exclusive(id, async () => { + const job = this.get(id); + if (terminalImageState(job.state)) return job; + if (job.endpoint === null) + return this.options.store.update(id, { + state: "cancelled", + error: null, + }); + try { + const connection = await this.options.client.connect( + this.shutdown.signal, + ); + if ( + connection.baseUrl !== job.endpoint || + connection.version !== job.runtimeVersion + ) + throw Error("Image runtime changed"); + await this.options.client.cancel(id, this.shutdown.signal); + await this.apply( + id, + await this.options.client.read(id, this.shutdown.signal), + ); + } catch (error) { + this.options.store.update(id, { + state: "cancelling", + error: safeError(error), + }); + } + await this.flushNotices(); + return this.get(id); + }); + } + async wait(id: string, signal?: AbortSignal): Promise { + const deadline = Date.now() + WAIT_MS; + while (!this.shutdown.signal.aborted) { + if (signal?.aborted) return this.cancel(id); + const job = await this.reconcile(id); + if ( + terminalImageState(job.state) || + job.state === "uncertain" || + Date.now() >= deadline + ) + return job; + try { + await delay(POLL_MS, undefined, { signal: this.signal(signal) }); + } catch { + if (signal?.aborted) return this.cancel(id); + break; + } + } + return this.get(id); + } + async recover(): Promise { + for (const job of this.list()) { + if (this.shutdown.signal.aborted) return; + await this.reconcile(job.id); + } + } + resume(): void { + if (this.poll) return; + this.poll = (async () => { + while (!this.shutdown.signal.aborted) { + try { + await this.recover(); + this.recoveryError = null; + } catch { + this.recoveryError = + "이미지 작업 복구를 완료하지 못했습니다. 저장소 상태를 확인해주세요."; + } + await delay(POLL_MS, undefined, { signal: this.shutdown.signal }); + } + })().catch((error) => { + if (!this.shutdown.signal.aborted) throw error; + }); + // Errors remain observable via close(), without an unhandled rejection. + void this.poll.catch(() => undefined); + } + async flushNotices(): Promise { + if (this.notices) return this.notices; + this.notices = Promise.resolve().then(async () => { + for (const job of this.list()) { + if (!terminalImageState(job.state) || job.deliveredEntryId) continue; + try { + const artifact = job.artifact; + if (artifact) this.options.attachments.get(artifact.id); + const suffix = `?sessionId=${encodeURIComponent(this.options.store.binding.sessionId)}`; + const text = artifact + ? `${job.sourceArtifactId ? "이미지를 수정했습니다." : "이미지를 만들었습니다."}\n\n![생성 이미지](/api/attachments/${artifact.id}/preview${suffix})\n\n[이미지 다운로드](/api/attachments/${artifact.id}${suffix})` + : job.state === "cancelled" + ? "이미지 생성을 취소했습니다." + : `${job.resultFilename ? "이미지는 생성됐지만 저장하지 못했습니다." : "이미지를 만들지 못했습니다."} ${job.error ?? "연결 상태를 확인해주세요."}`; + const entry = await this.options.notify( + { jobId: `image_${job.id}`, terminalRevision: 1 }, + text, + ); + if (entry) + this.options.store.update(job.id, { + deliveredEntryId: entry, + deliveryError: null, + }); + } catch { + this.options.store.update(job.id, { + deliveryError: + "이미지 결과를 대화에 전달하지 못했습니다. 저장된 결과로 다시 전달합니다.", + }); + } + } + }); + try { + await this.notices; + } finally { + this.notices = undefined; + } + } + async close(): Promise { + this.shutdown.abort(); + await Promise.allSettled([...this.active.values()]); + await this.poll; + await this.notices; + } + private signal(signal?: AbortSignal): AbortSignal { + return signal + ? AbortSignal.any([signal, this.shutdown.signal]) + : this.shutdown.signal; + } + private exclusive( + id: string, + run: () => Promise, + ): Promise { + const current = this.active.get(id); + const promise = ( + current + ? current.then( + () => undefined, + () => undefined, + ) + : Promise.resolve() + ) + .then(run) + .finally(() => { + if (this.active.get(id) === promise) this.active.delete(id); + }); + this.active.set(id, promise); + return promise; + } + private async apply( + id: string, + result: Awaited>, + ): Promise { + if ( + result.requestId !== id || + (result.result && result.result.requestId !== id) + ) + throw Error("Foreign image result"); + const job = this.get(id); + if (result.state === "completed" && !result.result) + throw Error("Missing image result"); + if (result.state === "completed" && result.result) { + this.options.store.update(id, { resultFilename: result.result.filename }); + try { + const output = await this.options.client.download( + result.result, + this.shutdown.signal, + ); + if (!isImageMime(output.mime)) throw Error("Unsupported image format"); + const artifact = this.options.attachments.put( + `image-${id}.${output.mime === "image/png" ? "png" : "jpg"}`, + output.bytes, + id, + ); + this.options.store.update(id, { + state: "completed", + artifact, + error: null, + }); + } catch (error) { + if ( + error instanceof AttachmentError || + (error instanceof Ima2Error && + ["BODY_TOO_LARGE", "INVALID_IMAGE", "INVALID_RESULT"].includes( + error.code, + )) + ) { + this.options.store.update(id, { + state: "failed", + error: safeError(error), + }); + } else throw error; + } + } else if (result.state === "failed" || result.state === "cancelled") { + this.options.store.update(id, { + state: result.state, + error: result.state === "failed" ? safeError(result.error) : null, + }); + } else { + this.options.store.update(id, { + state: job.cancelRequested + ? "cancelling" + : result.state === "unknown" || result.state === "timed_out" + ? "uncertain" + : result.state, + error: + result.state === "unknown" || result.state === "timed_out" + ? "생성 결과가 아직 확인되지 않았습니다. 같은 요청을 조회하며 다시 생성하지 않습니다." + : null, + }); + } + } +} diff --git a/packages/lina-runtime/src/images/store.ts b/packages/lina-runtime/src/images/store.ts new file mode 100644 index 0000000..803c683 --- /dev/null +++ b/packages/lina-runtime/src/images/store.ts @@ -0,0 +1,233 @@ +import { randomUUID } from "node:crypto"; +import { existsSync } from "node:fs"; +import { join } from "node:path"; +import { isDeepStrictEqual } from "node:util"; +import { type Static, Type } from "typebox"; +import { Value } from "typebox/value"; +import { + atomicJson, + checkedDirectory, + checkedRegular, + readRegular, +} from "../../../lina-core/src/attachments/filesystem.ts"; +import type { AttachmentMetadata } from "../../../lina-core/src/attachments/types.ts"; +import { + ATTACHMENT_MAX_BYTES, + metadataShape, +} from "../../../lina-core/src/attachments/validation.ts"; +import type { BotBinding } from "../../../lina-core/src/protocol.ts"; +import { validateBinding } from "../../../lina-core/src/session-binding.ts"; +import { resultSchema } from "./client-contract.ts"; + +const MAX_JOBS = 256; +const MAX_STORE_BYTES = 8 * 1024 * 1024; +const uuid = Type.String({ + pattern: + "^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$", +}); +const text = (maxLength: number) => Type.String({ minLength: 1, maxLength }); +const nullable = >(schema: T) => + Type.Union([schema, Type.Null()]); +const inputSchema = Type.Object( + { + requestId: text(256), + callId: text(256), + provider: text(128), + model: text(256), + prompt: text(16000), + sourceArtifactId: nullable(uuid), + }, + { additionalProperties: false }, +); +const recordSchema = Type.Object( + { + ...inputSchema.properties, + id: uuid, + state: Type.Union([ + Type.Literal("prepared"), + Type.Literal("submitting"), + Type.Literal("queued"), + Type.Literal("running"), + Type.Literal("post_processing"), + Type.Literal("uncertain"), + Type.Literal("cancelling"), + Type.Literal("completed"), + Type.Literal("failed"), + Type.Literal("cancelled"), + ]), + endpoint: nullable(text(2048)), + runtimeVersion: nullable(text(128)), + createdAt: text(64), + updatedAt: text(64), + error: nullable(text(512)), + cancelRequested: Type.Boolean(), + resultFilename: nullable(text(255)), + deliveryError: nullable(text(512)), + artifact: Type.Union([ + Type.Object( + { + id: uuid, + name: text(120), + mime: text(128), + size: Type.Integer({ minimum: 1, maximum: ATTACHMENT_MAX_BYTES }), + sha256: Type.String({ pattern: "^[a-f0-9]{64}$" }), + }, + { additionalProperties: false }, + ), + Type.Null(), + ]), + deliveredEntryId: nullable(text(256)), + }, + { additionalProperties: false }, +); +export type ImageJobInput = Static; +export type ImageJobState = Static["state"]; +export type ImageJob = Omit, "artifact"> & { + artifact: AttachmentMetadata | null; +}; +export function terminalImageState(state: ImageJobState): boolean { + return state === "completed" || state === "failed" || state === "cancelled"; +} +function parseJob(raw: unknown): ImageJob { + if (!Value.Check(recordSchema, raw)) throw Error("Invalid image job record"); + if ( + raw.resultFilename !== null && + !resultSchema.safeParse({ requestId: raw.id, filename: raw.resultFilename }) + .success + ) + throw Error("Invalid image result filename"); + if (raw.artifact) { + metadataShape(raw.artifact); + if (raw.artifact.id !== raw.id) + throw Error("Invalid image artifact identity"); + } + if ((raw.state === "completed") !== (raw.artifact !== null)) + throw Error("Invalid image artifact state"); + return raw as ImageJob; +} +/** The session lifetime lease owns writes; this manifest never shares task-engine state. */ +export class ImageJobStore { + readonly binding: BotBinding; + private readonly path: string; + private jobs: ImageJob[] = []; + constructor(root: string, binding: BotBinding) { + this.binding = validateBinding(binding); + const dir = checkedDirectory(join(root, "images"), true); + this.path = join(dir, "jobs.json"); + if (!existsSync(this.path)) { + this.save([]); + return; + } + checkedRegular(this.path); + const raw: unknown = JSON.parse( + new TextDecoder().decode(readRegular(this.path, MAX_STORE_BYTES)), + ); + if ( + !raw || + typeof raw !== "object" || + !("binding" in raw) || + !("jobs" in raw) || + !("version" in raw) || + raw.version !== 1 || + Object.keys(raw).length !== 3 || + !isDeepStrictEqual(raw.binding, this.binding) || + !Array.isArray(raw.jobs) + ) + throw Error("Invalid image store binding or schema"); + this.jobs = raw.jobs.map(parseJob); + if ( + this.jobs.length > MAX_JOBS || + new Set(this.jobs.map((j) => j.id)).size !== this.jobs.length || + new Set(this.jobs.map((j) => `${j.requestId}\0${j.callId}`)).size !== + this.jobs.length + ) + throw Error("Invalid image store records"); + } + list(): ImageJob[] { + return structuredClone(this.jobs); + } + get(id: string): ImageJob { + const job = this.jobs.find((j) => j.id === id); + if (!job) throw Error("Image job not found in this conversation"); + return structuredClone(job); + } + create(input: ImageJobInput): ImageJob { + if (!Value.Check(inputSchema, input)) throw Error("Invalid image request"); + const previous = this.jobs.find( + (j) => j.requestId === input.requestId && j.callId === input.callId, + ); + if (previous) { + if ( + Object.keys(input).some( + (key) => + !isDeepStrictEqual( + previous[key as keyof ImageJob], + input[key as keyof ImageJobInput], + ), + ) + ) + throw Error("Image request ID conflict"); + return structuredClone(previous); + } + if (this.jobs.length >= MAX_JOBS) + throw Error("Image job storage limit reached"); + const stamp = new Date().toISOString(); + const job: ImageJob = { + ...input, + id: randomUUID(), + state: "prepared", + endpoint: null, + runtimeVersion: null, + createdAt: stamp, + updatedAt: stamp, + error: null, + cancelRequested: false, + resultFilename: null, + deliveryError: null, + artifact: null, + deliveredEntryId: null, + }; + this.save([...this.jobs, job]); + return structuredClone(job); + } + update( + id: string, + patch: Partial< + Pick< + ImageJob, + | "state" + | "endpoint" + | "runtimeVersion" + | "error" + | "artifact" + | "deliveredEntryId" + | "cancelRequested" + | "resultFilename" + | "deliveryError" + > + >, + ): ImageJob { + const current = this.get(id); + if ( + terminalImageState(current.state) && + Object.keys(patch).some( + (k) => k !== "deliveredEntryId" && k !== "deliveryError", + ) + ) + throw Error("Image job is terminal"); + const job = parseJob({ + ...current, + ...patch, + updatedAt: new Date().toISOString(), + }); + this.save(this.jobs.map((j) => (j.id === id ? job : j))); + return structuredClone(job); + } + private save(jobs: ImageJob[]): void { + const next = { version: 1, binding: this.binding, jobs }; + if (Buffer.byteLength(JSON.stringify(next)) > MAX_STORE_BYTES) + throw Error("Image job storage limit reached"); + atomicJson(this.path, next); + this.jobs = jobs; + } +} diff --git a/packages/lina-runtime/src/images/tools.ts b/packages/lina-runtime/src/images/tools.ts new file mode 100644 index 0000000..52cf85d --- /dev/null +++ b/packages/lina-runtime/src/images/tools.ts @@ -0,0 +1,141 @@ +import { Type } from "typebox"; +import type { LinaToolResult } from "../host.ts"; +import type { ImageJobs } from "./jobs.ts"; +import type { ImageJob } from "./store.ts"; + +type Jobs = Pick< + ImageJobs, + "connect" | "list" | "get" | "start" | "wait" | "reconcile" | "cancel" +>; +const field = (maxLength: number) => Type.String({ minLength: 1, maxLength }); +const jobId = field(128); +const selection = { + provider: field(128), + model: field(256), + prompt: field(16000), +}; +const textResult = (details: unknown): LinaToolResult => ({ + content: [{ type: "text", text: JSON.stringify(details) }], + details, +}); +function project(job: ImageJob) { + return { + jobId: job.id, + requestId: job.requestId, + provider: job.provider, + model: job.model, + state: job.state, + sourceArtifactId: job.sourceArtifactId, + artifact: job.artifact, + resultFilename: job.resultFilename, + error: job.error, + cancelRequested: job.cancelRequested, + deliveryError: job.deliveryError, + delivered: !!job.deliveredEntryId, + }; +} +export function createImageTools( + jobs: Jobs, + requestId: () => string | undefined, +) { + const run = async ( + callId: string, + input: { + provider: string; + model: string; + prompt: string; + sourceArtifactId?: string; + }, + signal?: AbortSignal, + ) => { + const owner = requestId(); + if (!owner) throw Error("Image tool has no active request owner"); + signal?.throwIfAborted(); + const job = await jobs.start( + { + ...input, + sourceArtifactId: input.sourceArtifactId ?? null, + callId, + requestId: owner, + }, + signal, + ); + const final = signal?.aborted + ? await jobs.cancel(job.id) + : await jobs.wait(job.id, signal); + if (final.state === "failed") + throw Error(`Image job ${final.id}: ${final.error}`); + return textResult(project(final)); + }; + return [ + { + name: "lina_image_models", + label: "이미지 연결 확인", + description: + "Inspect ima2 image service, setup URL, exact provider/model catalog and supported operations. Conversation models and credentials are separate. Ready/configured is not successful generation. Ask which exact available image model to use if the user has not selected one; never silently switch provider or billing lane.", + parameters: Type.Object({}, { additionalProperties: false }), + async execute(_id: string, _input: unknown, signal?: AbortSignal) { + return textResult(await jobs.connect(signal)); + }, + }, + { + name: "lina_image_generate", + label: "이미지 만들기", + description: + "Generate exactly one image through ima2 using the user's explicitly chosen provider and model. Requires user authorization for external generation. The result is saved as a managed attachment and delivered once to this conversation. If uncertain, inspect the same job; never create a replacement or switch provider automatically. Do not repeat the image in your reply: the managed completion notice displays it.", + parameters: Type.Object(selection, { additionalProperties: false }), + execute: run, + }, + { + name: "lina_image_edit", + label: "이미지 수정하기", + description: + "Edit a managed image from this conversation, passing its exact sourceArtifactId as the reference to ima2. Inspect image jobs for the previous artifact ID. Explicit provider/model and authorization are required. This creates one new image while preserving the original. No mask/Studio editing. Never retry an uncertain edit or re-display the generated image yourself.", + parameters: Type.Object( + { ...selection, sourceArtifactId: field(128) }, + { additionalProperties: false }, + ), + execute: run, + }, + { + name: "lina_image_jobs", + label: "이미지 작업 목록", + description: + "List image jobs and their source/result artifact IDs owned by this conversation, including pending, failed and recovered jobs. Use before re-editing an image or when a prior request is uncertain.", + parameters: Type.Object({}, { additionalProperties: false }), + execute() { + return textResult({ jobs: jobs.list().map(project) }); + }, + }, + { + name: "lina_image_read", + label: "이미지 작업 확인", + description: + "Reconcile a known image job against ima2 using its original request ID. Never submits generation. Unknown means the outcome is uncertain, not failed or safe to retry.", + parameters: Type.Object({ jobId }, { additionalProperties: false }), + async execute( + _id: string, + input: { jobId: string }, + signal?: AbortSignal, + ) { + signal?.throwIfAborted(); + return textResult(project(await jobs.reconcile(input.jobId))); + }, + }, + { + name: "lina_image_cancel", + label: "이미지 작업 취소", + description: + "Request cancellation of a known image job in this conversation. Cancelling is pending until ima2 reports a terminal outcome; cancellation acknowledgements do not prove the provider stopped or refunded generation.", + parameters: Type.Object({ jobId }, { additionalProperties: false }), + async execute( + _id: string, + input: { jobId: string }, + signal?: AbortSignal, + ) { + signal?.throwIfAborted(); + return textResult(project(await jobs.cancel(input.jobId))); + }, + }, + ]; +} diff --git a/packages/lina-runtime/src/session-app.ts b/packages/lina-runtime/src/session-app.ts index 1779f74..0fb128a 100644 --- a/packages/lina-runtime/src/session-app.ts +++ b/packages/lina-runtime/src/session-app.ts @@ -40,6 +40,10 @@ import { startControlServer } from "./control-server.ts"; import { ExecutionCoordinator } from "./execution.ts"; import { installExecutionHooks } from "./execution-hooks.ts"; import type { LinaHost, SdkSessionOptions } from "./host.ts"; +import { Ima2Client, type Ima2ClientOptions } from "./images/client.ts"; +import { ImageJobs } from "./images/jobs.ts"; +import { ImageJobStore } from "./images/store.ts"; +import { createImageTools } from "./images/tools.ts"; import type { ModelSettings } from "./models/types.ts"; import { installPersona } from "./persona/hooks.ts"; import { nativePreferences } from "./persona/native-preferences.ts"; @@ -55,6 +59,8 @@ import { createStatusTool } from "./tools/status.ts"; export type AppOptions = { engine: SessionEngine; + /** ima2 owns its server and credentials. false disables this optional adapter. */ + imageEngine?: Ima2ClientOptions | false; registerTools?: (host: LinaHost) => void; memoryBackend?: "native" | "honcho" | "disabled"; modelSettings?: () => ModelSettings; @@ -110,6 +116,7 @@ export async function startPersistentApp(options: AppOptions) { const startedAt = new Date().toISOString(); let lastAgentEndAt: string | undefined; let attachments: AttachmentStore | undefined; + let images: ImageJobs | undefined; let stopped = false, stopping: Promise | undefined; const stop = (): Promise => { @@ -117,6 +124,7 @@ export async function startPersistentApp(options: AppOptions) { if (stopping) return stopping; stopping = (async () => { await server?.stop(); + await images?.close(); const reflectionClosing = reflection?.close(); if (runtime) await runtime.close(); else await native?.close(); @@ -233,6 +241,16 @@ export async function startPersistentApp(options: AppOptions) { }), }); execution = coordinator; + if (options.imageEngine !== false) { + images = new ImageJobs({ + store: new ImageJobStore(lease.root, binding), + attachments: attached, + client: new Ima2Client(options.imageEngine ?? {}), + notify: (marker, text) => + native?.appendNotice(marker, text) ?? Promise.resolve(null), + }); + } + const imageJobs = images; native = await (options.createSession ?? engine.create)({ workspace, sessionFile, @@ -303,6 +321,11 @@ export async function startPersistentApp(options: AppOptions) { } coordinator.configurePermissions(permissions); installExecutionHooks(host, coordinator); + if (imageJobs) + for (const tool of createImageTools(imageJobs, () => + runtime?.currentRequestId(), + )) + host.registerTool(tool); host.registerTool( createAttachmentTool( attached, @@ -382,6 +405,9 @@ export async function startPersistentApp(options: AppOptions) { coordinator.refresh(); if (event.snapshot.state === "idle") { reflection?.settled(); + void imageJobs?.flushNotices().catch(() => { + // The durable image job remains undelivered for recovery. + }); } } }); @@ -393,6 +419,7 @@ export async function startPersistentApp(options: AppOptions) { port: options.port, }); void channel.refresh(); + imageJobs?.resume(); return { binding, runtime, @@ -401,6 +428,7 @@ export async function startPersistentApp(options: AppOptions) { memory: memoryBridge, reflection, attachments: attached, + images: imageJobs, execution: coordinator, port: server.port, stop, diff --git a/packages/lina-runtime/test/ima2-client-fixture.ts b/packages/lina-runtime/test/ima2-client-fixture.ts new file mode 100644 index 0000000..11ccf24 --- /dev/null +++ b/packages/lina-runtime/test/ima2-client-fixture.ts @@ -0,0 +1,81 @@ +import { Ima2Client, type Ima2ClientOptions } from "../src/images/client.ts"; + +// Contract fixtures: ima2-gen 36aa6fc (v3.14.0), routes/health.ts, +// routes/models.ts, lib/generatePipeline.ts and lib/inflight.ts. +export const origin = "http://127.0.0.1:43127"; +export const requestId = "lina-image-001"; +export const input = { + requestId, + provider: "api", + model: "image-model", + prompt: "A tree", +}; +export const png = Buffer.from( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVR4nGNgAAIAAAUAAXpeqz8AAAAASUVORK5CYII=", + "base64", +); +export const lane = { + status: "ready", + defaults: { image: "different-default" }, + models: { + image: [ + { + id: "image-model", + label: "Image", + capabilities: { + source: "verified-contract", + inputRoles: ["text", "image_references"], + parameters: [], + aspectRatios: [], + }, + }, + ], + video: [], + }, + surfaces: { + generate: { + supported: true, + references: true, + mask: false, + streaming: false, + catalogAccess: "static", + }, + }, +}; +export const catalog = { ok: true, lanes: { api: lane } }; +export function terminal( + status = "completed", + meta: Record = { filenames: ["image.png"], imageCount: 1 }, +) { + return { + requestId, + kind: "classic", + status, + startedAt: 1, + finishedAt: 2, + durationMs: 1, + phase: "streaming", + phaseAt: 1, + meta, + }; +} +export function fixture( + handler: (request: Request) => Response | Promise = () => + Response.json({ requestId, async: true }, { status: 202 }), + options: Omit = {}, +) { + const requests: Request[] = []; + const fetch = async (url: string, init: RequestInit) => { + const request = new Request(url, init); + requests.push(new Request(url, init)); + if (new URL(url).pathname === "/api/health") + return Response.json({ ok: true, version: "3.14.0" }); + if (new URL(url).pathname === "/api/models") return Response.json(catalog); + return handler(request); + }; + return { + client: new Ima2Client({ baseUrl: origin, ...options, fetch }), + fetch, + requests, + }; +} diff --git a/packages/lina-runtime/test/ima2-client-protocol.test.ts b/packages/lina-runtime/test/ima2-client-protocol.test.ts new file mode 100644 index 0000000..80cf818 --- /dev/null +++ b/packages/lina-runtime/test/ima2-client-protocol.test.ts @@ -0,0 +1,274 @@ +import { expect, test } from "bun:test"; +import { Ima2Client } from "../src/images/client.ts"; +import { + catalog, + fixture, + input, + lane, + origin, + png, + requestId, + terminal, +} from "./ima2-client-fixture.ts"; + +test("generated JPEG bytes are downloadable and usable for a reference edit", async () => { + // Existing Lina JPEG fixture (attachment-document-tool.test.ts). + const jpeg = Buffer.from( + "/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAP//////////////////////////////////////////////////////////////////////////////////////wgALCAABAAEBAREA/8QAFBABAAAAAAAAAAAAAAAAAAAAAP/aAAgBAQABPxA=", + "base64", + ); + const { client, requests } = fixture((request) => + request.method === "POST" + ? Response.json({ requestId, async: true }, { status: 202 }) + : new Response(jpeg, { headers: { "content-type": "image/jpeg" } }), + ); + const image = await client.download({ requestId, filename: "result.jpg" }); + expect(image).toEqual({ bytes: new Uint8Array(jpeg), mime: "image/jpeg" }); + await client.submit({ ...input, reference: image }); + expect(await requests.at(-1)?.json()).toMatchObject({ + references: [`data:image/jpeg;base64,${jpeg.toString("base64")}`], + }); +}); + +test("a model without reference support rejects edit without a POST", async () => { + let posts = 0; + const client = new Ima2Client({ + baseUrl: origin, + fetch: async (url, init) => { + if (init.method === "POST") posts++; + return Response.json( + url.endsWith("/api/health") + ? { ok: true, version: "3.14.0" } + : { + ...catalog, + lanes: { + api: { + ...lane, + surfaces: { + generate: { supported: true, references: false }, + }, + }, + }, + }, + ); + }, + }); + await expect( + client.submit({ ...input, reference: { bytes: png, mime: "image/png" } }), + ).rejects.toMatchObject({ + code: "UNSUPPORTED_OPERATION", + outcome: "rejected", + }); + expect(posts).toBe(0); +}); + +test("foreign, duplicate, or paginated history cannot become this request's result", async () => { + for (const page of [ + { items: [{ requestId: "other-request", filename: "other.png" }] }, + { + items: [ + { requestId, filename: "one.png" }, + { requestId, filename: "two.png" }, + ], + }, + { items: [{ requestId, filename: "one.png" }], nextCursor: "more" }, + ]) { + const { client } = fixture((r) => + Response.json( + new URL(r.url).pathname === "/api/inflight" + ? { jobs: [], terminalJobs: [] } + : page, + ), + ); + await expect(client.read(requestId)).rejects.toMatchObject({ + code: "INVALID_RESPONSE", + }); + } +}); + +test("malformed success JSON, unexpected HTTP status and provider switches are uncertain submissions", async () => { + for (const response of [ + new Response("{private", { + headers: { "content-type": "application/json" }, + }), + new Response("private", { headers: { "content-type": "text/html" } }), + Response.json({ requestId, async: true }, { status: 201 }), + Response.json({ + requestId, + idempotentReplay: true, + filename: "x.png", + provider: "oauth", + model: input.model, + }), + ]) { + const { client } = fixture(() => response); + await client.connect(); + await expect(client.submit(input)).rejects.toMatchObject({ + code: "INVALID_RESPONSE", + outcome: "unknown", + }); + } +}); + +test("malformed cancellation identity cannot be reported as acknowledged", async () => { + const { client } = fixture(() => + Response.json({ requestId: "other", active: true, aborted: true }), + ); + await expect(client.cancel(requestId)).rejects.toMatchObject({ + code: "INVALID_RESPONSE", + outcome: "unknown", + }); +}); + +test("JSON byte limit terminates oversized bodies without echoing their contents", async () => { + const { client } = fixture( + () => + new Response( + new ReadableStream({ + start(controller) { + controller.enqueue(new Uint8Array(4 * 1024 * 1024 + 1)); + controller.close(); + }, + }), + { headers: { "content-type": "application/json" } }, + ), + ); + await client.connect(); + await expect(client.submit(input)).rejects.toMatchObject({ + code: "BODY_TOO_LARGE", + outcome: "unknown", + }); +}); + +test("JSON and downloads use manual redirects with omitted credentials", async () => { + const calls: RequestInit[] = []; + const base = fixture(); + const client = new Ima2Client({ + baseUrl: origin, + fetch: async (url, init) => { + calls.push(init); + return base.fetch(url, init); + }, + }); + await client.connect(); + await client.submit(input); + for (const call of calls) + expect(call).toMatchObject({ redirect: "manual", credentials: "omit" }); +}); + +test("terminal success spelling variants retain result identity", async () => { + for (const status of ["done", "complete"]) { + const { client } = fixture(() => + Response.json({ jobs: [], terminalJobs: [terminal(status)] }), + ); + expect((await client.read(requestId)).result).toEqual({ + requestId, + filename: "image.png", + }); + } +}); + +test("409 idempotency collision remains unknown and can recover the existing job", async () => { + const { client, requests } = fixture((request) => + request.method === "POST" + ? Response.json( + { + code: "IDEMPOTENCY_KEY_CONFLICT", + requestId, + error: "private details", + }, + { status: 409 }, + ) + : Response.json({ jobs: [], terminalJobs: [terminal()] }), + ); + await client.connect(); + await expect(client.submit(input)).rejects.toMatchObject({ + code: "CONFLICT", + outcome: "unknown", + status: 409, + }); + expect(await client.read(requestId)).toEqual({ + requestId, + state: "completed", + result: { requestId, filename: "image.png" }, + }); + expect(requests.filter((request) => request.method === "POST")).toHaveLength( + 1, + ); +}); + +test("unknown-kind cancellation tombstone does not confirm a generation was cancelled", async () => { + const { client, requests } = fixture((request) => + Response.json( + new URL(request.url).pathname === "/api/inflight" + ? { + jobs: [], + terminalJobs: [{ ...terminal("canceled", {}), kind: "unknown" }], + } + : { items: [], nextCursor: null }, + ), + ); + expect(await client.read(requestId)).toEqual({ requestId, state: "unknown" }); + expect(requests.at(-1)?.url).toBe( + `${origin}/api/history?requestId=${requestId}&limit=2`, + ); +}); + +test("matching history proves completed after restart despite an unknown cancellation tombstone", async () => { + const { client, requests } = fixture((request) => + Response.json( + new URL(request.url).pathname === "/api/inflight" + ? { + jobs: [], + terminalJobs: [{ ...terminal("canceled", {}), kind: "unknown" }], + } + : { + items: [{ requestId, filename: "recovered.png" }], + nextCursor: null, + }, + ), + ); + expect(await client.read(requestId)).toEqual({ + requestId, + state: "completed", + result: { requestId, filename: "recovered.png" }, + }); + expect(requests.every((request) => request.method === "GET")).toBe(true); +}); + +test("unexpected raw failure details never enter normalized terminal jobs", async () => { + const { client } = fixture(() => + Response.json({ + jobs: [], + terminalJobs: [ + { + ...terminal("failed", { + prompt: "private prompt", + credential: "secret credential", + }), + errorCode: "SECRET_CREDENTIAL", + error: "private response", + }, + ], + }), + ); + const result = JSON.stringify(await client.read(requestId)); + expect(result).not.toContain("private"); + expect(result).not.toContain("SECRET_CREDENTIAL"); + expect(result).not.toContain("secret credential"); +}); + +test("unknown input and output URL fields do not open alternative execution or download surfaces", async () => { + const { client, requests } = fixture(); + await expect( + client.submit({ ...input, ...{ providerUrl: "https://foreign.invalid" } }), + ).rejects.toMatchObject({ code: "INVALID_INPUT" }); + await expect( + client.download({ + requestId, + filename: "image.png", + ...{ url: "https://foreign.invalid/image.png" }, + }), + ).rejects.toMatchObject({ code: "INVALID_RESULT" }); + expect(requests).toHaveLength(0); +}); diff --git a/packages/lina-runtime/test/ima2-client-security.test.ts b/packages/lina-runtime/test/ima2-client-security.test.ts new file mode 100644 index 0000000..ec6f9b9 --- /dev/null +++ b/packages/lina-runtime/test/ima2-client-security.test.ts @@ -0,0 +1,396 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { Ima2Client } from "../src/images/client.ts"; +import { + catalog, + fixture, + input, + lane, + origin, + png, + requestId, + terminal, +} from "./ima2-client-fixture.ts"; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) + rmSync(root, { recursive: true, force: true }); +}); +function serverFile(data?: unknown) { + const root = mkdtempSync(join(tmpdir(), "lina-ima2-")); + roots.push(root); + const path = join(root, "server.json"); + if (data !== undefined) writeFileSync(path, JSON.stringify(data)); + return path; +} + +test("server.json discovers backend.url without trusting proxy, port, or admin credentials", async () => { + const { fetch, requests } = fixture(); + const client = new Ima2Client({ + serverFile: serverFile({ + backend: { url: origin }, + url: "http://127.0.0.1:9", + adminToken: "private-admin-token", + oauth: { url: "http://127.0.0.1:8" }, + }), + fetch, + }); + expect((await client.connect()).baseUrl).toBe(origin); + expect( + requests.every( + (r) => !r.headers.has("authorization") && !r.headers.has("x-ima2-token"), + ), + ).toBe(true); +}); + +test("explicit URL wins over absent discovery and absent discovery never probes a default port", async () => { + const path = serverFile(); + const { fetch, requests } = fixture(); + expect( + ( + await new Ima2Client({ + baseUrl: origin, + serverFile: path, + fetch, + }).connect() + ).baseUrl, + ).toBe(origin); + const before = requests.length; + await expect( + new Ima2Client({ serverFile: path, fetch }).connect(), + ).rejects.toMatchObject({ + code: "DISCOVERY_UNAVAILABLE", + outcome: "rejected", + }); + expect(requests.length).toBe(before); +}); + +test("discovered endpoint is fixed for client lifetime across reconnect and recovery", async () => { + const path = serverFile({ backend: { url: origin } }); + const { fetch, requests } = fixture(() => + Response.json({ jobs: [], terminalJobs: [terminal()] }), + ); + const client = new Ima2Client({ serverFile: path, fetch }); + expect((await client.connect()).baseUrl).toBe(origin); + const changedOrigin = "http://127.0.0.1:43128"; + writeFileSync(path, JSON.stringify({ backend: { url: changedOrigin } })); + expect((await client.connect()).baseUrl).toBe(origin); + expect((await client.read(requestId)).state).toBe("completed"); + expect( + requests.every((request) => new URL(request.url).origin === origin), + ).toBe(true); + expect( + (await new Ima2Client({ serverFile: path, fetch }).connect()).baseUrl, + ).toBe(changedOrigin); +}); + +test("catalog snapshots returned to callers cannot override submission admission", async () => { + const { client } = fixture(); + const connection = await client.connect(); + connection.lanes[0]?.models.push({ + id: "foreign-model", + label: "Foreign", + generate: true, + edit: true, + }); + await expect( + client.submit({ ...input, model: "foreign-model" }), + ).rejects.toMatchObject({ code: "MODEL_UNAVAILABLE" }); +}); + +test("bounded request timeout covers headers and stalled response bodies", async () => { + for (const phase of ["headers", "body"]) { + const { client } = fixture( + () => + phase === "headers" + ? new Promise(() => {}) + : new Response( + new ReadableStream({ + start(c) { + c.enqueue(new TextEncoder().encode("{")); + }, + }), + { headers: { "content-type": "application/json" } }, + ), + { timeoutMs: 20 }, + ); + await client.connect(); + await expect(client.submit(input)).rejects.toMatchObject({ + code: "TIMEOUT", + outcome: "unknown", + }); + } +}); + +test("pre-aborted submit cannot send a request even when connection is cached", async () => { + const { client, requests } = fixture(); + await client.connect(); + const abort = new AbortController(); + abort.abort("secret reason"); + await expect(client.submit(input, abort.signal)).rejects.toMatchObject({ + code: "ABORTED", + outcome: "rejected", + }); + expect(requests).toHaveLength(2); +}); + +test.each([ + "https://remote.invalid", + "http://127.0.0.1:43127/path", + "http://secret@127.0.0.1:43127", + "file:///etc/passwd", +])("unsafe discovery %s makes no request", async (url) => { + const { fetch, requests } = fixture(); + await expect( + new Ima2Client({ + serverFile: serverFile({ backend: { url } }), + fetch, + }).connect(), + ).rejects.toMatchObject({ outcome: "rejected" }); + expect(requests).toHaveLength(0); +}); + +test.each([ + "http://secret@localhost:1", + "http://localhost:1/path", + "http://localhost:1/?token=x", + "http://localhost:1/#x", + "file:///etc/passwd", +])("invalid explicit origin %s rejected", async (baseUrl) => { + const { fetch, requests } = fixture(); + await expect( + new Ima2Client({ baseUrl, fetch }).connect(), + ).rejects.toMatchObject({ code: "INVALID_SERVER_URL" }); + expect(requests).toHaveLength(0); +}); + +test("version mismatch and malformed catalogs cannot be marked ready", async () => { + for (const body of [ + { ok: true, version: "3.13.0" }, + { ok: true, version: "4.0.0" }, + { ok: true }, + ]) { + await expect( + new Ima2Client({ + baseUrl: origin, + fetch: async () => Response.json(body), + }).connect(), + ).rejects.toMatchObject({ outcome: "rejected" }); + } + await expect( + new Ima2Client({ + baseUrl: origin, + fetch: async (url) => + Response.json( + url.endsWith("/api/health") + ? { ok: true, version: "3.14.0" } + : { ok: true, lanes: [] }, + ), + }).connect(), + ).rejects.toMatchObject({ code: "INVALID_RESPONSE" }); +}); + +test("unknown/MCP and agy lanes cannot silently route to oauth or a forced model", async () => { + for (const provider of ["higgsfield", "unrecognized", "agy"]) { + const client = new Ima2Client({ + baseUrl: origin, + fetch: async (url) => + Response.json( + url.endsWith("/api/health") + ? { ok: true, version: "3.14.0" } + : { ...catalog, lanes: { [provider]: lane } }, + ), + }); + expect((await client.connect()).lanes[0]?.models[0]?.generate).toBe(false); + await expect(client.submit({ ...input, provider })).rejects.toMatchObject({ + code: "UNSUPPORTED_OPERATION", + outcome: "rejected", + }); + } +}); + +test("unknown model and auto provider never reach generate", async () => { + const { client, requests } = fixture(); + await client.connect(); + await expect( + client.submit({ ...input, model: "missing" }), + ).rejects.toMatchObject({ code: "MODEL_UNAVAILABLE" }); + await expect( + client.submit({ ...input, provider: "auto" }), + ).rejects.toMatchObject({ outcome: "rejected" }); + expect(requests).toHaveLength(2); +}); + +test("request identifiers are rejected instead of normalized to a different upstream ID", async () => { + const { client, requests } = fixture(); + for (const id of ["", "x".repeat(129), "../other", "id\nsecret"]) { + await expect( + client.submit({ ...input, requestId: id }), + ).rejects.toMatchObject({ code: "INVALID_INPUT", outcome: "rejected" }); + } + expect(requests).toHaveLength(0); +}); + +test("invalid or oversized reference bytes are rejected before network submission", async () => { + const { client, requests } = fixture(); + for (const bytes of [ + new Uint8Array(), + new TextEncoder().encode("secret"), + new Uint8Array(2 * 1024 * 1024 + 1), + ]) { + await expect( + client.submit({ ...input, reference: { bytes, mime: "image/png" } }), + ).rejects.toMatchObject({ outcome: "rejected" }); + } + await expect( + client.submit({ ...input, reference: { bytes: png, mime: "image/jpeg" } }), + ).rejects.toMatchObject({ code: "INVALID_IMAGE" }); + expect(requests).toHaveLength(0); +}); + +test.each([ + "../secret.png", + "http://evil.invalid/image.png", + "/etc/file.png", + "%2e%2e.png", + "x.png?token=secret", + "image.svg", + "x\\a.png", +])( + "unsafe result filename %s is refused without download", + async (filename) => { + const { client, requests } = fixture(); + await expect( + client.download({ requestId, filename }), + ).rejects.toMatchObject({ code: "INVALID_RESULT" }); + expect(requests).toHaveLength(0); + }, +); + +test("foreign identities, multiple images, wrong kind, and malformed results fail closed", async () => { + for (const row of [ + terminal("completed", { filenames: ["a.png", "b.png"] }), + terminal("completed", { filenames: ["../a.png"] }), + { ...terminal(), kind: "video" }, + terminal("completed", {}), + ]) { + const { client } = fixture(() => + Response.json({ jobs: [], terminalJobs: [row] }), + ); + await expect(client.read(requestId)).rejects.toMatchObject({ + code: "INVALID_RESPONSE", + }); + } + const { client } = fixture(() => + Response.json({ requestId: "other", async: true }, { status: 202 }), + ); + await client.connect(); + await expect(client.submit(input)).rejects.toMatchObject({ + code: "INVALID_RESPONSE", + outcome: "unknown", + }); +}); + +test.each([401, 403, 409, 429, 500, 503])( + "HTTP %s errors are redacted and classify submission uncertainty", + async (status) => { + const { client, requests } = fixture(() => + Response.json( + { + error: "SECRET provider key https://secret.invalid/?token=private", + code: "PRIVATE_SECRET", + }, + { status }, + ), + ); + await client.connect(); + try { + await client.submit(input); + throw new Error("expected failure"); + } catch (error) { + expect(error).toMatchObject({ + status, + outcome: status < 500 && status !== 409 ? "rejected" : "unknown", + }); + expect(String(error)).not.toContain("SECRET"); + expect(JSON.stringify(error)).not.toContain("private"); + } + expect(requests.filter((r) => r.method === "POST")).toHaveLength(1); + }, +); + +test("network loss after submit is unknown and is never retried", async () => { + const { client, requests } = fixture(() => { + throw new Error("secret provider key"); + }); + await client.connect(); + await expect(client.submit(input)).rejects.toMatchObject({ + code: "NETWORK_ERROR", + outcome: "unknown", + }); + expect(requests.filter((r) => r.method === "POST")).toHaveLength(1); +}); + +test("caller abort stops pending request without issuing upstream cancellation", async () => { + const started = Promise.withResolvers(); + const { client, requests } = fixture( + (r) => + new Promise((_resolve, reject) => { + started.resolve(); + r.signal.addEventListener("abort", () => reject(r.signal.reason), { + once: true, + }); + }), + ); + await client.connect(); + const abort = new AbortController(); + const work = client.submit(input, abort.signal); + await started.promise; + abort.abort(new Error("private abort reason")); + await expect(work).rejects.toMatchObject({ + code: "ABORTED", + outcome: "unknown", + }); + expect(requests.some((r) => r.method === "DELETE")).toBe(false); +}); + +test("download refuses redirect, mismatched MIME, invalid container and oversized stream", async () => { + for (const response of [ + new Response(null, { + status: 302, + headers: { location: "https://evil.invalid" }, + }), + new Response(png, { headers: { "content-type": "image/jpeg" } }), + new Response("private", { + headers: { "content-type": "image/png" }, + }), + new Response(png, { + headers: { "content-type": "image/png", "content-length": "2097153" }, + }), + ]) { + const { client } = fixture(() => response); + await expect( + client.download({ requestId, filename: "image.png" }), + ).rejects.toBeInstanceOf(Error); + } + const { client } = fixture( + () => + new Response( + new ReadableStream({ + start(c) { + c.enqueue(png); + c.enqueue(new Uint8Array(100)); + c.close(); + }, + }), + { headers: { "content-type": "image/png" } }, + ), + { maxImageBytes: 100 }, + ); + await expect( + client.download({ requestId, filename: "image.png" }), + ).rejects.toMatchObject({ code: "BODY_TOO_LARGE" }); +}); diff --git a/packages/lina-runtime/test/ima2-client.test.ts b/packages/lina-runtime/test/ima2-client.test.ts new file mode 100644 index 0000000..84d3409 --- /dev/null +++ b/packages/lina-runtime/test/ima2-client.test.ts @@ -0,0 +1,233 @@ +import { expect, test } from "bun:test"; +import { Ima2Client } from "../src/images/client.ts"; +import { + catalog, + fixture, + input, + lane, + origin, + png, + requestId, + terminal, +} from "./ima2-client-fixture.ts"; + +test("connect projects actual lanes and never treats catalog readiness as generation proof", async () => { + const { client, requests } = fixture(); + expect(await client.connect()).toMatchObject({ + baseUrl: origin, + version: "3.14.0", + ready: true, + lanes: [ + { + provider: "api", + status: "ready", + models: [{ id: "image-model", generate: true, edit: true }], + }, + ], + }); + expect(requests.map((r) => new URL(r.url).pathname)).toEqual([ + "/api/health", + "/api/models", + ]); +}); + +test("submit sends ONE async generation with explicit selection and same request/idempotency identity", async () => { + const { client, requests } = fixture(); + await client.connect(); + expect(await client.submit(input)).toEqual({ requestId, state: "queued" }); + const request = requests.at(-1); + expect(request?.headers.get("idempotency-key")).toBe(requestId); + expect(request?.headers.get("x-request-id")).toBe(requestId); + expect(await request?.json()).toEqual({ + ...input, + async: true, + n: 1, + references: [], + format: "png", + }); +}); + +test("reference edits transmit provided bytes as one data URL", async () => { + const { client, requests } = fixture(); + await client.connect(); + await client.submit({ + ...input, + reference: { bytes: png, mime: "image/png" }, + }); + expect(await requests.at(-1)?.json()).toMatchObject({ + references: [`data:image/png;base64,${png.toString("base64")}`], + }); +}); + +test("a fresh client recovers completed output by the same requestId without a POST", async () => { + const { client, requests } = fixture(() => + Response.json({ jobs: [], terminalJobs: [terminal()] }), + ); + expect(await client.read(requestId)).toEqual({ + requestId, + state: "completed", + result: { requestId, filename: "image.png" }, + }); + expect(requests.every((r) => r.method === "GET")).toBe(true); + expect(requests.at(-1)?.url).toBe(`${origin}/api/inflight?includeTerminal=1`); +}); + +test.each([ + ["queued", "queued"], + ["streaming", "running"], + ["decoding", "post_processing"], + ["new-upstream-phase", "unknown"], +] as const)("active phase %s is %s", async (phase, state) => { + const { client } = fixture(() => + Response.json({ + jobs: [{ requestId, kind: "classic", phase }], + terminalJobs: [], + }), + ); + expect((await client.read(requestId)).state).toBe(state); +}); + +test.each([ + ["failed", "failed"], + ["error", "failed"], + ["canceled", "cancelled"], + ["cancelled", "cancelled"], + ["unexpected", "unknown"], +] as const)( + "terminal status %s is %s without fabricating a result", + async (status, state) => { + const { client } = fixture(() => + Response.json({ jobs: [], terminalJobs: [terminal(status, {})] }), + ); + const job = await client.read(requestId); + expect(job.state).toBe(state); + expect(job.result).toBeUndefined(); + }, +); + +test("tracking timeout is terminal uncertainty, never cancellation or automatic retry", async () => { + const { client } = fixture(() => + Response.json({ + jobs: [], + terminalJobs: [ + { + ...terminal("error", {}), + errorCode: "JOB_TRACKING_TIMEOUT", + httpStatus: 504, + }, + ], + }), + ); + expect(await client.read(requestId)).toMatchObject({ + requestId, + state: "timed_out", + error: { code: "JOB_TRACKING_TIMEOUT" }, + }); +}); + +test("expired terminal record is recovered only from matching history", async () => { + const { client, requests } = fixture((r) => + Response.json( + new URL(r.url).pathname === "/api/inflight" + ? { jobs: [], terminalJobs: [] } + : { items: [{ requestId, filename: "image.png" }], nextCursor: null }, + ), + ); + expect((await client.read(requestId)).result).toEqual({ + requestId, + filename: "image.png", + }); + expect(requests.at(-1)?.url).toBe( + `${origin}/api/history?requestId=${requestId}&limit=2`, + ); +}); + +test("missing records remain unknown with no resubmission", async () => { + const { client, requests } = fixture((r) => + Response.json( + new URL(r.url).pathname === "/api/inflight" + ? { jobs: [], terminalJobs: [] } + : { items: [], nextCursor: null }, + ), + ); + expect(await client.read(requestId)).toEqual({ requestId, state: "unknown" }); + expect(requests.every((r) => r.method === "GET")).toBe(true); +}); + +test.each([ + { active: true, aborted: true }, + { active: false, aborted: false }, + { active: true, aborted: false }, +])("cancel preserves registry acknowledgement %j", async (ack) => { + const { client, requests } = fixture(() => + Response.json({ requestId, ...ack }), + ); + expect(await client.cancel(requestId)).toEqual({ requestId, ...ack }); + expect(requests.at(-1)?.method).toBe("DELETE"); + expect(requests.at(-1)?.url).toBe(`${origin}/api/inflight/${requestId}`); +}); + +test("download uses only generated filename on configured origin", async () => { + const { client, requests } = fixture( + () => new Response(png, { headers: { "Content-Type": "image/png" } }), + ); + const result = await client.download({ requestId, filename: "image.png" }); + expect(result.mime).toBe("image/png"); + expect(result.bytes).toEqual(new Uint8Array(png)); + expect(requests.at(-1)?.url).toBe(`${origin}/generated/image.png`); +}); + +test("upstream filenames retain model dots, Unicode prompt slugs and encoded punctuation", async () => { + // lib/filename.ts preserves dots and CJK/emoji in its prompt slug. + const filename = "gpt-5.5_1x1_20260907_나무🌲-(초록)_0.png"; + const { client, requests } = fixture( + () => new Response(png, { headers: { "Content-Type": "image/png" } }), + ); + expect((await client.download({ requestId, filename })).mime).toBe( + "image/png", + ); + expect(requests.at(-1)?.url).toBe( + `${origin}/generated/${encodeURIComponent(filename)}`, + ); +}); + +test("idempotent terminal replay accepts exact same request identity", async () => { + const { client } = fixture(() => + Response.json({ + requestId, + filename: "image.png", + image: "data:image/png;base64,ignored", + provider: "api", + model: "image-model", + idempotentReplay: true, + }), + ); + await client.connect(); + expect((await client.submit(input)).result).toEqual({ + requestId, + filename: "image.png", + }); +}); + +test.each(["key-missing", "disconnected", "locked"])( + "unready lane %s is visible but cannot submit", + async (status) => { + const requests: string[] = []; + const client = new Ima2Client({ + baseUrl: origin, + fetch: async (url) => { + requests.push(url); + return Response.json( + url.endsWith("/api/health") + ? { ok: true, version: "3.14.0" } + : { ...catalog, lanes: { api: { ...lane, status } } }, + ); + }, + }); + expect((await client.connect()).ready).toBe(false); + await expect(client.submit(input)).rejects.toMatchObject({ + code: "LANE_UNAVAILABLE", + }); + expect(requests.some((url) => url.endsWith("/api/generate"))).toBe(false); + }, +); diff --git a/packages/lina-runtime/test/image-app-fixture.ts b/packages/lina-runtime/test/image-app-fixture.ts new file mode 100644 index 0000000..76e7518 --- /dev/null +++ b/packages/lina-runtime/test/image-app-fixture.ts @@ -0,0 +1,438 @@ +import { randomUUID } from "node:crypto"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { deflateSync } from "node:zlib"; +import type { + CodexRpc, + CodexRpcRequestHandler, +} from "../../lina-codex/src/rpc.ts"; +import { + type CodexSession, + createCodexEngine, + createCodexSession, +} from "../../lina-codex/src/session.ts"; +import type { SessionSnapshot } from "../../lina-core/src/protocol.ts"; +import type { ContextServices } from "../src/context/port.ts"; +import type { ModelControl } from "../src/models/port.ts"; +import { startPersistentApp } from "../src/session-app.ts"; +import { catalog, terminal } from "./ima2-client-fixture.ts"; + +// Deterministic 320x200 RGB PNGs, generated locally without an image provider. +function solidPng( + rgb: readonly [number, number, number], +): Uint8Array { + const width = 320, + height = 200; + const raw = Buffer.alloc((width * 3 + 1) * height); + for (let y = 0; y < height; y++) + for (let x = 0; x < width; x++) + raw.set(rgb, y * (width * 3 + 1) + 1 + x * 3); + const header = Buffer.alloc(13); + header.writeUInt32BE(width, 0); + header.writeUInt32BE(height, 4); + header[8] = 8; + header[9] = 2; + const chunk = (type: string, data: Uint8Array) => { + const result = Buffer.alloc(data.length + 12); + result.writeUInt32BE(data.length, 0); + result.write(type, 4, 4, "ascii"); + result.set(data, 8); + let crc = 0xffffffff; + for (const byte of result.subarray(4, -4)) { + crc ^= byte; + for (let bit = 0; bit < 8; bit++) + crc = (crc >>> 1) ^ (crc & 1 ? 0xedb88320 : 0); + } + result.writeUInt32BE((crc ^ 0xffffffff) >>> 0, result.length - 4); + return result; + }; + return new Uint8Array( + Buffer.concat([ + Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]), + chunk("IHDR", header), + chunk("IDAT", deflateSync(raw)), + chunk("IEND", new Uint8Array()), + ]), + ); +} +export const GENERATED_PNG = solidPng([53, 93, 167]); +export const EDITED_PNG = solidPng([53, 127, 105]); + +type GenerateBody = { + requestId: string; + provider: string; + model: string; + prompt: string; + async: boolean; + n: number; + format: string; + references: string[]; +}; +type ToolReply = { + success: boolean; + contentItems: Array<{ type: "inputText"; text: string }>; +}; +const MODEL = "synthetic-image-driver"; + +function startIma2Fixture() { + const submissions: Array<{ + body: GenerateBody; + idempotencyKey: string | null; + }> = []; + const downloads: string[] = []; + const outputs = new Map(); + const terminalJobs: ReturnType[] = []; + const server = Bun.serve({ + hostname: "127.0.0.1", + port: 0, + async fetch(request) { + const url = new URL(request.url); + if (url.pathname === "/api/health") + return Response.json({ ok: true, version: "3.14.0" }); + if (url.pathname === "/api/models") return Response.json(catalog); + if (url.pathname === "/api/generate" && request.method === "POST") { + const body = (await request.json()) as GenerateBody; + submissions.push({ + body, + idempotencyKey: request.headers.get("Idempotency-Key"), + }); + const filename = `${body.requestId}.png`; + const failed = body.prompt.includes("fail"); + terminalJobs.push({ + ...terminal( + failed ? "failed" : "completed", + failed ? {} : { filenames: [filename], imageCount: 1 }, + ), + requestId: body.requestId, + }); + if (!failed) + outputs.set( + filename, + body.references.length ? EDITED_PNG : GENERATED_PNG, + ); + return Response.json( + { requestId: body.requestId, async: true }, + { status: 202 }, + ); + } + if (url.pathname === "/api/inflight") + return Response.json({ jobs: [], terminalJobs }); + if (url.pathname.startsWith("/generated/")) { + const filename = url.pathname.slice("/generated/".length); + const bytes = outputs.get(filename); + if (bytes) { + downloads.push(filename); + return new Response(bytes, { + headers: { "Content-Type": "image/png" }, + }); + } + } + return new Response("Unknown synthetic ima2 route", { status: 404 }); + }, + }); + return { + baseUrl: `http://127.0.0.1:${server.port}`, + submissions, + downloads, + stop: () => server.stop(true), + }; +} + +class ImageCodexRpc implements CodexRpc { + closed = false; + readonly pid = undefined; + readonly methods: string[] = []; + readonly calls: Array<{ + tool: string; + arguments: unknown; + result: ToolReply; + }> = []; + readonly failures: unknown[] = []; + private threadId: string = randomUUID(); + private readonly listeners = new Set< + (method: string, params: unknown) => void + >(); + private readonly handlers = new Set(); + private readonly pending = new Set>(); + + async request(method: string, params?: unknown): Promise { + if (this.closed) throw Error("Synthetic RPC closed"); + this.methods.push(method); + return this.reply(method, params) as T; + } + private reply(method: string, params: unknown): unknown { + if (method === "initialize") return { userAgent: "image-qa" }; + if (method === "model/list") return { data: [{ id: MODEL, model: MODEL }] }; + if (method === "skills/list") return { data: [] }; + if (method === "thread/resume") + this.threadId = (params as { threadId: string }).threadId; + if (["thread/start", "thread/resume", "thread/read"].includes(method)) { + // Empty native history makes the adapter's actual persisted journal the only replay source. + return { + thread: { + id: this.threadId, + turns: [], + modelProvider: "synthetic", + status: { type: "idle" }, + }, + }; + } + if ( + [ + "thread/name/set", + "thread/inject_items", + "skills/extraRoots/set", + ].includes(method) + ) + return {}; + if (method === "turn/start") { + const text = (params as { input: Array<{ text: string }> }).input + .map((part) => part.text) + .join("\n"); + const turnId = randomUUID(); + const task = this.turn(turnId, text) + .catch((error: unknown) => { + this.failures.push(error); + this.emit("eof", { + message: + error instanceof Error ? error.message : "Synthetic RPC failed", + }); + }) + .finally(() => this.pending.delete(task)); + this.pending.add(task); + return { turn: { id: turnId, items: [], status: "inProgress" } }; + } + if (method === "turn/interrupt") { + this.emit("turn/completed", { + threadId: this.threadId, + turn: { + id: (params as { turnId: string }).turnId, + items: [], + status: "interrupted", + }, + }); + return {}; + } + throw Error(`Unexpected synthetic RPC method: ${method}`); + } + private emit(method: string, params: unknown): void { + for (const listener of this.listeners) listener(method, params); + } + private async tool( + turnId: string, + tool: string, + args: unknown, + ): Promise { + const handler = [...this.handlers][0]; + if (!handler) throw Error("Codex adapter did not register its RPC handler"); + const callId = randomUUID(); + const result = (await handler("item/tool/call", { + threadId: this.threadId, + turnId, + callId, + tool, + arguments: args, + })) as ToolReply; + this.calls.push({ tool, arguments: args, result }); + this.emit("item/completed", { + threadId: this.threadId, + turnId, + item: { + id: callId, + type: "dynamicToolCall", + tool, + arguments: args, + ...result, + }, + }); + return result; + } + private async turn(turnId: string, text: string): Promise { + const frame = { threadId: this.threadId, turnId }; + this.emit("turn/started", { + ...frame, + turn: { id: turnId, items: [], status: "inProgress" }, + }); + this.emit("item/completed", { + ...frame, + item: { + id: randomUUID(), + type: "userMessage", + content: [{ type: "text", text, text_elements: [] }], + }, + }); + const args = { provider: "api", model: "image-model", prompt: text }; + let result: ToolReply; + if (/\bedit\b/i.test(text)) { + const listing = await this.tool(turnId, "lina_image_jobs", {}); + if (!listing.success) throw Error("Image job listing failed"); + const jobs = JSON.parse(listing.contentItems[0]?.text ?? "{}") as { + jobs: Array<{ artifact: { id: string } | null }>; + }; + const source = jobs.jobs.findLast( + (job) => job.artifact !== null, + )?.artifact; + if (!source) throw Error("Generate an image before editing"); + result = await this.tool(turnId, "lina_image_edit", { + ...args, + sourceArtifactId: source.id, + }); + } else result = await this.tool(turnId, "lina_image_generate", args); + this.emit("item/completed", { + ...frame, + item: { + id: randomUUID(), + type: "agentMessage", + text: result.success + ? "이미지 작업을 처리했습니다." + : "이미지 작업에 실패했습니다.", + phase: "final_answer", + }, + }); + this.emit("turn/completed", { + threadId: this.threadId, + turn: { id: turnId, items: [], status: "completed" }, + }); + } + notify(): void {} + subscribe(listener: (method: string, params: unknown) => void): () => void { + this.listeners.add(listener); + return () => { + this.listeners.delete(listener); + }; + } + onRequest(handler: CodexRpcRequestHandler): () => void { + this.handlers.add(handler); + return () => { + this.handlers.delete(handler); + }; + } + async close(): Promise { + this.closed = true; + await Promise.all(this.pending); + } +} + +const services: ContextServices = { + estimateText: (text) => Math.ceil(text.length / 4), + estimateMessages: (messages) => + Math.ceil(JSON.stringify(messages).length / 4), + systemTokens: 0, + contextWindow: 128000, + reserveTokens: 1000, + summarize: async () => { + throw Error("Unexpected summarization in image QA"); + }, + prepare: () => { + throw Error("Unexpected compaction in image QA"); + }, +}; +const models: ModelControl = { + catalog: () => [], + state: () => ({ + provider: "synthetic", + model: MODEL, + settingsRevision: 1, + error: null, + }), + test: async () => { + throw Error("External model calls are disabled in image QA"); + }, +}; + +export async function createImageAppFixture() { + const root = mkdtempSync(join(tmpdir(), "lina-image-app-")); + const workspace = join(root, "workspace"); + const agentDir = join(root, "agent"); + for (const dir of [workspace, agentDir]) mkdirSync(dir, { mode: 0o700 }); + const ima2 = startIma2Fixture(); + let rpc = new ImageCodexRpc(); + let native: CodexSession; + const start = () => + startPersistentApp({ + engine: createCodexEngine({ services, models }), + createSession: async (options) => { + native = await createCodexSession({ + ...options, + services, + models, + rpcClient: rpc, + }); + return native; + }, + imageEngine: { baseUrl: ima2.baseUrl }, + workspace, + agentDir, + stateRoot: join(root, "state"), + systemPrompt: "Isolated synthetic image QA. No external model calls.", + memoryBackend: "disabled", + port: 0, + }); + let app: Awaited>; + try { + app = await start(); + } catch (error) { + await ima2.stop(); + rmSync(root, { recursive: true, force: true }); + throw error; + } + return { + root, + ima2, + get app() { + return app; + }, + get native() { + return native; + }, + get rpc() { + return rpc; + }, + async restart() { + await app.stop(); + await rpc.close(); + rpc = new ImageCodexRpc(); + app = await start(); + }, + async submit(text: string): Promise { + const requestId = randomUUID(); + const previous = new Set( + app.runtime.snapshot().messages.map((m) => m.entryId), + ); + const done = Promise.withResolvers(); + const check = (snapshot: SessionSnapshot) => { + const request = snapshot.requests.find( + (request) => request.id === requestId, + ); + if (request?.status === "rejected" || request?.status === "interrupted") + done.reject(Error(request.error ?? request.status)); + if ( + request?.status === "settled" && + snapshot.messages.some( + (m) => m.entryId.startsWith("notice-") && !previous.has(m.entryId), + ) + ) + done.resolve(); + }; + const off = app.runtime.subscribe((event) => { + if (event.type === "snapshot") check(event.snapshot); + }); + try { + app.runtime.submit(requestId, text); + await done.promise; + // The notice was already observed; join its pending manifest acknowledgement. + await app.images?.flushNotices(); + return requestId; + } finally { + off(); + } + }, + async close(keepRoot = false) { + await app.stop(); + await rpc.close(); + await ima2.stop(); + if (!keepRoot) rmSync(root, { recursive: true, force: true }); + }, + }; +} diff --git a/packages/lina-runtime/test/image-app.test.ts b/packages/lina-runtime/test/image-app.test.ts new file mode 100644 index 0000000..e9dd225 --- /dev/null +++ b/packages/lina-runtime/test/image-app.test.ts @@ -0,0 +1,179 @@ +import { afterEach, expect, test } from "bun:test"; +import { randomUUID } from "node:crypto"; +import { dirname } from "node:path"; +import { loadCodexJournal } from "../../lina-codex/src/identity.ts"; +import { ImageJobStore } from "../src/images/store.ts"; +import { + createImageAppFixture, + EDITED_PNG, + GENERATED_PNG, +} from "./image-app-fixture.ts"; + +const cleanup: Array<() => Promise> = []; +afterEach(async () => { + for (const close of cleanup.splice(0).reverse()) await close(); +}); + +test("real image app persists a native notice and reuses the saved image after restart", async () => { + const fixture = await createImageAppFixture(); + cleanup.push(() => fixture.close()); + const firstRequest = await fixture.submit("generate"); + const first = fixture.app.images?.list()[0]; + if (!first?.artifact) throw Error("Generated attachment missing"); + expect(first.state).toBe("completed"); + expect(first.requestId).toBe(firstRequest); + expect(first.sourceArtifactId).toBeNull(); + expect(fixture.app.attachments.bytes(first.artifact.id)).toEqual( + GENERATED_PNG, + ); + expect(fixture.ima2.submissions).toHaveLength(1); + expect(fixture.ima2.submissions[0]?.body).toEqual({ + requestId: first.id, + provider: "api", + model: "image-model", + prompt: "generate", + async: true, + n: 1, + references: [], + format: "png", + }); + expect(fixture.ima2.submissions[0]?.idempotencyKey).toBe(first.id); + expect(fixture.rpc.calls.map((call) => call.tool)).toEqual([ + "lina_image_generate", + ]); + const binding = fixture.app.binding; + const expectedText = `이미지를 만들었습니다.\n\n![생성 이미지](/api/attachments/${first.artifact.id}/preview?sessionId=${binding.sessionId})\n\n[이미지 다운로드](/api/attachments/${first.artifact.id}?sessionId=${binding.sessionId})`; + const notice = fixture.app.runtime + .snapshot() + .messages.find((m) => m.text === expectedText); + expect(notice?.role).toBe("assistant"); + expect(notice?.entryId ?? null).toBe(first.deliveredEntryId); + expect(loadCodexJournal(binding.sessionFile)).toContainEqual( + expect.objectContaining({ + type: "custom_message", + id: first.deliveredEntryId, + content: expectedText, + details: { jobId: `image_${first.id}`, terminalRevision: 1 }, + }), + ); + const previewPath = `/api/attachments/${first.artifact.id}/preview?sessionId=${binding.sessionId}`; + const preview = await fetch( + `http://127.0.0.1:${fixture.app.port}${previewPath}`, + ); + expect(preview.status).toBe(200); + expect(preview.headers.get("Content-Type")).toBe("image/png"); + expect(new Uint8Array(await preview.arrayBuffer())).toEqual(GENERATED_PNG); + const foreign = await fetch( + `http://127.0.0.1:${fixture.app.port}/api/attachments/${first.artifact.id}/preview?sessionId=${randomUUID()}`, + ); + expect(foreign.status).toBe(403); + await foreign.arrayBuffer(); + + // Simulate a crash after the native journal write but before recording delivery. + await fixture.app.stop(); + const interruptedStore = new ImageJobStore( + dirname(binding.sessionFile), + binding, + ); + interruptedStore.update(first.id, { deliveredEntryId: null }); + expect(interruptedStore.get(first.id).deliveredEntryId).toBeNull(); + await fixture.restart(); + expect(fixture.app.binding).toEqual(binding); + expect(fixture.rpc.methods).toContain("thread/resume"); + expect(fixture.rpc.methods).not.toContain("thread/start"); + expect(fixture.native.history()).toContainEqual( + expect.objectContaining({ + id: first.deliveredEntryId, + content: expectedText, + }), + ); + await fixture.app.images?.recover(); + await fixture.app.images?.flushNotices(); + expect( + fixture.app.runtime + .snapshot() + .messages.filter((m) => m.text === expectedText), + ).toHaveLength(1); + const recovered = fixture.app.images?.get(first.id); + if (!recovered) throw Error("Recovered image job missing"); + expect(recovered).toEqual({ ...first, updatedAt: expect.any(String) }); + expect(fixture.app.images?.list()).toHaveLength(1); + expect(fixture.ima2.submissions).toHaveLength(1); + const restoredPreview = await fetch( + `http://127.0.0.1:${fixture.app.port}${previewPath}`, + ); + expect(restoredPreview.status).toBe(200); + expect(new Uint8Array(await restoredPreview.arrayBuffer())).toEqual( + GENERATED_PNG, + ); + + const editRequest = await fixture.submit("edit"); + const edit = fixture.app.images?.list()[1]; + if (!edit?.artifact) throw Error("Edited attachment missing"); + expect(edit.state).toBe("completed"); + expect(edit.requestId).toBe(editRequest); + expect(edit.sourceArtifactId).toBe(first.artifact.id); + expect(edit.artifact.id).not.toBe(first.artifact.id); + expect( + fixture.app.runtime + .snapshot() + .messages.find((m) => m.entryId === edit.deliveredEntryId)?.text, + ).toBe( + `이미지를 수정했습니다.\n\n![생성 이미지](/api/attachments/${edit.artifact.id}/preview?sessionId=${binding.sessionId})\n\n[이미지 다운로드](/api/attachments/${edit.artifact.id}?sessionId=${binding.sessionId})`, + ); + expect(fixture.app.attachments.bytes(first.artifact.id)).toEqual( + GENERATED_PNG, + ); + expect(fixture.app.attachments.bytes(edit.artifact.id)).toEqual(EDITED_PNG); + expect(fixture.ima2.submissions).toHaveLength(2); + expect(fixture.ima2.submissions[1]?.body.references).toEqual([ + `data:image/png;base64,${Buffer.from(GENERATED_PNG).toString("base64")}`, + ]); + expect(fixture.rpc.calls.map((call) => call.tool)).toEqual([ + "lina_image_jobs", + "lina_image_edit", + ]); + await fixture.restart(); + await fixture.app.images?.recover(); + expect(fixture.ima2.submissions).toHaveLength(2); + expect(fixture.app.images?.list()).toEqual([recovered, edit]); + expect( + fixture.app.runtime + .snapshot() + .messages.filter((m) => m.entryId.startsWith("notice-")), + ).toHaveLength(2); + expect( + loadCodexJournal(binding.sessionFile).filter( + (entry) => + typeof entry === "object" && + entry !== null && + "type" in entry && + entry.type === "custom_message", + ), + ).toHaveLength(2); +}, 15_000); + +test("failed image generation records one failure notice without creating an attachment", async () => { + const fixture = await createImageAppFixture(); + cleanup.push(() => fixture.close()); + await fixture.submit("generate fail"); + const failed = fixture.app.images?.list()[0]; + expect(failed?.state).toBe("failed"); + expect(failed?.artifact).toBeNull(); + expect(failed?.deliveredEntryId).toBeString(); + expect(fixture.rpc.calls[0]?.result.success).toBe(false); + expect(fixture.ima2.downloads).toHaveLength(0); + const notices = fixture.app.runtime + .snapshot() + .messages.filter((m) => m.entryId.startsWith("notice-")); + expect(notices).toHaveLength(1); + expect(notices[0]?.text).toContain("이미지를 만들지 못했습니다."); + await fixture.restart(); + await fixture.app.images?.recover(); + expect(fixture.ima2.submissions).toHaveLength(1); + expect( + fixture.app.runtime + .snapshot() + .messages.filter((m) => m.entryId.startsWith("notice-")), + ).toEqual(notices); +}, 15_000); diff --git a/packages/lina-runtime/test/image-jobs.test.ts b/packages/lina-runtime/test/image-jobs.test.ts new file mode 100644 index 0000000..a9c7d86 --- /dev/null +++ b/packages/lina-runtime/test/image-jobs.test.ts @@ -0,0 +1,545 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { AttachmentStore } from "../../lina-core/src/attachments/store.ts"; +import { Ima2Error } from "../src/images/client-types.ts"; +import { ImageJobs } from "../src/images/jobs.ts"; +import { ImageJobStore } from "../src/images/store.ts"; + +const cleanups: Array<() => unknown> = []; +afterEach(async () => { + for (const cleanup of cleanups.splice(0).reverse()) await cleanup(); +}); +const png = Buffer.from( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVR4nGNgAAIAAAUAAXpeqz8AAAAASUVORK5CYII=", + "base64", +); +const request = { + requestId: "chat-1", + callId: "call-1", + provider: "fixture", + model: "image-one", + prompt: "A blue circle", + sourceArtifactId: null, +}; +function fixture() { + const root = mkdtempSync(join(tmpdir(), "lina-image-jobs-")); + cleanups.push(() => rmSync(root, { recursive: true, force: true })); + const binding = { + version: 1 as const, + botId: "lina", + sessionId: "a80be8ed-a493-4e88-8523-b5f85d73d652", + sessionFile: join(root, "session.jsonl"), + workspace: root, + }; + const attached = new AttachmentStore(root, binding); + cleanups.push(() => attached.close()); + const store = new ImageJobStore(root, binding); + return { root, binding, attached, store }; +} +// The transport fixtures will use the pinned adapter's exported contract. +test("generation completion imports once, delivers once after restart, and edits owned bytes", async () => { + const f = fixture(); + const submitted: unknown[] = []; + const notices = new Map(); + const client = { + connect: async () => ({ + baseUrl: "http://127.0.0.1:45678", + version: "3.14.0", + ready: true, + lanes: [], + }), + submit: async (input: { requestId: string }) => { + submitted.push(input); + return { state: "queued" as const, requestId: input.requestId }; + }, + read: async (id: string) => ({ + state: "completed" as const, + requestId: id, + result: { requestId: id, filename: "result.png" }, + }), + cancel: async () => ({ requestId: "fixture", active: true, aborted: true }), + download: async () => ({ bytes: png, mime: "image/png" as const }), + }; + const notify = async (marker: { jobId: string }, text: string) => { + notices.set(marker.jobId, text); + return marker.jobId; + }; + const jobs = new ImageJobs({ + store: f.store, + attachments: f.attached, + client, + notify, + }); + const first = await jobs.start(request); + await jobs.reconcile(first.id); + expect(f.store.get(first.id)).toMatchObject({ + state: "completed", + artifact: { id: first.id, mime: "image/png" }, + }); + expect(notices.size).toBe(1); + expect([...notices.values()][0]).toStartWith("이미지를 만들었습니다.\n\n"); + expect([...notices.values()][0]).toContain( + `/api/attachments/${first.id}/preview?sessionId=${f.binding.sessionId}`, + ); + const restored = new ImageJobs({ + store: new ImageJobStore(f.root, f.binding), + attachments: f.attached, + client, + notify, + }); + await restored.recover(); + expect(submitted).toHaveLength(1); + expect(notices.size).toBe(1); + await restored.start({ + ...request, + requestId: "chat-2", + callId: "call-2", + prompt: "Make it green", + sourceArtifactId: first.id, + }); + expect(submitted).toHaveLength(2); + expect(submitted[1]).toMatchObject({ + reference: { bytes: new Uint8Array(png), mime: "image/png" }, + }); + await jobs.close(); + await restored.close(); +}); + +function backend( + overrides: Partial = {}, +) { + const submits: Array<{ requestId: string }> = []; + const client: import("../src/images/jobs.ts").ImageClient = { + connect: async () => ({ + baseUrl: "http://127.0.0.1:45678", + version: "3.14.0", + ready: true, + lanes: [], + ownership: "external", + setupUrl: "http://127.0.0.1:45678", + }), + submit: async (input) => { + submits.push(input); + return { requestId: input.requestId, state: "queued" }; + }, + read: async (id) => ({ requestId: id, state: "unknown" }), + cancel: async (id) => ({ requestId: id, active: true, aborted: true }), + download: async () => ({ bytes: new Uint8Array(png), mime: "image/png" }), + ...overrides, + }; + return { client, submits }; +} +function manager( + f: ReturnType, + client: import("../src/images/jobs.ts").ImageClient, +) { + const jobs = new ImageJobs({ + store: f.store, + attachments: f.attached, + client, + notify: async () => null, + }); + cleanups.push(() => jobs.close()); + return jobs; +} + +test("lost submit response is reconciled on restart without another POST", async () => { + const f = fixture(); + let posts = 0; + const first = backend({ + submit: async () => { + posts++; + throw Error("lost response secret-token"); + }, + }); + const jobs = manager(f, first.client); + const pending = await jobs.start(request); + expect(pending.state).toBe("uncertain"); + expect(pending.error).not.toContain("secret-token"); + await jobs.close(); + const recovered = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename: "result.png" }, + }), + }); + const next = new ImageJobs({ + store: new ImageJobStore(f.root, f.binding), + attachments: f.attached, + client: recovered.client, + notify: async () => null, + }); + cleanups.push(() => next.close()); + await next.recover(); + expect(next.get(pending.id).state).toBe("completed"); + expect(posts).toBe(1); + expect(recovered.submits).toHaveLength(0); +}); + +test("cancel acknowledgement stays cancelling until upstream confirms a terminal outcome", async () => { + const f = fixture(); + let terminal = false; + let cancellations = 0; + const { client } = backend({ + cancel: async (id) => { + cancellations++; + return { requestId: id, active: true, aborted: true }; + }, + read: async (id) => ({ + requestId: id, + state: terminal ? "cancelled" : "running", + }), + }); + const jobs = manager(f, client); + const job = await jobs.start(request); + expect((await jobs.cancel(job.id)).state).toBe("cancelling"); + const saved = new ImageJobStore(f.root, f.binding); + expect(saved.get(job.id).state).toBe("cancelling"); + terminal = true; + await jobs.reconcile(job.id); + expect(jobs.get(job.id).state).toBe("cancelled"); + expect(cancellations).toBe(1); +}); + +test("a completion racing cancellation is retained and shown as completed", async () => { + const f = fixture(); + const { client } = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename: "result.png" }, + }), + }); + const jobs = manager(f, client); + const job = await jobs.start(request); + expect((await jobs.cancel(job.id)).state).toBe("completed"); + expect(f.attached.bytes(job.id)).toEqual(new Uint8Array(png)); +}); + +test("foreign reference is rejected before generation and creates no image job", async () => { + const f = fixture(); + const other = fixture(); + const source = other.attached.put("other.png", png); + const { client, submits } = backend(); + const jobs = manager(f, client); + await expect( + jobs.start({ ...request, sourceArtifactId: source.id }), + ).rejects.toThrow("not found"); + expect(jobs.list()).toEqual([]); + expect(submits).toEqual([]); +}); + +test("a changed endpoint cannot receive reads or cancellation of an existing request", async () => { + const f = fixture(); + const b = backend(); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + let reads = 0; + let cancels = 0; + b.client.connect = async () => ({ + baseUrl: "http://127.0.0.1:49999", + version: "3.14.0", + ready: true, + lanes: [], + ownership: "external", + setupUrl: "http://127.0.0.1:49999", + }); + b.client.read = async (id) => { + reads++; + return { requestId: id, state: "unknown" }; + }; + b.client.cancel = async (id) => { + cancels++; + return { requestId: id, active: false, aborted: false }; + }; + expect((await jobs.reconcile(job.id)).state).toBe("uncertain"); + expect((await jobs.cancel(job.id)).state).toBe("cancelling"); + expect(reads).toBe(0); + expect(cancels).toBe(0); +}); + +test("corrupt downloaded bytes produce an import failure without being displayed", async () => { + const f = fixture(); + const b = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename: "image.png" }, + }), + download: async () => ({ + bytes: new TextEncoder().encode("not an image"), + mime: "image/png", + }), + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + await jobs.reconcile(job.id); + expect(jobs.get(job.id).state).toBe("failed"); + expect(jobs.get(job.id).artifact).toBeNull(); + expect(() => f.attached.get(job.id)).toThrow("not found"); +}); + +test("a busy conversation defers durable completion delivery until idle", async () => { + const f = fixture(); + const b = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename: "image.png" }, + }), + }); + let idle = false; + let sends = 0; + const jobs = new ImageJobs({ + store: f.store, + attachments: f.attached, + client: b.client, + notify: async () => { + if (!idle) return null; + sends++; + return "native-notice"; + }, + }); + cleanups.push(() => jobs.close()); + const job = await jobs.start(request); + await jobs.reconcile(job.id); + expect(sends).toBe(0); + expect(jobs.get(job.id).deliveredEntryId).toBeNull(); + idle = true; + await jobs.flushNotices(); + await jobs.recover(); + expect(sends).toBe(1); + expect(jobs.get(job.id).deliveredEntryId).toBe("native-notice"); +}); + +test("definite provider/model admission rejection is failed without leaving an uncertain job", async () => { + const f = fixture(); + const b = backend({ + submit: async () => { + throw new Ima2Error("MODEL_UNAVAILABLE", "no model", "rejected"); + }, + }); + const jobs = manager(f, b.client); + const result = await jobs.start(request); + expect(result.state).toBe("failed"); + expect(result.error).toContain("MODEL_UNAVAILABLE"); +}); + +test("foreign completion identity cannot be imported into an owned job", async () => { + const f = fixture(); + const b = backend({ + read: async () => ({ + requestId: "foreign", + state: "completed", + result: { requestId: "foreign", filename: "image.png" }, + }), + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + await jobs.reconcile(job.id); + expect(jobs.get(job.id).state).toBe("uncertain"); + expect(jobs.get(job.id).artifact).toBeNull(); +}); + +test("restart preserves cancellation intent after a lost cancel response", async () => { + const f = fixture(); + let attempts = 0; + let cancelled = false; + const b = backend({ + cancel: async (id) => { + attempts++; + if (attempts === 1) throw Error("connection lost"); + cancelled = true; + return { requestId: id, active: true, aborted: true }; + }, + read: async (id) => ({ + requestId: id, + state: cancelled ? "cancelled" : "running", + }), + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + await jobs.cancel(job.id); + await jobs.close(); + const next = new ImageJobs({ + store: new ImageJobStore(f.root, f.binding), + attachments: f.attached, + client: b.client, + notify: async () => null, + }); + cleanups.push(() => next.close()); + await next.recover(); + expect(next.get(job.id).state).toBe("cancelled"); + expect(attempts).toBe(2); +}); + +test("a concurrent read cannot consume a cancellation request", async () => { + const f = fixture(); + let cancels = 0; + const b = backend({ + read: async (id) => ({ requestId: id, state: "running" }), + cancel: async (id) => { + cancels++; + return { requestId: id, active: true, aborted: true }; + }, + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + const cancellation = jobs.cancel(job.id); + const competing = jobs.reconcile(job.id); + await Promise.all([cancellation, competing]); + expect(cancels).toBeGreaterThan(0); + expect(jobs.get(job.id).state).toBe("cancelling"); +}); + +test("queued and post-processing phases remain visible in the persisted job", async () => { + const f = fixture(); + const b = backend({ + read: async (id) => ({ requestId: id, state: "post_processing" }), + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + expect(job.state).toBe("queued"); + await jobs.reconcile(job.id); + expect(jobs.get(job.id).state).toBe("post_processing"); +}); + +test("synchronous notice events do not enter completion delivery twice", async () => { + const f = fixture(); + const b = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename: "image.png" }, + }), + }); + let deliveries = 0; + let nested: Promise | undefined; + const jobs = new ImageJobs({ + store: f.store, + attachments: f.attached, + client: b.client, + notify: async () => { + deliveries++; + if (deliveries === 1) nested = jobs.flushNotices(); + return "saved-notice"; + }, + }); + cleanups.push(() => jobs.close()); + const job = await jobs.start(request); + await jobs.reconcile(job.id); + await nested; + expect(deliveries).toBe(1); +}); + +test("an undeliverable notice remains recoverable and never poisons shutdown", async () => { + const f = fixture(); + let fail = true; + const b = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename: "image.png" }, + }), + }); + const jobs = new ImageJobs({ + store: f.store, + attachments: f.attached, + client: b.client, + notify: async () => { + if (fail) throw Error("transient journal failure"); + return "notice-after-recovery"; + }, + }); + cleanups.push(() => jobs.close()); + const job = await jobs.start(request); + jobs.resume(); + await jobs.reconcile(job.id); + expect(jobs.get(job.id).deliveredEntryId).toBeNull(); + expect(jobs.get(job.id).deliveryError).toBeString(); + fail = false; + await jobs.recover(); + expect(jobs.get(job.id).deliveredEntryId).toBe("notice-after-recovery"); + await jobs.close(); + await jobs.close(); +}); + +test("confirmed oversized output becomes an import failure and allows the next explicit generation", async () => { + const f = fixture(); + const b = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename: "oversized.png" }, + }), + download: async () => { + throw new Ima2Error("BODY_TOO_LARGE", "too large"); + }, + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + await jobs.reconcile(job.id); + expect(jobs.get(job.id)).toMatchObject({ + state: "failed", + artifact: null, + resultFilename: "oversized.png", + }); + await jobs.start({ + ...request, + requestId: "other-request", + callId: "other-call", + prompt: "A small simple image", + }); + expect(b.submits).toHaveLength(2); +}); + +test("cancellation intent is durable while an earlier read is still blocked", async () => { + const f = fixture(); + const entered = Promise.withResolvers(); + const release = Promise.withResolvers(); + let reads = 0; + const b = backend({ + read: async (id) => { + if (++reads === 1) { + entered.resolve(); + await release.promise; + } + return { requestId: id, state: "running" }; + }, + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + const read = jobs.reconcile(job.id); + await entered.promise; + const cancelled = jobs.cancel(job.id); + const stored = new ImageJobStore(f.root, f.binding).get(job.id); + release.resolve(); + await Promise.all([read, cancelled]); + expect(stored.cancelRequested).toBe(true); + expect(stored.state).toBe("cancelling"); +}); + +test("upstream filenames with model-version dots and Korean prompt text remain valid provenance", async () => { + const f = fixture(); + const filename = "gpt-5.5_1x1_20260907_나무_0.png"; + const b = backend({ + read: async (id) => ({ + requestId: id, + state: "completed", + result: { requestId: id, filename }, + }), + }); + const jobs = manager(f, b.client); + const job = await jobs.start(request); + await jobs.reconcile(job.id); + expect(jobs.get(job.id)).toMatchObject({ + state: "completed", + resultFilename: filename, + }); + expect(new ImageJobStore(f.root, f.binding).get(job.id).resultFilename).toBe( + filename, + ); +}); diff --git a/packages/lina-runtime/test/image-store.test.ts b/packages/lina-runtime/test/image-store.test.ts new file mode 100644 index 0000000..f6d9ab8 --- /dev/null +++ b/packages/lina-runtime/test/image-store.test.ts @@ -0,0 +1,88 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { ImageJobStore } from "../src/images/store.ts"; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) + rmSync(root, { recursive: true, force: true }); +}); +function setup() { + const root = mkdtempSync(join(tmpdir(), "lina-image-store-")); + roots.push(root); + const binding = { + version: 1 as const, + botId: "lina", + sessionId: "a80be8ed-a493-4e88-8523-b5f85d73d652", + sessionFile: join(root, "session.jsonl"), + workspace: root, + }; + return { root, binding }; +} +const input = { + requestId: "chat-1", + callId: "call-1", + provider: "fixture", + model: "image-one", + prompt: "A blue circle", + sourceArtifactId: null, +}; +test("image jobs persist before dispatch, deduplicate the original call and reject conflicts", () => { + const { root, binding } = setup(); + const first = new ImageJobStore(root, binding); + const job = first.create(input); + expect(job.state).toBe("prepared"); + const second = new ImageJobStore(root, binding); + expect(second.create(input)).toEqual(job); + expect(() => second.create({ ...input, prompt: "different" })).toThrow( + "conflict", + ); + expect(second.list()).toHaveLength(1); + expect(() => new ImageJobStore(root, { ...binding, botId: "other" })).toThrow( + "binding", + ); +}); +test("states and provenance survive restart without returning mutable records", () => { + const { root, binding } = setup(); + const first = new ImageJobStore(root, binding); + const job = first.create(input); + first.update(job.id, { + state: "uncertain", + endpoint: "http://127.0.0.1:45678", + runtimeVersion: "3.14.0", + error: "Outcome unknown", + }); + const second = new ImageJobStore(root, binding); + const restored = second.get(job.id); + expect(restored).toMatchObject({ + state: "uncertain", + provider: "fixture", + model: "image-one", + requestId: "chat-1", + error: "Outcome unknown", + }); + restored.state = "failed"; + expect(second.get(job.id).state).toBe("uncertain"); + expect(() => second.get("../foreign")).toThrow(); +}); + +test("an image record cannot claim a different artifact ID", () => { + const { root, binding } = setup(); + const store = new ImageJobStore(root, binding); + const job = store.create(input); + expect(() => + store.update(job.id, { + state: "completed", + artifact: { + id: "22222222-2222-4222-8222-222222222222", + name: "result.png", + mime: "image/png", + size: 1, + sha256: "a".repeat(64), + }, + }), + ).toThrow("artifact"); + expect(store.get(job.id).state).toBe("prepared"); +}); diff --git a/packages/lina-runtime/test/image-tools.test.ts b/packages/lina-runtime/test/image-tools.test.ts new file mode 100644 index 0000000..3fd9ed5 --- /dev/null +++ b/packages/lina-runtime/test/image-tools.test.ts @@ -0,0 +1,50 @@ +import { expect, test } from "bun:test"; +import { CodexHost } from "../../lina-codex/src/host.ts"; +import { createImageTools } from "../src/images/tools.ts"; + +test("Codex registers image tools with explicit provider/model and rejects unowned/no-request input", async () => { + let starts = 0; + const jobs = { + connect: async () => ({ + ready: false, + lanes: [], + baseUrl: "http://127.0.0.1:45678", + version: "3.14.0", + generationVerified: [], + recoveryError: null, + ownership: "external" as const, + setupUrl: "http://127.0.0.1:45678", + }), + list: () => [], + get: () => { + throw Error("not found"); + }, + start: async () => { + starts++; + throw Error("should not execute"); + }, + wait: async () => { + throw Error("unused"); + }, + reconcile: async () => { + throw Error("not found"); + }, + cancel: async () => { + throw Error("not found"); + }, + }; + const tools = createImageTools(jobs, () => undefined); + const host = new CodexHost("/tmp", () => ({ action: "allow" })); + for (const tool of tools) host.registerTool(tool); + expect(host.tools.has("lina_image_generate")).toBe(true); + const generate = host.tools.get("lina_image_generate"); + if (!generate) throw Error("missing tool"); + await expect( + generate.execute( + "call", + { provider: "api", model: "image", prompt: "hello" }, + new AbortController().signal, + ), + ).rejects.toThrow("active request"); + expect(starts).toBe(0); +}); diff --git a/packages/lina-runtime/test/session-app.test.ts b/packages/lina-runtime/test/session-app.test.ts index 0188528..6d04f27 100644 --- a/packages/lina-runtime/test/session-app.test.ts +++ b/packages/lina-runtime/test/session-app.test.ts @@ -169,3 +169,46 @@ test("failed native shutdown retains ownership until a successful retry", async const lease = acquireSessionLease(config.stateRoot, "lina", config.workspace); lease.close(); }); + +test("image tools register in the Codex app with isolated state and no provider call at startup", async () => { + const config = options(); + let host: CodexHost | undefined; + let requests = 0; + const create = config.createSession; + config.createSession = async (sdkOptions) => { + host = new CodexHost(config.workspace, () => ({ action: "allow" })); + sdkOptions.register?.( + host.asLinaHost(), + { + estimateText: (text) => text.length, + estimateMessages: (messages) => messages.length, + systemTokens: 0, + contextWindow: 96000, + reserveTokens: 1000, + summarize: async () => "", + prepare: () => { + throw Error("unused"); + }, + }, + () => ({ action: "allow" }), + ); + return create(sdkOptions); + }; + const app = await startPersistentApp({ + ...config, + imageEngine: { + baseUrl: "http://127.0.0.1:45678", + fetch: async () => { + requests++; + throw Error("must not call on startup"); + }, + }, + }); + cleanups.push(app.stop); + expect(host?.tools.has("lina_image_models")).toBe(true); + expect(host?.tools.has("lina_image_generate")).toBe(true); + expect(host?.tools.has("lina_image_edit")).toBe(true); + expect(app.images?.list()).toEqual([]); + expect(existsSync(join(config.stateRoot, "images/jobs.json"))).toBe(true); + expect(requests).toBe(0); +}); diff --git a/packages/lina-web/client/markdown-inline.ts b/packages/lina-web/client/markdown-inline.ts index 4ee63a9..5977950 100644 --- a/packages/lina-web/client/markdown-inline.ts +++ b/packages/lina-web/client/markdown-inline.ts @@ -7,6 +7,10 @@ import { const UUID = "[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}"; +const IMAGE_PREVIEW = new RegExp( + `^/api/attachments/${UUID}/preview\\?sessionId=(${UUID})$`, + "u", +); type LinkMatch = { readonly end: number; readonly label: string; @@ -35,6 +39,24 @@ export function parseInline( if (source[index] === "!" && source[index + 1] === "[") { const image = linkAt(source, index + 1); if (image !== undefined) { + const preview = IMAGE_PREVIEW.exec(image.destination); + // Match the raw URL exactly: never normalize paths, queries or whitespace. + if ( + preview !== null && + preview[0] === image.destination && + preview[1] !== undefined + ) { + flushPlain(index); + budget.take(); + tokens.push({ + type: "attachment-image", + src: image.destination, + alt: image.label, + sessionId: preview[1], + source: `!${image.source}`, + }); + plainStart = image.end; + } index = image.end; continue; } @@ -80,7 +102,7 @@ export function parseInline( if (source[index] === "[") { const link = linkAt(source, index); if (link !== undefined) { - const kind = classifyDestination(link.destination); + const kind = classifyDestination(link.destination.trim()); if (kind !== undefined) { flushPlain(index); const children = parseInline(link.label, budget, depth + 1); @@ -126,7 +148,7 @@ function linkAt(source: string, start: number): LinkMatch | undefined { return { end: end + 1, label: source.slice(start + 1, labelEnd), - destination: source.slice(labelEnd + 2, end).trim(), + destination: source.slice(labelEnd + 2, end), source: source.slice(start, end + 1), }; } diff --git a/packages/lina-web/client/markdown-types.ts b/packages/lina-web/client/markdown-types.ts index d320796..95cd8fb 100644 --- a/packages/lina-web/client/markdown-types.ts +++ b/packages/lina-web/client/markdown-types.ts @@ -27,6 +27,13 @@ export type MarkdownInlineToken = | { readonly type: "text"; readonly value: string } | { readonly type: "inline-code"; readonly value: string } | { readonly type: "strong"; readonly children: MarkdownInlineToken[] } + | { + readonly type: "attachment-image"; + readonly src: string; + readonly alt: string; + readonly sessionId: string; + readonly source: string; + } | { readonly type: "link"; readonly href: string; diff --git a/packages/lina-web/client/markdown.css b/packages/lina-web/client/markdown.css index 81b41a4..473c96a 100644 --- a/packages/lina-web/client/markdown.css +++ b/packages/lina-web/client/markdown.css @@ -73,6 +73,11 @@ .markdown-body a:hover { color: var(--accent-hover); } +.markdown-attachment-image { + display: block; + max-width: 100%; + height: auto; +} .markdown-inline-code { padding: 0.12em 0.35em; border: 1px solid var(--line); diff --git a/packages/lina-web/client/markdown.ts b/packages/lina-web/client/markdown.ts index 7875e81..5c51b75 100644 --- a/packages/lina-web/client/markdown.ts +++ b/packages/lina-web/client/markdown.ts @@ -136,6 +136,18 @@ function appendInline( parent.append(link); continue; } + if (token.type === "attachment-image") { + if (sessionId !== undefined && token.sessionId === sessionId) { + const image = dom.createElement("img"); + image.className = "markdown-attachment-image"; + image.setAttribute("alt", token.alt); + image.setAttribute("loading", "lazy"); + image.setAttribute("decoding", "async"); + image.setAttribute("src", token.src); + parent.append(image); + } else parent.append(dom.createTextNode(token.source)); + continue; + } if (sessionId !== undefined && token.sessionId === sessionId) { const link = dom.createElement("a"); link.className = "markdown-attachment-link"; diff --git a/packages/lina-web/scripts/qa-image-engine.ts b/packages/lina-web/scripts/qa-image-engine.ts new file mode 100644 index 0000000..dfb3094 --- /dev/null +++ b/packages/lina-web/scripts/qa-image-engine.ts @@ -0,0 +1,53 @@ +import { createImageAppFixture } from "../../lina-runtime/test/image-app-fixture.ts"; +import { loadWebAssets } from "../src/assets.ts"; +import { startWebServer } from "../src/server.ts"; + +const fixture = await createImageAppFixture(); +let web: ReturnType; +try { + web = startWebServer({ + port: 0, + upstream: `ws://127.0.0.1:${fixture.app.port}`, + assets: await loadWebAssets(), + }); +} catch (error) { + await fixture.close(); + throw error; +} + +console.log( + JSON.stringify({ + url: `http://127.0.0.1:${web.port}`, + root: fixture.root, + sessionId: fixture.app.binding.sessionId, + imageService: fixture.ima2.baseUrl, + externalProviderCalls: 0, + commands: ["generate", "edit", "generate fail"], + images: "320x200: blue generation, green edit", + }), +); + +let stopping: Promise | undefined; +function stop(): void { + if (stopping) return; + stopping = (async () => { + await web.stop(true); + await fixture.close(true); + console.log( + JSON.stringify({ + root: fixture.root, + generationRequests: fixture.ima2.submissions.length, + toolCalls: fixture.rpc.calls.map((call) => call.tool), + externalProviderCalls: 0, + }), + ); + })().catch((error: unknown) => { + console.error( + "[image-engine-qa]", + error instanceof Error ? error.message : String(error), + ); + process.exitCode = 1; + }); +} +process.once("SIGINT", stop); +process.once("SIGTERM", stop); diff --git a/packages/lina-web/test/markdown.test.ts b/packages/lina-web/test/markdown.test.ts index 358ec94..f0a5ded 100644 --- a/packages/lina-web/test/markdown.test.ts +++ b/packages/lina-web/test/markdown.test.ts @@ -261,3 +261,120 @@ test("empty list rows cannot bypass the DOM node budget", () => { const source = "- \n".repeat(5000); expect(parseMarkdown(source)).toEqual([{ type: "plain", value: source }]); }); + +const imageSessionId = "11111111-1111-4111-8111-111111111111"; +const imageAttachmentId = "22222222-2222-4222-8222-222222222222"; +const imageDownload = `/api/attachments/${imageAttachmentId}?sessionId=${imageSessionId}`; +const imagePreview = `/api/attachments/${imageAttachmentId}/preview?sessionId=${imageSessionId}`; + +test("renders a same-session generated image beside its download link", async () => { + await withFakeDocument((document) => { + const container = document.createElement("div"); + renderMarkdown( + container, + `완료\n\n![생성 이미지](${imagePreview})\n\n[다운로드](${imageDownload})`, + imageSessionId, + ); + + const images = container.find("img"); + expect(images).toHaveLength(1); + expect(images[0]?.getAttribute("src")).toBe(imagePreview); + expect(images[0]?.getAttribute("alt")).toBe("생성 이미지"); + expect(images[0]?.className).toBe("markdown-attachment-image"); + expect(images[0]?.getAttribute("loading")).toBe("lazy"); + expect( + container.find("a").map((link) => link.getAttribute("href")), + ).toEqual([imageDownload]); + expect(container.textContent).toContain("완료"); + }); +}); + +test.each([undefined, "", "33333333-3333-4333-8333-333333333333"])( + "keeps image syntax as text when the current session is %s", + async (sessionId) => { + await withFakeDocument((document) => { + const container = document.createElement("div"); + const source = `![생성 이미지](${imagePreview})`; + renderMarkdown(container, source, sessionId); + expect(container.find("img")).toHaveLength(0); + expect(container.find("a")).toHaveLength(0); + expect(container.textContent).toBe(source); + }); + }, +); + +test.each([ + "https://example.com/image.png", + `https://example.com${imagePreview}`, + `//example.com${imagePreview}`, + "data:image/png;base64,aGVsbG8=", + "javascript:alert(1)", + "blob:https://example.com/image", + "/image.png", + imageDownload, + imagePreview.replace("/preview", "/meta"), + imagePreview.replace(imageAttachmentId, "not-a-uuid"), + imagePreview.replace(imageSessionId, "not-a-uuid"), + imagePreview.replace("/preview", "/../preview"), + imagePreview.replace("/preview", "/%2e%2e/preview"), + imagePreview.replace("/preview", "%2fpreview"), + imagePreview.replace("/api/", "/api/../api/"), + imagePreview.replace("/api/", "\\api\\"), + imagePreview.replace("/api/", "/API/"), + imagePreview.replace("sessionId", "SessionId"), + imagePreview.replace("sessionId", "%73essionId"), + imagePreview.replace(imageSessionId, `%31${imageSessionId.slice(1)}`), + imagePreview.replace("?sessionId", "?other=1&sessionId"), + imagePreview.replace("?sessionId", "sessionId"), + `${imagePreview}&sessionId=${imageSessionId}`, + `${imagePreview}&other=1`, + `${imagePreview}?other=1`, + `${imagePreview}#fragment`, + `${imagePreview}%26other=1`, + `${imagePreview}&other=1`, + ` ${imagePreview}`, + `${imagePreview} `, + `${imagePreview}\t`, + `${imagePreview}\u2028`, + `${imagePreview} "title"`, + `<${imagePreview}>`, +])("preserves unsupported image destination %s as exact text", async (url) => { + await withFakeDocument((document) => { + const container = document.createElement("div"); + const source = `![생성 **이미지**](${url})`; + renderMarkdown(container, source, imageSessionId); + expect(container.find("img")).toHaveLength(0); + expect(container.find("a")).toHaveLength(0); + expect(container.textContent).toBe(source); + }); +}); + +test("uses image alt as literal text rather than HTML or nested Markdown", async () => { + await withFakeDocument((document) => { + const container = document.createElement("div"); + const alt = '생성 " onerror="alert(1) **이미지**'; + renderMarkdown(container, `![${alt}](${imagePreview})`, imageSessionId); + expect(container.find("img")).toHaveLength(1); + expect(container.find("img")[0]?.getAttribute("alt")).toBe(alt); + expect(container.find("img")[0]?.getAttribute("onerror")).toBeNull(); + expect(container.find("b")).toHaveLength(0); + expect(container.find("strong")).toHaveLength(0); + }); +}); + +test("keeps generated-image examples inert inside inline and fenced code", async () => { + await withFakeDocument((document) => { + const container = document.createElement("div"); + const source = `![생성 이미지](${imagePreview})`; + renderMarkdown( + container, + `\`${source}\`\n\n\`\`\`\n${source}\n\`\`\``, + imageSessionId, + ); + expect(container.find("img")).toHaveLength(0); + expect(container.find("code").map((code) => code.textContent)).toEqual([ + source, + `${source}\n`, + ]); + }); +});