Problem
Today input: requires format + path pointing to a physical file (csv/parquet/json). This makes it impossible to:
- Run a self-contained demo against a remote worker (e.g. Fly) that has no data on disk
- Write spec-level examples that execute without external fixtures
- Quickly prototype transformations without provisioning files
Proposal
Add an inline (a.k.a. literal / values) input format that embeds the rows directly in the YAML:
input:
name: sales
format: inline
schema:
- { name: order_id, type: int }
- { name: product, type: string }
- { name: quantity, type: int }
- { name: price, type: double }
rows:
- [1, "Book", 2, 15.0]
- [2, "Laptop", 1, 900.0]
- [3, "Pen", 10, 1.5]
Alternative row syntax (object form) also acceptable:
rows:
- { order_id: 1, product: "Book", quantity: 2, price: 15.0 }
Scope
- Spec: define
inline format, required schema, supported primitive types, row validation rules
- Backends: DataFusion (
MemTable), Polars (DataFrame::new), Spark (createDataFrame)
- Size guardrail: recommend cap (e.g. 1k rows) — this is for demos/tests, not production ingestion
Motivation
Unblocks end-to-end demos against deployed workers without needing volumes, uploads, or bundled fixtures.
Problem
Today
input:requiresformat+pathpointing to a physical file (csv/parquet/json). This makes it impossible to:Proposal
Add an
inline(a.k.a.literal/values) input format that embeds the rows directly in the YAML:Alternative row syntax (object form) also acceptable:
Scope
inlineformat, requiredschema, supported primitive types, row validation rulesMemTable), Polars (DataFrame::new), Spark (createDataFrame)Motivation
Unblocks end-to-end demos against deployed workers without needing volumes, uploads, or bundled fixtures.