Skip to content

Add inline/literal input format for demos and testing #4

Description

@rafafrdz

Problem

Today input: requires format + path pointing to a physical file (csv/parquet/json). This makes it impossible to:

  • Run a self-contained demo against a remote worker (e.g. Fly) that has no data on disk
  • Write spec-level examples that execute without external fixtures
  • Quickly prototype transformations without provisioning files

Proposal

Add an inline (a.k.a. literal / values) input format that embeds the rows directly in the YAML:

input:
  name: sales
  format: inline
  schema:
    - { name: order_id, type: int }
    - { name: product,  type: string }
    - { name: quantity, type: int }
    - { name: price,    type: double }
  rows:
    - [1, "Book",   2, 15.0]
    - [2, "Laptop", 1, 900.0]
    - [3, "Pen",    10, 1.5]

Alternative row syntax (object form) also acceptable:

rows:
  - { order_id: 1, product: "Book", quantity: 2, price: 15.0 }

Scope

  • Spec: define inline format, required schema, supported primitive types, row validation rules
  • Backends: DataFusion (MemTable), Polars (DataFrame::new), Spark (createDataFrame)
  • Size guardrail: recommend cap (e.g. 1k rows) — this is for demos/tests, not production ingestion

Motivation

Unblocks end-to-end demos against deployed workers without needing volumes, uploads, or bundled fixtures.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions