Skip to content

Define per-backend connection schemas (Spark, Postgres, S3, Kafka…) #2

Description

@rafafrdz

Motivation

Depends on #1 (top-level `connections:` section).

Once `connections:` exists, each connection `type` needs a well-defined schema so validators, UIs, and runtimes can agree on required/optional fields.

Proposal

Define a reference schema per backend, starting with the ones we already support or plan to support short-term:

`spark`

field type required notes
url string `sc://host:port/`
app_name string default: "teckel"
connect_timeout_seconds int default: 30
request_timeout_seconds int default: 300
auth_token string bearer token if the cluster requires it

`postgres`

field type required
url string
user string
password string
schema string

`s3`

field type required
bucket string
region string
access_key_id string
secret_access_key string
endpoint string

`kafka`

field type required
bootstrap_servers string
security_protocol enum
sasl_mechanism enum
sasl_username string
sasl_password string

Scope

  • Document each schema in the spec.
  • Encode as JSON Schema so validators can check `connections.*` entries at pipeline-load time.
  • Mark fields that support `${VAR}` interpolation vs. literal-only.

Out of scope

  • Runtime implementation (lives in teckel-api).
  • UI editor (lives in teckel-ui).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions