Flechtwerk Examples

Easy + Reliable + Scalable => Productive

Complete, runnable examples for Flechtwerk — an async stream-processing framework for Kafka with real transactions for exactly-once delivery and an MQTT→Kafka bridge.

This repo complements the minimal snippets in the framework's own docs. The main repo keeps quickstart snippets, CI-tested via testcontainers; this repo carries full scenarios with real infrastructure, pinned to a released PyPI version and upgraded deliberately. It doubles as an integration test of the published package, exercised exactly the way a consumer would use it.

 

 

 

 

Seven of the examples, live in Grafana — ADS-B Flight Tracker, GDELT News Stories, GTFS German Rail Delays, SMARD German Electricity Market, Odds Arbitrage Radar, Wildfire Watch, and F1 Live Timing — plus the shared Observability dashboard watching every running stage.

What's Inside

A single batteries-included stack (docker compose up) plus one package per scenario under examples/:

#	Example	Shows off	Run
1	adsb_flight_tracker	a 3-stage Extractor→Transformer→Transformer pipeline (+ a config-driven boundary-loader stage): raw capture, spread-through projection, state-as-enrichment-cache (live Wikidata labels + a ClickHouse polygon-dictionary reverse geocoder), derived events, baby-TCAS conflict detection	adsb
2	clickhouse_sink	a sink transformer with honest at-least-once side-effect semantics (idempotent writes)	setup-sink → adsb + run-sink
3	chaos_harness	an executable exactly-once proof — SIGKILL a stage mid-batch, assert zero dupes/gaps	chaos
4	fermentation_monitor	the MqttExtractor bridge (ACK only after Kafka) + a stateful gravity monitor	fermentation
5	gdelt_news_stories	batch-file firehose ingestion (a resume-cursor Extractor over the GDELT 15-min feed), a co-partitioned Events⋈Mentions join with out-of-order buffering, online clustering of articles into stories in keyed state, and config-topic (GlobalKTable-style) outlet enrichment	gdelt
6	gtfs_german_rail_delays	a binary (protobuf) source decoded at the edge (Germany's GTFS-Realtime feed), a stream⋈static-dimension join (live delays ⋈ the schedule, co-partitioned by trip_id via a compacted profile topic), and a self-healing snapshot source — a live German long-distance rail delay monitor	gtfs
7	smard_german_electricity_market	late data / revisions: a resume-cursor Extractor that diffs each snapshot against a 48 h window to re-emit corrections, stream-time punctuation (the poller emits settled markers the transformer turns into a preliminary→final lifecycle + state tombstone), and a co-partitioned join whose key is time — Germany's live electricity generation mix, load, and day-ahead prices from Bundesnetzagentur SMARD.de	smard
8	odds_arbitrage_radar	N-source fan-in + event-time staleness: two independent extractors sharing one config topic, each polling a different venue into one pair-keyed quote stream, merged by a transformer into a per-pair best-price state that flags cross-venue arbitrage — live Polymarket × Kalshi prediction-market odds, with a fresh net-positive edge (after fees) as the signal. Read-only, keyless public data	odds
9	wildfire_watch	spatiotemporal sessionization + bounded event-time dedupe: NASA FIRMS satellite fire detections clustered into persistent fire objects in keyed state (ignition → growth → merge → extinction by event-time timeout, with sweep markers as the transformer's only clock), and a pruned, hard-capped seen-set standing in for a cursor the source doesn't have — a live wildfire map from VIIRS, regional or planet-wide (request-wildfire world tiles the globe adaptively into a few hundred ordinary regions). Free NASA FIRMS_MAP_KEY required (the repo's first)	wildfire
10	f1_live_timing	an append-only file as an exactly-once source: Formula 1's public live-timing archive records each session as line-framed .jsonStream files, so a per-feed byte offset is the resume cursor — making backfill, live tailing, and crash recovery one code path, and making downtime cost timeliness but never data. Plus delta accumulation into a materialized view (the feed sends partial patches, never a leaderboard; the board folds them into keyed state and emits wide snapshots only when something changed) and a broadcast SCD join (a 585-byte, 12-record flag feed tags every lap). The only deterministic, replayable example here — run it on a quiet Tuesday and get the Hungarian Grand Prix, byte for byte — and its dashboards make live, scrub, and animated replay one as-of query with no timestamp ever rewritten. Keyless public data, read-only, unofficial endpoint	f1

Each example is self-contained under its own directory with its own README.

Requirements

• Python 3.14 and uv — pinned via requires-python in pyproject.toml.
• Docker — for the shared stack and the integration test tier.

Quickstart

uv sync                 # create the venv and install the pinned dependencies
uv run poe up           # start the shared stack, wait until healthy
uv run poe adsb         # set up + run the ADS-B flight tracker (stays in the foreground)
uv run poe request-region "Brussels" 250   # (second terminal) track a region; radius in nautical miles
uv run poe down         # stop the stack, preserving its volumes (resume later)
uv run poe clean        # stop the stack AND wipe its volumes (a full reset)

The tasks above are poethepoet targets declared in pyproject.toml; run uv run poe with no argument to list them all.

The Stack

docker compose up (or uv run poe up) brings up six long-running services (plus two one-shots that exit: kafka-init chowns the Kafka volume, and kafka-features turns off ELR on the single broker — see the note in docker-compose.yaml):

Service	URL / port	What it's for
Kafka	localhost:9092	the Kafka broker (KRaft, single node); examples connect here
Kafbat UI (web UI for Kafka)	http://localhost:8080	browse topics, messages, consumer groups
Mosquitto	localhost:1883	MQTT broker for the bridge examples
ClickHouse	http://localhost:8123 (HTTP), localhost:9000 (native)	OLAP sink for every example's output (database flechtwerk)
Prometheus	http://localhost:9090	scrapes each running stage's metrics endpoint
Grafana	http://localhost:3000	provisioned dashboards (anonymous access)

Grafana provisions dashboards tagged flechtwerk: Observability (the flechtwerk_* Prometheus metrics — throughput, undecodable records by topic and by what the invalid-message policy did with them, record and state sizes against the ~1 MiB record ceiling, latency, extractor poll cycles, MQTT, config store, transformer tasks & state restore, process health — filterable by example, stage, and client), Stream Data (a ClickHouse datasource smoke test), and per-example dashboards — ADS-B Flight Tracker (a live map + enriched table), ADS-B Aviation Events (emergencies, rapid descents, going-dark, near-misses), Fermentation Monitor (gravity curves + alerts), GDELT News Stories (breaking-news velocity, top stories, a tone-coloured world map, coverage spread), GTFS German Rail Delays (a delay-coloured map of ICE/IC trains, network punctuality, most-delayed trains, a network-delay timeseries), SMARD German Electricity Market (the generation mix by source, day-ahead price into tomorrow, renewables share and CO₂ intensity, and a live corrections feed of revised values), Odds Arbitrage Radar (the net edge after fees per pair and direction, a closest-to-free-money leaderboard, gross vs. net, the two venues' YES asks, signals, and quote freshness), Wildfire Watch (a fire map with markers sized by detection count and coloured by radiative power over the raw hotspot pixels, total FRP per region, the largest active fires, the ignition/merge/extinction log, and a poll-heartbeat panel that distinguishes "nothing burning" from "poller stopped"), and the three F1 dashboards — F1 Live Timing (the race wall: the leaderboard as of the cursor, track status, lap clock, tape freshness, gap-to-leader and lap times with flag annotations, the battle radar, pit stops, race control, weather, speed traps, a track map), F1 Strategy (stint timeline, tyre degradation, position by lap, rolling pace, the pit-loss ledger) and F1 Season (every session on tape with one click to open or replay it, championship points, podiums, season tallies) — which share one as-of cursor so that live, scrub, and animated replay are the same query.

Stages run on the host and expose Prometheus metrics on a per-example port (9101 ADS-B ingest + 9105 ADS-B enrich + 9106 ADS-B conflict + 9107 ADS-B boundary loader, 9102 sink, 9103 fermentation monitor + 9104 fermentation bridge, 9108 GDELT ingest + 9109 GDELT coverage + 9110 GDELT stories + 9111 GDELT sink, 9112 GTFS ingest + 9113 GTFS delays + 9114 GTFS loader, 9115 SMARD ingest + 9116 SMARD mix, 9117 odds Polymarket + 9118 odds Kalshi + 9119 odds radar, 9120 wildfire ingest + 9121 wildfire tracker, 9122 F1 ingest + 9123 F1 timing; the chaos harness runs metrics-off); Prometheus reaches them via host.docker.internal, so a target reads "down" until you start its example.

Testing

Every example ships tests in three tiers, mirroring the framework's own suite:

1. Pure-logic tier — no framework, no mocks. A stage is a plain async generator: build a State, drive the generator, assert on the yielded Message/State. This is the two-yield contract's biggest practical payoff.
2. Runner tier — the shipped flechtwerk.testing fakes (InMemoryStateStore, FakeKafkaConsumer/FakeKafkaProducer, FakeMqttConnection, make_record, RecordingObserver), never parallel scaffolding.
3. Integration tier — ephemeral Kafka/Mosquitto/ClickHouse via testcontainers, marked integration and run with -m integration.

uv run poe test               # tiers 1 + 2 — Docker-free
uv run poe test-integration   # tier 3 — needs Docker
uv run poe test-all           # everything (what CI runs)
uv run poe cov                # everything, with a coverage report

The Docker-free tiers require no services running. The integration tier starts its own containers and skips cleanly when Docker is unreachable.

Issues and Contributions

This repo is the home for Flechtwerk's runnable examples (they deliberately live here, not in the framework repo). Found a bug, or have a scenario you'd like to see? Open an issue or PR here. For the framework itself, use the flechtwerk issue tracker.

Versioning Policy

flechtwerk is pinned to an exact released version in pyproject.toml (flechtwerk[mqtt]==0.9.1) with the full resolution captured in uv.lock — never a path or git dependency; the Docker images are pinned to specific tags too. Upgrades are deliberate: bump the pins, relock, and let the tests and a live end-to-end pass verify the new release.

License

MIT, the same as the framework.