Frizzle - Lightweight Event Bus

Overview

Frizzle is a minimalistic event bus designed to receive telemetry data and publish it to Kafka topics. The current development phase focuses on proving out JWT verification for secure data ingestion.

Containerization Design

Goals

1. Provide a consistent development environment
2. Support local Kafka integration
3. Enable future WebSocket to Kafka functionality
4. Maintain simplicity and scalability

Directory Structure

.docker/
├── Dockerfile
├── docker-compose.yaml
└── .env

Dockerfile

• Base image: Debian 12
• Includes:
  • Zsh with Oh-My-Zsh
  • Go 1.21+
  • Development tools (golangci-lint, delve)
  • Kafka client libraries
• Configured working directory
• Entrypoint setup

docker-compose (single source)

Services (profiles):

• frizzle (dev): Devcontainer for iterative development.
• app (ci): Production-like server image for CI/e2e.
• zookeeper: Zookeeper service.
• kafka: Kafka service (depends on Zookeeper).
• keycloak: Keycloak IdP (dev mode; imports frizzle realm).
• postgres: Postgres for Keycloak persistence.
• zoonavigator (dev): ZooKeeper GUI (dev convenience).

Service Startup Order

1. Zookeeper
2. Kafka
3. Postgres
4. Keycloak
5. Frizzle app

Environment Variables (.env)

• Kafka configuration (broker, topic)
• Application settings
• Development flags

Future Compatibility

• Pre-installed Kafka client
• Environment variables for Kafka configuration
• Network setup for Kafka communication
• Volume mounts for development and testing

Development Status

Refactor in progress to internal layering (auth, websocket, kafka, config, server). Basic endpoints implemented: /healthz, /readyz, /ws with bounded backpressure and Kafka publish interface.

Running the Application

Compose Quickstart

• Start stack (dev profile): make up
• Create topic (optional if auto-create): make setup-topics
• Health: curl http://localhost:8800/healthz
• Readiness: curl http://localhost:8800/readyz
• Stop stack: make down

To run the production-like app container (CI profile):

• Start stack: PROFILE=ci make up
• Stop stack: PROFILE=ci make down
• Build: make build
• Run server locally: CONFIG_PATH=configs/config.dev.yaml go run ./cmd/server
• Setup Kafka topic: make setup-topics
• Health: GET http://localhost:8800/healthz
• Readiness: GET http://localhost:8800/readyz
• WebSocket: GET ws://localhost:8800/ws (Authorization: Bearer )

When running via Compose (deployments/docker-compose.yml), the app is configured to validate JWTs against Keycloak at http://keycloak:8080/realms/frizzle using JWKS (/protocol/openid-connect/certs).

Testing

• Unit: make test
  • Runs go test ./... -race -cover with unit tests only.
• Integration (E2E): make itest
  • Brings up the CI profile from deployments/docker-compose.yml (ZK → Kafka → Postgres → Keycloak → app), ensures the Kafka topic, then runs tests under ./itest with -tags=integration.
  • The E2E test exercises: Keycloak ROPC token → WS auth → send → Kafka produce → Kafka consume → WS forward.
• Manual E2E (advanced):
  • PROFILE=ci make up then go test -race -tags=integration ./itest -v; tear down with PROFILE=ci make down.
• Lint/Vet/Format: make fmt, make vet, make lint, make tidy.

Policy (summary; see AGENTS.md for full requirements):

• Coverage targets: global ≥ 80%, critical paths ≥ 90% (auth, WS pumps, Kafka I/O).
• Devcontainer is canonical; tests should pass locally and in CI.

Troubleshooting Tests

• Zookeeper health: docker logs zookeeper --tail=80 and check srvr four-letter command is enabled; the compose healthcheck uses echo srvr | nc -w 2 localhost 2181 | grep -q Mode.
• Kafka listeners: host uses localhost:29092, containers use kafka:9092. If Kafka fails to start with dual listeners error, ensure ports and envs match the compose file.
• Kafka topic: ensure the topic exists before testing. go run ./cmd/setup -brokers localhost:29092 -topic frizzle -partitions 1 -replication 1 -timeout 30.
• Keycloak availability: root should respond 200/3xx at http://localhost:8080/. JWKS at http://localhost:8080/realms/frizzle/protocol/openid-connect/certs.
• JWKS warmup failed: confirm app is on the frizzle-net network and can reach keycloak:8080. Reset volumes with docker compose -f deployments/docker-compose.yml down -v to re-import realm.
• Token audience/issuer: tokens from localhost have iss=http://localhost:8080/realms/frizzle and aud=frizzle-app. App uses JWKS from keycloak and checks issuer localhost.
• WS handshake 401: verify Authorization header; ensure Keycloak client has Direct Access Grants enabled (realm import sets this).
• Architecture mismatch: exec format error indicates wrong image arch. Rebuild app with --no-cache --pull and ensure compose platform and build args align.
• Quick diagnostics:
  • App: docker logs app --tail=80, readiness: curl -v http://localhost:8800/readyz.
  • Kafka: docker logs kafka --tail=80, list topics via container: docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --list.
  • Keycloak: docker logs keycloak --tail=80.

CI Summary

• Static: gofmt -s, go vet, staticcheck, and go mod tidy must be clean (no diff).
• Unit: go test ./... -race -cover with coverage gates (global ≥ 80%, critical paths ≥ 90%).
• Integration: Compose boots; WS ↔ Kafka round-trip passes using real Keycloak JWKS.
• Security: No plaintext secrets committed; JWT/JWKS verification enforced; dependency scans if configured.
• Docs: README quickstart and docs/api.md match public endpoints and message schemas.
• Devcontainer: postCreate completes on the CI image.

Configuration (KISS defaults)

• Minimal config works with just Kafka and Auth basics in YAML (configs/config.*.yaml).
• JWKS and HTTP tuning have sane defaults; override via env if needed:
  • AUTH_JWKS_REFRESH_INTERVAL_SECS (default 60)
  • AUTH_JWKS_CACHE_TTL_SECS (default 300)
  • AUTH_HTTP_TIMEOUT_SECS (default 5)
  • AUTH_CLOCK_SKEW_SECS (default 0, sample uses 5)
  • AUTH_ALLOWED_ALGS (default RS256)

Example run with overrides:

AUTH_JWKS_REFRESH_INTERVAL_SECS=30 AUTH_HTTP_TIMEOUT_SECS=3 \
CONFIG_PATH=configs/config.dev.yaml go run ./cmd/server

Realm Users

The following users are generated for the Realm:

• admin-user:
  • Username: admin-user
  • Email: admin@frizzle.example
  • Password: admin123
  • Role: admin
• employee-user:
  • Username: employee-user
  • Email: employee@frizzle.example
  • Password: employee123
  • Role: employee
• customer-user:
  • Username: customer-user
  • Email: customer@frizzle.example
  • Password: customer123
  • Role: customer