Skip to content

hscale/duradx-oss

BranchesTags

Repository files navigation

DuraDX

Durable workflow engine for Rust. Embed it as a library or run it as a standalone server.

Write workflows as async Rust functions or define them in YAML/JSON. Every step is checkpointed to the database. If the process crashes, restarts, or redeploys, workflows resume from the last completed step. No work is lost. No step runs twice.

Why DuraDX

  • No external dependencies -- the database is the only infrastructure. No Redis, no message broker, no separate state store.
  • Two ways to define workflows -- programmatic (Rust closures) or declarative (YAML/JSON). Both get the same durability guarantees.
  • Five database backends -- PostgreSQL, CockroachDB, SQLite, RocksDB, FoundationDB. Pick the one that fits your deployment. Switch backends by changing the connection URL.
  • Built for AI workloads -- LLM step handler, tool invocation, step output streaming, DAG-based pipelines with bounded parallelism.

Quickstart

[dependencies]
duradx = { version = "0.1" }
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
use duradx::prelude::*;

#[tokio::main]
async fn main() -> Result<()> {
    let config = Config::new("my-app", "postgres://localhost:5432/mydb");
    let ctx = DuradxContext::new(config).await?;

    ctx.register_workflow(
        "greet",
        |wf_ctx: WorkflowContext, name: String| async move {
            let msg: String = wf_ctx.run_as_step(
                "build_greeting",
                &StepConfig::default(),
                || async move { Ok(format!("Hello, {name}!")) },
            ).await?;
            Ok(msg)
        },
        50,
        None,
    );

    ctx.launch().await?;

    let handle = ctx.run_workflow("greet", Some(serde_json::to_string("world")?), &[]).await?;
    let status = handle.get_result().await?;
    println!("{:?}", status.output);

    ctx.shutdown().await;
    Ok(())
}

Or define the same workflow in YAML:

name: greet
steps:
  - name: build_greeting
    type: log
    config:
      message: "Hello, ${input}!"
duradx workflow run --file greet.yaml --input '"world"'

Features

Workflows and Steps

Register async Rust functions as durable workflows. Every side-effect is wrapped in a step -- the atomic checkpoint unit. Steps support configurable exponential-backoff retries, parallel execution (go_step), and racing (select_step).

On crash recovery, completed steps return cached results instantly. Only the incomplete step re-executes.

Declarative Workflows (YAML/JSON)

Define workflows in YAML or JSON without writing Rust code. Two execution modes:

Mode Field Description
Sequential (default) steps Steps execute in order, each receiving prior step outputs
DAG type: dag + nodes Nodes execute in topological order with bounded parallelism

10 built-in step types: http, transform, condition, sleep, log, parallel, loop, workflow, llm, tool

Expression syntax: ${input.field}, ${steps.prev.output}, ${env.VAR}

Sequential example:

name: order_processor
steps:
  - name: validate
    type: http
    config:
      method: POST
      url: "https://api.example.com/validate"
      body: { order_id: "${input.order_id}" }
  - name: charge
    type: http
    config:
      method: POST
      url: "https://api.example.com/charge"
      body: { amount: "${steps.validate.output.body.total}" }

DAG example:

name: etl_pipeline
type: dag
max_parallel: 8
nodes:
  - id: extract_users
    type: http
    config: { method: GET, url: "https://api.example.com/users" }
  - id: extract_products
    type: http
    config: { method: GET, url: "https://api.example.com/products" }
  - id: load
    type: http
    depends_on: [extract_users, extract_products]
    config:
      method: POST
      url: "https://api.example.com/warehouse/load"

Queues

Named workflow queues with per-worker and global concurrency limits, token-bucket rate limiting, priority ordering, partition-key isolation, deduplication, debouncing, and backpressure (max_depth).

Communication

  • send / recv -- durable FIFO messages between workflows by topic
  • set_event / get_event -- durable key-value signals readable by any workflow
  • sleep -- durable sleep that resumes correctly after crash recovery
  • Durable append-only streams with multi-reader support

Scheduling

Cron-based scheduling with 5-field expressions. Missed executions are caught up on restart. Each fire creates an independent workflow instance.

Multi-Database Backends

Backend Feature Flag URL Scheme Notes
PostgreSQL postgres (default) postgres:// Production-grade. LISTEN/NOTIFY.
CockroachDB cockroachdb cockroachdb:// Distributed SQL. Polling fallback.
SQLite sqlite sqlite:// Embedded, zero-config.
RocksDB rocksdb rocksdb:// Embedded KV. High throughput.
FoundationDB foundationdb fdb:// Distributed KV. ACID.

Backend is selected automatically from the database_url scheme at runtime.

DAG Workflows

Define workflows as directed acyclic graphs via the Rust API (register_dag_workflow) or YAML/JSON (type: dag). Nodes declare dependencies and execute in topological order with bounded parallelism. Per-node failure policies (FailFast / Continue), path-based step IDs for replay-safe checkpointing, and dynamic step expansion.

Distributed Workers

Worker registry with capabilities, heartbeat, and step leases. Workers acquire time-limited locks on steps to prevent duplicate execution across processes. Expired leases are automatically reclaimed.

Admin HTTP API

Axum-based HTTP server starts with launch(). Key endpoints:

Method Path Description
GET /duradx-healthz Health check
POST /workflows List/filter workflows
GET /workflows/{id} Get workflow detail
GET /workflows/{id}/steps Step execution history
POST /workflows/{id}/cancel Cancel
POST /workflows/{id}/resume Resume
POST /workflows/{id}/fork Fork from step
POST /enqueue Enqueue for deferred execution
POST /workflow/run Run inline declarative workflow (sequential or DAG)
GET /workflows/{id}/state Progress info
GET /workflows/{id}/events Event log
GET /workers List workers

CLI

duradx init [name]                  Scaffold a new project
duradx start                        Start via duradx-config.yaml
duradx postgres start|stop          Local PostgreSQL via Docker

duradx workflow list                List workflows (with filters)
duradx workflow get <id>            Workflow details
duradx workflow steps <id>          Step history
duradx workflow cancel <id>         Cancel
duradx workflow resume <id>         Resume
duradx workflow fork <id>           Fork from step
duradx workflow validate --file     Validate YAML/JSON definition
duradx workflow run --file --input  Run declarative workflow

duradx db migrate                   Run migrations
duradx db status                    Schema version and table sizes
duradx db export / import           NDJSON export/import
duradx db backup / restore          Native backup/restore
duradx db seed                      Load seed data
duradx db drop / reset              Drop or reset schema

Console UI

SvelteKit 5 web dashboard:

  • Dashboard -- status counts, recent workflows, health
  • Workflows -- list, filter, bulk actions, detail with compact/timeline/graph/events/raw tabs
  • Queues -- queue list with metadata
  • Schedules -- cron schedules with next-run estimation
  • Workers -- registered workers with utilization
  • Admin -- recovery, global timeout, garbage collect
  • Settings -- theme, API endpoint, refresh interval
  • Command palette -- Cmd+K / Ctrl+K

External Client

Connect to the database without running the runtime. Enqueue, monitor, cancel, resume, and fork workflows from CLIs, sidecars, or external services.

Security

Bearer token authentication for the admin API. Role-based access control (RBAC) with per-workflow permissions (execute, read, cancel, admin).

Requirements

  • Rust 1.75+ (stable)
  • tokio async runtime
  • PostgreSQL: 14+ (or CockroachDB)
  • SQLite: no external dependency
  • RocksDB: C++ toolchain (compiled from source)
  • FoundationDB: 7.1+ C client library
  • Docker (optional, for duradx postgres start)

Project Structure

src/
  lib.rs              Public API and prelude
  config.rs           Configuration
  context.rs          DuradxContext (runtime handle)
  workflow.rs         Workflow engine
  step.rs             Step execution and retries
  communication.rs    send/recv, events, sleep
  stream.rs           Durable streams
  queue.rs            Workflow queues
  recovery.rs         Recovery, cancel, resume, fork
  scheduler.rs        Cron scheduling
  client.rs           External client
  admin.rs            HTTP admin server (Axum)
  conductor.rs        Conductor WebSocket client
  declarative/        YAML/JSON workflow engine (sequential + DAG)
  dag/                DAG scheduler, graph validation, dynamic expansion
  worker/             Distributed workers, step leases
  security/           API auth, RBAC
  observability/      Workflow event logging
  db/                 Multi-backend database layer
  bin/
    duradx.rs         CLI
    dev_server.rs     Development server
duradx-console/       SvelteKit web UI
docs/                 Documentation (27 files)
tests/                Integration tests
examples/             Example app and workflow files

Environment Variables

Variable Description
DURADX_DATABASE_URL Database connection string
DURADX_ADMIN_PORT Admin server port (default: 3001)
DURADX__APPVERSION Application version tag
DURADX__VMID Executor identifier
RUST_LOG Log level (debug, info, warn, error)

Documentation

Document Description
Architecture System design, checkpoint model, database schema
Configuration Config fields, environment variables, defaults
Backends Backend comparison, setup guides, feature matrix
Workflows Registration, execution, handles, status types
Steps Step execution, retries, concurrent steps
Communication send/recv, events, durable sleep
Streams Durable append-only streams
Queues Flow control, rate limiting, priority, partitioning
Recovery Startup recovery, cancel, resume, fork
Scheduler Cron-based workflow scheduling
Declarative Workflows YAML/JSON definitions (sequential + DAG)
DAG Workflows DAG scheduling, Rust API, YAML/JSON
Workers Distributed workers, step leases
Client External client for enqueue/monitor/control
Admin API HTTP endpoints reference
Security API auth, RBAC
Observability Event logging, tracing
CLI Command-line reference
Error Handling Error types and retry semantics
AI Integration Guide Complete guide for AI agents
Patterns & Recipes Real-world patterns: saga, fan-out, approval gates, ETL
Operations Guide Production ops: monitoring, scaling, DR, troubleshooting
Console Web UI
Development Local dev setup
Deployment Production deployment

License

MIT

Built by DuraDX Contributors

About

No description, website, or topics provided.

Resources

Readme

License

MIT license

Security policy

Security policy
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages