Go scheduler that keeps work on disk, runs it at-least-once, and tries not to lie when something crashes mid-flight.
I started with JSON files on disk. That was easy to debug and slow under contention. Tenant B’s p99 sat around 3s when A flooded the workers. SQLite brought that down to roughly 300 to 350ms in the same noisy-neighbor test. About 10x. Good enough to ship. Not Redis-fast, and I’m fine saying that out loud.
- Tasks land in SQLite (
data/scheduler.db) before they count as accepted. - Execution is at-least-once. Retries and dead-letter are explicit.
- A crashed process can lose the in-memory queue. It should not lose persisted tasks. On boot we rebuild from schedules.
- Tenants get their own queues, rate limits, and an optional
MaxInFlightcap so one loud tenant cannot occupy every worker. - There is a kill-switch. Pause execution, keep the work.
What it does not promise: exactly-once, multi-region anything, or Temporal-shaped workflows. Those were cut on purpose.
Lease expiry is how we notice a dead worker. Default visibility is 30s; heartbeats keep the lease alive.
That creates a classic mess. Worker A stalls. Lease expires. Worker B takes the task. Then A wakes up and tries to write success.
So leases carry a generation token. Acquire bumps it. Completion paths (PutTaskIfLease, PutAttemptIfLease, PutScheduleIfLease) check worker id and generation. Stale writers get ErrStaleLease and their write does not stick. There is a test where a zombie finishes “successfully” and the store still rejects it.
Fair dequeue across tenants helps. It is not enough by itself.
MaxInFlight is the hard knob. Set tenant A to 1 and two workers will not run two A tasks at once; the spare worker picks up B. Soft latency win in the isolation test is smaller, around 1.2x when A is capped. The hard “never more than N in flight” check is the part I trust more than the p99 story.
SQLite via modernc.org/sqlite, WAL mode. Short busy timeout, immediate transactions, and retries at the transaction boundary so multi-connection writers do not die on BUSY_SNAPSHOT.
I kept one database file as the ledger. Queueing still happens in memory after restore. If I wanted another step-change in speed I would put the hot queue somewhere else and leave SQLite as the source of truth. That is a different project.
Auth is bearer tokens mapped to tenants (token-tenant-a, token-tenant-b in the local binary).
| Method | Path | Notes |
|---|---|---|
| POST | /v1/tasks |
Enqueue. Idempotent on (tenant, Idempotency-Key). |
| GET | /v1/tasks/{taskID} |
Status |
| POST | /v1/tasks/{taskID}/cancel |
Cancel |
| GET/PUT | /v1/execution-control |
Kill-switch. Body like {"paused":true} |
| GET | /healthz |
Liveness |
Under sustained overload, enqueue fails fast instead of filling disk forever. Prefer a loud 429/503 over a quiet meltdown.
go run ./cmd/server -data data -addr :8080
# flip kill-switch without standing up HTTP
go run ./cmd/server -data data -set-execution-paused=true
go run ./cmd/server -data data -set-execution-paused=falseUseful flags: -tenant-max-queued, -tenant-rate-per-sec, -tenant-rate-burst.
go test ./...Worth running if you care about the claims above: fencing (zombie_fencing_test.go), MaxInFlight, and the noisy-neighbor isolation test.
- At-least-once over exactly-once. Coordinating global exactly-once was out of scope.
- SQLite over a broker. One process, one file, real durability lessons. Writer serialization is the ceiling.
- Time-based leases over process probes. Simple. Sensitive to clock weirdness and long GC pauses.
- In-memory queues restored at startup. Live cross-process handoff was not the hill I died on for v1.
Built a multi-tenant durable task scheduler in Go with SQLite persistence, lease fencing so stale workers cannot commit, and per-tenant concurrency caps. Moved noisy-neighbor p99 from ~3s (JSON files) to ~300 to 350ms. Validated the fencing and in-flight guarantees with tests.
If you only read one design choice: the generation token on the lease. Everything else is setup for that moment where a dead worker comes back wrong.