Cleave

A self-hosted CUT&RUN/CUT&Tag bioinformatics web platform for the Ferguson Lab at UCSD. Cleave replicates EpiCypher's CUTANA Cloud and extends it with lab-specific pipeline features -- trimming, SEACR peak calling, MACS2 broad mode, DiffBind differential analysis, custom heatmaps, Pearson correlation, Roman normalization, and auto-pipeline mode.

Built for ~8-10 lab members. Runs on a single AWS EC2 instance. Live at cleave.nazalibhai.com. 525+ backend tests passing.

What It Does

FASTQ upload (tus resumable + FTP/SFTP import) → FastQC → Trimming (Trimmomatic + kseq 42bp) → Alignment (Bowtie2 + SAMtools + BEDTools + Picard + deepTools) → Peak Calling (MACS2 / SICER2 / SEACR + HOMER annotation) → Visualization (IGV.js + heatmaps) → Lab Extensions (DiffBind, correlation, normalization) → File download

Auto-pipeline mode chains FastQC, Trimming, Alignment, and Peak Calling into a single one-click operation.

Feature	CUTANA Cloud	Cleave
FASTQ upload + FastQC	Yes	Yes
FTP/SFTP server import	Yes	Yes
Bowtie2 alignment + QC	Yes	Yes
MACS2 narrow peaks	Yes	Yes
SICER2 broad peaks	Yes	Yes
SEACR peak calling	-	Yes
MACS2 broad mode	-	Yes
FASTQ trimming (Trimmomatic + kseq)	-	Yes
Fragment size filter (<120bp)	-	Yes
DiffBind differential analysis	-	Yes
Custom reference-point heatmaps	-	Yes
Pearson correlation matrices	-	Yes
Roman normalization	-	Yes
SNAP-CUTANA spike-in QC	Yes	Yes
E. coli spike-in normalization	Yes	Yes
IGV.js genome browser	Yes	Yes
Auto-generated methods text	Yes	Yes
One-click auto-pipeline	-	Yes
Parallel pipeline processing	-	Yes
Dark mode	-	Yes
Superuser admin panel	-	Yes
In-app documentation	-	Yes
Self-hosted (no per-credit cost)	-	Yes

Tech Stack

Layer	Technology
Frontend	React 18 (Vite), TypeScript, Tailwind CSS, shadcn/ui (Radix primitives), TanStack Table, TanStack Query, Recharts, IGV.js, tus-js-client, lucide-react, next-themes, sonner
Backend	FastAPI (Python 3.11+), Uvicorn, SQLAlchemy 2.0 (async), Alembic, Pydantic v2, tuspyserver, aioftp, asyncssh
Database	PostgreSQL 15+
Auth	fastapi-users (JWT access 30-min + httpOnly refresh cookie 7-day), Argon2, slowapi rate limiting
Pipeline	Python worker process calling Bowtie2, SAMtools, BEDTools, Picard, deepTools, MACS2, SICER2, SEACR, HOMER, Trimmomatic, DiffBind (R) via subprocess
Real-time	SSE (server-sent events) with @microsoft/fetch-event-source for JWT-authenticated streaming
Dev	Docker Compose (Postgres + FastAPI + Worker + Vite), local dev script (`scripts/run-local.sh`)
Prod	NGINX reverse proxy, systemd, single EC2 instance

Quick Start

Prerequisites

Docker and Docker Compose
Node.js 20+ (for local frontend dev outside Docker)
Python 3.11+ (for local backend dev outside Docker)

Run with Docker Compose

git clone <repo-url> cleave && cd cleave

# Start all services (Postgres + FastAPI + Vite dev server)
docker compose up -d

# Generate and apply database migrations
docker compose exec api alembic upgrade head

# Verify
curl http://localhost:8000/api/v1/health    # {"status":"ok"}
open http://localhost:5173                   # React app
open http://localhost:8000/docs              # OpenAPI docs

Run Locally (without Docker)

# Backend
cd backend
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
cp ../.env.example .env   # edit DATABASE_URL to point to your Postgres
alembic upgrade head
uvicorn main:app --reload --host 0.0.0.0 --port 8000

# Frontend (separate terminal)
cd frontend
npm install
npm run dev

Project Structure

cleave/
├── backend/
│   ├── main.py                  FastAPI app entry point
│   ├── auth.py                  fastapi-users config (UserManager, JWT strategies)
│   ├── config.py                Pydantic Settings (reads .env)
│   ├── database.py              Async SQLAlchemy engine + session
│   ├── dependencies.py          FastAPI Depends (auth, permissions)
│   ├── logging_config.py        structlog setup (JSON in prod, console in dev)
│   ├── worker.py                Standalone job queue worker
│   ├── models/                  SQLAlchemy 2.0 ORM models (11 tables)
│   ├── schemas/                 Pydantic v2 request/response schemas
│   ├── routers/                 FastAPI route handlers (14 routers)
│   ├── services/                Business logic layer (23 services)
│   ├── migrations/              Alembic migrations (12 versioned migrations)
│   ├── pipelines/               Pipeline modules + reference data
│   │   ├── adapters/            Trimmomatic adapter FASTAs
│   │   ├── reference/           Blacklists, chrom sizes, masks, annotations
│   │   ├── scripts/             R scripts (DiffBind, Pearson, Roman normalization)
│   │   └── tools/               SEACR, kseq_test, filter_below.awk, change.bdg.py
│   └── tests/                   pytest tests (500+ passing)
│
├── frontend/
│   └── src/
│       ├── api/                 Axios client + API modules
│       ├── components/          Reusable UI components
│       │   ├── layout/          Navbar, Breadcrumbs, Card, GradientBackground
│       │   ├── ui/              shadcn/ui primitives, DataTable, WizardModal, EmptyState
│       │   ├── auth/            ProtectedRoute
│       │   ├── alignment/       Alignment wizard + QC report
│       │   ├── peak-calling/    Peak calling wizard + annotation charts
│       │   ├── diffbind/        DiffBind wizard + results (volcano, MA, PCA plots)
│       │   ├── igv/             IGV.js genome browser wrapper
│       │   ├── custom-heatmap/  Custom heatmap wizard
│       │   ├── pearson-correlation/  Correlation matrix UI
│       │   ├── normalization/   Roman normalization UI
│       │   └── docs/            In-app documentation components
│       ├── contexts/            AuthContext, ThemeProvider (dark mode)
│       ├── hooks/               TanStack Query hooks for all API resources
│       ├── lib/                 Constants, utilities, cn() helper, docs content
│       └── pages/               Route-level pages, LandingPage, AdminPage, DocsPages
│
├── scripts/                     Dev scripts (run-local.sh)
├── docs/                        Architecture, specs, UI reference, decisions
├── references/                  Lab pipeline scripts (read-only reference)
├── cutana/                      Exported CUTANA Cloud QC data (reference schemas)
├── test_data/                   Downsampled FASTQs for local testing
├── docker-compose.yml           Dev environment (db + api + worker + frontend)
└── .env.example                 Environment variable template

Database Schema

11 tables managed via 12 Alembic migrations:

users -- accounts extending fastapi-users base model (Argon2 password hashes)
projects -- top-level organizational containers
project_members -- role-based access (admin / contributor / viewer)
experiments -- CUT&RUN or CUT&Tag analysis units within a project
fastq_files -- uploaded paired-end FASTQ metadata (adapter status, trimming state)
reactions -- sample metadata linked to FASTQs (organism, antibody, spike-in)
analysis_jobs -- unified job queue (JSONB params, parent_job_id for dependency chains, termination/retry support, auto-pipeline tracking)
job_outputs -- files produced by analysis jobs
notifications -- in-app alerts (job completion, project invitations)
saved_servers -- FTP/SFTP server credentials (Fernet-encrypted passwords)
experiment_events -- audit log entries for experiment history tracking

API

68+ endpoints across 14 routers under /api/v1/. JWT required except /api/v1/auth/* and /api/v1/health.

# Auth & Users
POST   /api/v1/auth/login|register|refresh|logout
POST   /api/v1/auth/forgot-password|reset-password
GET    /api/v1/users/me
PATCH  /api/v1/users/me

# Projects & Members
GET    /api/v1/projects                     # paginated, member-filtered, 5 filter params
POST   /api/v1/projects
GET    /api/v1/projects/:id
PATCH  /api/v1/projects/:id                 # admin only
DELETE /api/v1/projects/:id                 # admin only (disk cleanup)
GET    /api/v1/projects/reference            # reference projects (all authenticated users)
GET    /api/v1/projects/:id/members
POST   /api/v1/projects/:id/members         # admin only
PATCH  /api/v1/projects/:id/members/:uid    # change role
DELETE /api/v1/projects/:id/members/:uid    # remove member

# Experiments
GET    /api/v1/experiments                   # ?projectId= filter
POST   /api/v1/experiments?projectId=
GET    /api/v1/experiments/:id
PATCH  /api/v1/experiments/:id
DELETE /api/v1/experiments/:id               # disk cleanup
GET    /api/v1/experiments/:id/history       # paginated audit log
POST   /api/v1/experiments/:id/auto-pipeline         # one-click full pipeline
POST   /api/v1/experiments/:id/auto-pipeline/cancel
POST   /api/v1/experiments/:id/auto-pipeline/retry

# FASTQ Files & Upload
GET    /api/v1/experiments/:id/fastqs
POST   /api/v1/experiments/:id/fastqs/upload          # multipart upload (legacy)
POST/PATCH/DELETE/GET /api/v1/experiments/:id/tus/*    # tus v1.0.0 resumable upload
DELETE /api/v1/experiments/:id/fastqs/:fid
GET    /api/v1/experiments/:id/fastqs/:fid/fastqc     # FastQC HTML report
GET    /api/v1/experiments/:id/fastqs/:fid/fastqc-token
GET    /api/v1/experiments/:id/fastqs/:fid/fastqc-summary

# Reactions
GET    /api/v1/experiments/:id/reactions
POST   /api/v1/experiments/:id/reactions
POST   /api/v1/experiments/:id/reactions/bulk
POST   /api/v1/experiments/:id/reactions/import-csv
GET    /api/v1/experiments/:id/reactions/template
GET    /api/v1/experiments/:id/reactions/prefixes
PATCH  /api/v1/experiments/:id/reactions/:rid
DELETE /api/v1/experiments/:id/reactions/:rid

# Analysis Jobs & Outputs
POST   /api/v1/experiments/:id/jobs          # submit job (any of 7 types)
GET    /api/v1/experiments/:id/jobs
GET    /api/v1/jobs                           # cross-project queue (filterable)
GET    /api/v1/jobs/:jid
PATCH  /api/v1/jobs/:jid                     # update notes
POST   /api/v1/jobs/:jid/terminate
POST   /api/v1/jobs/:jid/retry
GET    /api/v1/jobs/:jid/log-tail            # last N lines of pipeline log
GET    /api/v1/jobs/:jid/outputs             # list outputs (category filter)
GET    /api/v1/jobs/:jid/outputs/:oid/signed-url

# QC Reports & Pipeline Results
GET    /api/v1/jobs/:jid/qc-report           # alignment QC
GET    /api/v1/jobs/:jid/peak-qc-report      # peak calling QC
GET    /api/v1/jobs/:jid/diffbind-report     # DiffBind results
GET    /api/v1/jobs/:jid/heatmap-report      # custom heatmap
GET    /api/v1/jobs/:jid/pearson-report      # Pearson correlation
GET    /api/v1/jobs/:jid/normalization-report # Roman normalization
# (each with /download sub-endpoints for CSV/TSV export)

# Files & Downloads
GET    /api/v1/experiments/:id/files         # file tree (disk scan)
GET    /api/v1/experiments/:id/files/download # single file download
POST   /api/v1/experiments/:id/files/batch-download
POST   /api/v1/jobs/:jid/files/batch-download
POST   /api/v1/files/download-token          # HMAC-signed URL (5-min)
GET    /api/v1/files/signed-download
POST   /api/v1/files/igv-tokens              # batch IGV tokens (60-min)
GET    /api/v1/files/igv-serve               # Range header support (RFC 7233)
POST   /api/v1/experiments/:id/upload-bed    # BED file upload (<50MB)

# Server Import (FTP/SFTP)
POST   /api/v1/experiments/:id/server-import/browse
POST   /api/v1/experiments/:id/server-import/start
GET    /api/v1/experiments/:id/server-import/:iid/progress
GET    /api/v1/users/me/saved-servers
POST   /api/v1/users/me/saved-servers
PATCH  /api/v1/users/me/saved-servers/:id
DELETE /api/v1/users/me/saved-servers/:id

# Real-time & Notifications
GET    /api/v1/notifications/stream          # SSE endpoint (2s poll, 15s keepalive)
GET    /api/v1/notifications
PATCH  /api/v1/notifications/read-all
PATCH  /api/v1/notifications/:id/read

# Admin (superuser only)
GET    /api/v1/admin/stats
GET    /api/v1/admin/users
PATCH  /api/v1/admin/users/:id
GET    /api/v1/admin/projects
DELETE /api/v1/admin/projects/:id
GET    /api/v1/admin/jobs
POST   /api/v1/admin/jobs/:id/terminate
POST   /api/v1/admin/cleanup
GET    /api/v1/admin/storage-info

# Health
GET    /api/v1/health                        # {"status": "ok"}

Interactive docs at http://localhost:8000/docs.

Development

# Local dev (all services via Docker Compose)
docker compose up -d

# Local dev (without Docker — runs backend + worker + frontend)
./scripts/run-local.sh

# Lint backend
cd backend && ruff check . && ruff format .

# Type-check frontend
cd frontend && npx tsc --noEmit

# Run backend tests (MUST use Docker — tests need Postgres)
docker compose exec api pytest tests/test_specific.py           # single file
docker compose exec api pytest tests/test_specific.py -k "name" # single test
docker compose exec api pytest tests/                           # full suite (500+ tests)

# Create a new migration after model changes
docker compose exec api alembic revision --autogenerate -m "description"
docker compose exec api alembic upgrade head

# Rebuild API container after dependency changes
docker compose up -d --build api

Implementation Phases

Phase	Scope	Status
1. Foundation	Scaffold, auth, project/experiment CRUD, UI shell	Complete
2. Data Management	FASTQ upload (tus resumable), FastQC, reactions, trimming, file browser	Complete
3. Core Pipeline	Worker, SSE, alignment, QC reports, spike-in QC	Complete
4. Peak Calling	MACS2/SICER2/SEACR, HOMER, FRiP, fragment filter	Complete
5. Visualization	IGV.js genome browser, byte-range serving	Complete
6. Lab Extensions	DiffBind, custom heatmaps, Pearson correlation, Roman normalization	Complete
7. Polish & QA	Storage lifecycle, job termination/retry, auto-pipeline, audit log	Complete
8. UI Overhaul	shadcn/ui, dark mode, typography system, landing page	Complete
9. FTP/SFTP Import	Server import wizard, saved credentials, SSRF prevention	Complete
10. Admin Panel & Docs	Superuser admin panel, in-app documentation, deployment guide	Complete
11. EC2 Deployment	NGINX, systemd, Cloudflare DNS, reference project seeded	Complete
12. Training Wheels	First-project training mode, cleared defaults, educational hints	Complete

Documentation

Detailed specs in docs/:

Document	Contents
`SPEC.md`	Living technical specification -- architecture, schema, API, pipeline details
`DEPLOYMENT_GUIDE.md`	EC2 deployment instructions (NGINX, systemd, Cloudflare, SSL)
`cleave-user-guide.md`	End-user documentation, QC interpretation, step-by-step tutorials
`cf-lab-pipeline-spec.md`	Lab pipeline stages, scripts, parameters, feature gaps
`cleave-spec-decisions.md`	Resolved questions, script audit, parameter reference, bug fixes
`cutana-architecture-plan.md`	Original system architecture and data model
`cutana-cloud-ui.md`	Page-by-page CUTANA Cloud UI reference
`cutana-cloud-docs.md`	CUTANA Cloud platform behavior, QC interpretation, terminology

In-app documentation is also available at /docs with 17 pages covering all platform features.

UI & Design

Dark mode -- full light/dark theme support via CSS variables and next-themes, toggle in navbar
shadcn/ui -- 10 Radix UI primitives (Dialog, DropdownMenu, Select, Tabs, Tooltip, ScrollArea, Collapsible, Badge, Separator, Sonner) with CVA-based styling
Typography -- Source Serif 4 (headings), Source Sans 3 (body), Source Code Pro (monospace)
Icon system -- lucide-react replaces all inline SVGs
Toast notifications -- sonner for transient feedback
Landing page -- animated pipeline visualization at /, feature comparison table, live stats
EmptyState pattern -- consistent empty-state illustrations across all list views

Key Design Decisions

Async everywhere -- async SQLAlchemy engine, all handlers and services are async def
Single job queue -- analysis_jobs table polled by a standalone worker process (FOR UPDATE SKIP LOCKED). Configurable concurrency
Parallel pipeline processing -- ThreadPoolExecutor per-reaction for trimming, alignment, and peak calling
JSONB params -- analysis_jobs.params stores all job-specific config. No per-job-type tables
MACS2 q-value defaults to 0.01 (lab standard), not 0.05 (CUTANA Cloud). Both available in Advanced Settings
SEACR uses numeric threshold 0.01 by default (top 1% AUC), not IgG control. Both modes available
Fragment size filter (<120bp) is default ON before peak calling. Sub-nucleosomal fragments are the biologically relevant CUT&RUN signal
Auto-pipeline mode -- one-click FastQC, Trim, Align, Peak Call chain with parent_job_id dependency tracking
30-min access tokens -- JWT access (30-min) + httpOnly refresh cookie (7-day). Rate limited: 5/min login, 3/min register
HMAC-signed download tokens -- file downloads use time-limited HMAC tokens for auth instead of JWT, enabling direct browser downloads and IGV.js byte-range requests
Mock pipeline mode -- PIPELINE_MODE=mock stubs all pipeline calls for frontend/API dev without bioinformatics tools
Large files served via NGINX X-Accel-Redirect in production. FastAPI only checks auth, never streams large files
SSRF prevention -- FTP/SFTP server import blocks private IP ranges, localhost, AWS metadata endpoints, and IPv6-mapped IPv4
Fernet encryption -- saved server credentials encrypted at rest with per-instance key
DiffBind columns are dynamic -- Conc_X/Conc_Y parsed from TSV header, never hardcoded
Job termination & retry -- DB-polled termination between subprocess steps; retry creates a new job from failed/terminated
Admin panel -- superuser-only panel for user management, project/job oversight, storage cleanup, system stats

License

Private. Ferguson Lab, UCSD.

Navigation Menu

Repository files navigation

Cleave

What It Does

Tech Stack

Quick Start

Prerequisites

Run with Docker Compose

Run Locally (without Docker)

Project Structure

Database Schema

API

Development

Implementation Phases

Documentation

UI & Design

Key Design Decisions

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 208 Commits
backend		backend
cutana		cutana
docker		docker
docs		docs
frontend		frontend
logs		logs
plans		plans
references		references
scripts		scripts
test_data		test_data
.env.example		.env.example
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
README.md		README.md
TODO.md		TODO.md
docker-compose.yml		docker-compose.yml

Folders and files

Latest commit

History

Repository files navigation

Cleave

What It Does

Tech Stack

Quick Start

Prerequisites

Run with Docker Compose

Run Locally (without Docker)

Project Structure

Database Schema

API

Development

Implementation Phases

Documentation

UI & Design

Key Design Decisions

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages