sd-flux2.cpp

A tiny C++17 / ggml image generator for FLUX.2 [klein] by Black Forest Labs, built on top of stable-diffusion.cpp. One small binary + three model files. No Python, no PyTorch at build or run time, runs on an 8GB GPU, or CPU-only on a GPU-less edge box.

"a cinematic photo of a red fox sitting in an autumn forest, golden hour, sharp focus" — FLUX.2 [klein] 4B, Q4_K, 768×768, 4 steps, generated in ~14.5s on an RTX A2000 8GB.

Given a text prompt it produces a PNG through the full FLUX.2 pipeline (text encode → sampling → VAE decode), driven by quantized GGUF weights that fit in tight VRAM.

Not a from-scratch engine

sd-flux2.cpp is not its own inference engine. It is a thin, klein-specialized front-end over stable-diffusion.cpp (which vendors ggml). That library does all the real work — model loading, the FLUX.2 pipeline, sampling, VAE decode. Models are FLUX.2 by Black Forest Labs (klein 4B is Apache-2.0). See NOTICE.md for full credits and licenses.

---

Features

• FLUX.2 [klein] 4B text-to-image from a single command, no Python/PyTorch at inference.
• Single small binary. ~250MB with CUDA kernels baked in, ~54MB CPU-only.
• CPU-first, GPU-ready. Runs on an 8GB CUDA card with weight offload, or CPU-only on an edge box; the vendored ggml engine also has Vulkan / Metal / OpenCL backends.
• Quantized GGUF weights. Q2_K (~1.5GB) / Q4_K (~2.3GB) / Q5_K (~2.8GB) drop the 4B transformer from ~13GB FP16 into tight VRAM. Published on 🤗 Hugging Face.
• Step-distilled. klein generates in 4 steps at CFG 1.0 — interactive on GPU.
• curl-only model fetch. No Python downloader; scripts/fetch_models.sh grabs the GGUFs.

---

Why

FLUX.2 [klein] 4B is step-distilled and Apache-2.0, but ~13GB at FP16. GGUF quantization drops the transformer to 1.5–2.8GB so it runs on an 8GB GPU — or, CPU-only, on a GPU-less edge box. This repo packages that into a single small executable you can drop onto a device, without dragging in a Python/PyTorch stack.

---

Layout

CMakeLists.txt              # in-tree build of the submodule engine + sd-flux2
src/main.cpp                # the entire program (a thin wrapper over the C API)
vendor/stable-diffusion.cpp # git submodule: the engine (+ ggml). All heavy lifting is here.
scripts/fetch_models.sh     # curl-only model download (no Python)
scripts/benchmark.sh        # latency/size benchmark harness -> benchmarks/RESULTS.md
benchmarks/RESULTS.md       # measured numbers (mirrored below)
img/                        # sample renders shown in this README
NOTICE.md                   # acknowledgements + licenses

---

Build

Requirements: cmake (>=3.14), a C/C++17 compiler. For the GPU build, the CUDA toolkit (nvcc).

The engine is a git submodule (pinned to a known-good stable-diffusion.cpp commit, which itself pins ggml), so clone recursively:

git clone --recursive https://github.com/adithyab94/sd-flux2.cpp
# already cloned without --recursive?
git submodule update --init --recursive

GPU (CUDA)

cmake -B build -DSD_FLUX2_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86   # 86=Ampere; 89=Ada
cmake --build build -j --target sd-flux2

CPU-only (edge / no NVIDIA GPU)

cmake -B build-cpu -DSD_FLUX2_CUDA=OFF
cmake --build build-cpu -j --target sd-flux2

ggml also has Vulkan / Metal / OpenCL backends; pass e.g. -DSD_VULKAN=ON to the vendored engine instead of CUDA for other edge GPUs. For ARM, cross-compile with a CMake toolchain file.

---

Get the models

Pre-quantized GGUFs are published at adithya-balaji/FLUX.2-klein-4B-GGUF.

./scripts/fetch_models.sh              # Q4_K_M (default)
QUANT=Q2_K ./scripts/fetch_models.sh   # smallest (~1.5GB)

---

Run

./build/sd-flux2 \
  --model models/flux-2-klein-4b-Q4_K_M.gguf \
  --vae   models/flux2-vae.safetensors \
  --llm   models/Qwen3-4B-Q4_K_M.gguf \
  -p "a red fox in an autumn forest, golden hour" \
  -o fox.png -W 768 -H 768 --steps 4 --cfg 1.0 -s 42

--help lists all options. --no-offload keeps weights on the GPU (faster, more VRAM); the default offloads params to RAM for tight-VRAM devices.

---

Benchmarks / Results

All numbers measured on one reference machine, generating FLUX.2 [klein] 4B with 4 steps, CFG 1.0, euler, seed 42, prompt "a cinematic photo of a red fox sitting in an autumn forest, golden hour, sharp focus". gen (s) is end-to-end (text-encode + sampling + VAE decode), not just diffusion. Reproduce with ./scripts/benchmark.sh; full methodology in benchmarks/RESULTS.md.

• GPU: NVIDIA RTX A2000 8GB Laptop GPU (CUDA 12.4, arch sm_86)
• CPU: 12th Gen Intel Core i7-12800HX (24 threads)
• Engine: vendored stable-diffusion.cpp (ggml), --offload-to-cpu on, VAE tiling on
• Binary size: 250MB (CUDA, kernels baked in) / 54MB (CPU-only)

GPU (CUDA, RTX A2000 8GB)

quant	model size	resolution	load (s)	gen (s)	s/step
Q2_K	1.5GB	768×768	0.40	16.00	4.00
Q4_K	2.3GB	512×512	0.39	9.78	2.44
Q4_K	2.3GB	768×768	0.38	14.48	3.62
Q4_K	2.3GB	1024×1024	0.36	22.90	5.72
Q5_K	2.8GB	768×768	0.35	15.83	3.96

CPU-only (i7-12800HX, 24 threads)

quant	model size	resolution	load (s)	gen (s)	s/step
Q2_K	1.5GB	512×512	0.26	238.82	59.71
Q4_K	2.3GB	512×512	0.31	235.09	58.77
Q4_K	2.3GB	768×768	0.24	432.09	108.02

Takeaways

• GPU vs CPU: at 768×768 Q4_K, GPU is ~30× faster (14.5s vs 432s). CPU is usable for offline/edge batch jobs, not interactive use.
• Quant type barely changes speed. On CPU, Q2_K (238s) ≈ Q4_K (235s) at 512²; on GPU, Q2_K (16.0s) is even slightly slower than Q4_K (14.5s) at 768². K-quant weights are dequantized to compute anyway, and neither device is memory-bound here — so choose the quant for size/VRAM/quality, not for speed.
• Resolution dominates cost. GPU Q4_K scales 9.8s → 14.5s → 22.9s across 512²/768²/1024². CPU roughly doubles from 512² (235s) to 768² (432s).
• Load time is negligible (~0.3–0.4s) thanks to memory-mapped GGUF.

---

Size vs quality

quant	size	when to use
Q2_K	~1.5GB	tightest RAM / smallest storage; softest detail
Q4_K	~2.3GB	best balance — recommended default
Q5_K	~2.8GB	highest fidelity, if you have the headroom

Quant type has little effect on speed, so pick it for size/VRAM/quality. Same prompt, same seed, 768×768, GPU — Q2_K (smallest) next to Q5_K (highest fidelity):

Q2_K (~1.5GB)	Q5_K (~2.8GB)

A larger 1024×1024 Q4_K render:

---

Credits

sd-flux2.cpp stands on other people's work — full details in NOTICE.md:

• stable-diffusion.cpp (leejet and contributors, MIT) — the actual C++ inference engine, included as a git submodule at vendor/stable-diffusion.cpp/. All the hard work lives there.
• ggml (MIT) — the tensor library and CPU/CUDA/ Vulkan/Metal backends that stable-diffusion.cpp builds on.
• FLUX.2 models by Black Forest Labs (klein 4B is Apache-2.0). This project does not redistribute weights; you download them yourself.

---

License

sd-flux2.cpp (the src/main.cpp glue, CMake, and scripts) is released under the MIT License. Vendored code retains its own upstream licenses (stable-diffusion.cpp — MIT, ggml — MIT), and FLUX.2 model weights are governed by Black Forest Labs' terms (klein 4B is Apache-2.0). See NOTICE.md and LICENSE.