Ignis

Ignis is a from-scratch LLM inference engine written in Rust, with a real tensor-graph compiler at its core.

It loads a quantized open-weights model straight from a GGUF file and runs it end to end. It parses the container, dequantizes the weights on the fly, runs the full transformer forward pass, and streams tokens out of its own byte-level BPE tokenizer. There is no PyTorch, no ggml, no ONNX, no BLAS. Every layer of the stack (the GGUF reader, the quantized kernels, the attention math, the tokenizer, and the graph compiler) is written here by hand.

The part I care about most is the layer most inference demos skip. Instead of running the model as a fixed sequence of function calls, Ignis lowers the forward pass into a small intermediate representation, runs two optimization passes over it (operator fusion and liveness-based memory planning), and then executes the optimized plan. That is, in miniature, what a production engine like ggml or TensorRT does, and what an optimizing compiler does to a program.

$ ignis run -m models/qwen2.5-0.5b-instruct-q8_0.gguf \
            -p "In two sentences, what makes a compiler different from an interpreter?"

loaded qwen2 (24 layers, vocab 151936) in 0.07s
compiler: 435 ops -> 362 ops (49 rmsnorm + 24 swiglu fused)
compiler: 363 activation values -> 5 reused buffers (2760.5 KB -> 669.5 KB activations, 76% saved)
A compiler is a program that translates source code into machine code, while an
interpreter is a program that executes code directly.
[32 prompt tok in 0.55s (58.1 tok/s) | 24 gen tok in 0.46s (52.1 tok/s)]

What is actually built here

• GGUF loader (gguf.rs) for the v2/v3 container format: typed key/value metadata, tensor descriptors, and a memory-mapped weight blob so multi-gigabyte models are never copied into the heap.
• Quantized math (quant.rs) for four formats: F32, F16, Q8_0, and Q4_0. The matmul never materializes a dequantized weight matrix; it walks each weight row block-by-block, reconstructing 32 values at a time directly into the accumulation. The hot Q8_0 path and the f32 reducer are hand-written NEON kernels on Apple Silicon, with scalar fallbacks for other targets.
• Byte-level BPE tokenizer (tokenizer.rs): the GPT-2 / Qwen byte-to-unicode map, the merge-by-rank algorithm, ChatML special-token handling, and a streaming decoder that holds back partial UTF-8 so multi-byte glyphs never print half finished. The vocab and merges are read out of the GGUF file itself.
• The transformer (model.rs): Qwen2 with grouped-query attention, rotary position embeddings, RMSNorm, a SwiGLU feed-forward, and a KV cache.
• The tensor-graph compiler (graph.rs, compiler.rs, runtime.rs): an SSA IR, a fusion pass, a memory planner, and a runtime that executes the result. This is the heart of the project and is described below.
• Sampling (sampler.rs): greedy by default (deterministic), plus temperature with top-k and nucleus (top-p) filtering, on a self-contained RNG.

The compiler

A forward pass is first lowered to a flat list of operations in single-static- assignment form: every op writes a new value, and values are immutable once written. That representation makes the two passes simple and verifiably correct.

Fusion is local pattern rewriting. The lowered graph emits RMSNorm as a normalize step followed by a scale-by-gain step, and SwiGLU as a SiLU followed by an elementwise multiply. The fusion pass recognizes those pairs and collapses each into a single kernel, which removes a full pass over the activation and one intermediate buffer per match. On Qwen2.5-0.5B that is 73 fused ops (49 RMSNorm, 24 SwiGLU).

Memory planning is register allocation applied to activation tensors. Each value has a lifetime from the op that defines it to its last use; values whose lifetimes do not overlap can share a physical buffer. A single linear scan over the schedule reuses buffers accordingly. The naive footprint of one decode step is 363 separate activation buffers (about 2.7 MB); after planning it is 5 reused buffers (about 0.67 MB), a 76% reduction, with no change to the output.

The compiled graph engine and the original imperative implementation share the exact same kernels, so they are numerically identical. tests/parity.rs asserts that: it runs both engines on the same prompt and requires byte-identical token streams and matching logits. That test is what lets the optimization passes be trusted.

Quick start

You need a Rust toolchain (rustup, stable). Then:

git clone https://github.com/arya51-ai/ignis
cd ignis

# fetch a model (defaults to Qwen2.5-0.5B-Instruct, Q8_0, about 640 MB)
./scripts/download-model.sh

cargo build --release

./target/release/ignis run -m models/qwen2.5-0.5b-instruct-q8_0.gguf \
    -p "Write a haiku about compilers."

Commands

ignis run   -m <model.gguf> [-p <prompt>] [options]   generate text
ignis info  <model.gguf>                              print architecture + tensor types
ignis bench -m <model.gguf> [-n <tokens>]             compare both engines, tok/s

Useful run options: -n max new tokens, -t temperature (0 = greedy), --top-k / --top-p, -s system prompt, --raw for raw completion instead of chat, and --engine imperative to run the reference engine instead of the compiled one.

Benchmarks

Apple M3 (8 cores), Qwen2.5-0.5B-Instruct, greedy decoding. The compiled graph engine and the imperative reference land within noise of each other, as expected (same kernels); the win from the compiler is in memory and clarity, not raw speed.

format	size	decode tok/s
Q8_0	640MB	~52
Q4_0	410MB	~26
F16	1.2GB	~8

These are honest numbers from a clear, readable implementation. They are not yet competitive with llama.cpp's years of hand-tuned assembly; closing that gap (integer-domain quantized dot products, a persistent thread pool, batched prefill) is the natural next direction. Correctness and a real compiler came first.

Supported models

Anything in GGUF with the qwen2 architecture and weights in F32, F16, Q8_0, or Q4_0. Qwen2.5-0.5B-Instruct is the default target and the model the tests run against. Other Qwen2 / Qwen2.5 sizes load if their tensors are in the supported dtypes.

Project layout

src/
  gguf.rs        GGUF container parser (mmap-backed)
  tensor.rs      dtypes and ggml block geometry
  quant.rs       dequantization + fused dot products (NEON + scalar)
  f16.rs         half <-> single precision conversion
  kernels.rs     matmul, rmsnorm, rope, softmax, swiglu
  tokenizer.rs   byte-level BPE + ChatML + streaming decode
  graph.rs       the SSA tensor-graph IR
  compiler.rs    fusion + memory planning
  runtime.rs     executes a compiled plan against a buffer arena
  model.rs       Qwen2 config, weights, imperative forward, graph lowering
  sampler.rs     greedy / temperature / top-k / top-p
  main.rs        CLI: run / info / bench
tests/
  parity.rs      graph engine == imperative engine, end to end

Tests

cargo test --release

The unit tests cover the f16 conversion, the quantized kernels (including NEON vs scalar parity), softmax/RMSNorm/RoPE, the tokenizer byte map, and the compiler passes. The end-to-end parity test runs automatically when a model is present in models/ and skips cleanly otherwise.

License

MIT. See LICENSE.