AMALIA-9B (European Portuguese) support: SPM Metaspace prepend, ChatML detection, bf16 compute

CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **AMALIA-9B** (European Portuguese, EuroLLM-9B base) support, golden-gated
+  against mlx-lm (`reference/fixtures_amalia/`). The forward pass is the stock
+  Llama decoder, run in **bfloat16** via the new `ModelConfig::bf16_compute`
+  (this family's residual stream overflows fp16 from layer 9 — an fp16 forward
+  is all-NaN logits; selected by config fingerprint, all other checkpoints stay
+  on fp16 bit-for-bit). The other new pieces are tokenizer/prompt plumbing: the
+  SentencePiece-BPE backend honors the Metaspace `prepend_scheme: "always"`
+  pre-tokenizer and the `Strip` decode step (both parsed from `tokenizer.json`;
+  Gemma unchanged), a `ChatFormat::Amalia` ChatML template with the default
+  Portuguese system prompt (detected from the checkpoint's `chat_template.jinja`
+  via the new `chat_format_for_model_dir`, since AMALIA's `model_type` is
+  `"llama"`), and config-load rejection of non-affine quantization modes.
+  KV-cache quantization and the prefix cache are rejected for bf16-compute
+  models (no golden coverage yet); the golden compare harness now fails on
+  nonfinite values instead of passing NaN-vs-NaN vacuously.
+
 ## [0.2.0] - 2026-06-12
 
 This release adds a from-scratch **Qwen3-VL vision-language** model (image → text),

CLAUDE.md

@@ -138,11 +138,32 @@ reference/.venv/bin/python reference/dump_ref.py
   (`reference/fixtures*/tokenizer_corpus.json`, regenerated by `dump_ref.py`).
   Two backends: byte-level BPE (`tokenizer/bpe.{h,cpp}`, `BpeTokenizer`; Llama-3.2
   / Qwen) and SentencePiece-BPE (`tokenizer/spm.{h,cpp}`, `SpmBpeTokenizer`;
-  metaspace + `byte_fallback`, e.g. Gemma — tokenizer-only, no Gemma model class).
-  Both are pure/`const`/thread-safe (no mutex). `Tokenizer::from_file` throws on
-  still-unimplemented families (e.g. Unigram/WordPiece).
-- **Masks are additive fp16, never boolean** (avoids MLX bug #2894). See
-  `DecoderModel::batch_mask`.
+  metaspace + `byte_fallback` — Gemma (tokenizer-only, no Gemma model class) and
+  AMALIA-9B/EuroLLM). The SPM backend honors the Metaspace `prepend_scheme` per
+  checkpoint ("always" prepends `▁` to the *input-leading* segment only — never
+  after special tokens, never doubled onto a leading space; empirically pinned
+  by the AMALIA corpus fixture — Gemma never prepends; "first" throws) and a
+  decoder `Strip` step (drop N leading spaces on decode). Both are
+  pure/`const`/thread-safe (no mutex). `Tokenizer::from_file` throws on
+  still-unimplemented families (e.g. Unigram/WordPiece). AMALIA's chat format
+  can't come from `model_type` (it's `"llama"`): `chat_format_for_model_dir`
+  detects it from the checkpoint's `chat_template.jinja`, and because AMALIA's
+  BOS id 3 IS `<|im_start|>`, `apply_chat_template` encodes the fully-rendered
+  ChatML **without** the BOS prepend (HF `add_special_tokens=False`) — letting
+  the encoder prepend it would double the opening tag.
+- **Masks are additive fp16 (or the model's compute dtype), never boolean**
+  (avoids MLX bug #2894). See `DecoderModel::batch_mask`.
+- **The engine computes in fp16 — except checkpoints that overflow it.**
+  AMALIA-9B/EuroLLM-9B's residual stream exceeds fp16 range from layer 9, so an
+  fp16 forward is all-NaN logits and the fp16 mlx-lm reference is equally
+  degenerate (NaN goldens gate nothing — `compare_close` now fails on
+  nonfinite values for exactly this reason). `ModelConfig::bf16_compute`
+  (selected by the family's config fingerprint in `from_json`, since every MLX
+  conversion declares `dtype: bfloat16` and can't drive this) switches the
+  weight-load cast, the batched mask, and the prefill logits placeholders to
+  bf16; `skinny_mm` self-disables (fp16-only kernels) and kv-quant / prefix
+  cache are rejected at engine creation for bf16 models. All other checkpoints
+  keep fp16 bit-for-bit.
 - **RoPE scaling is precomputed and validated, never silent** —
   `compute_rope_setup` mirrors `mlx_lm` exactly (`Llama3RoPE`, `YarnRoPE`,
   `nn.RoPE(scale=1/factor)` for linear) and feeds `fast::rope` via `freqs`

README.md

@@ -81,10 +81,11 @@ the C-ABI / Node quickstart.
 ## Supported models
 
 The forward pass (the `DecoderModel` base in `model/`) is family-shared and runs
-Llama-3.2, Qwen3 (dense / MoE), and Qwen3.5 hybrid models from safetensors (fp16 /
-4-bit) or a single-file GGUF, plus **Qwen3-VL** vision-language (image → text). The
-chat template and special-token handling are selected from `config.json` — loading
-is org-agnostic.
+Llama-3.2, AMALIA-9B (European Portuguese), Qwen3 (dense / MoE), and Qwen3.5 hybrid
+models from safetensors (fp16 / 4-bit) or a single-file GGUF, plus **Qwen3-VL**
+vision-language (image → text). The chat template and special-token handling are
+selected from `config.json` and the checkpoint's chat template — loading is
+org-agnostic.
 
 | Family | Example repo | Precision |
 | --- | --- | --- |
@@ -95,6 +96,7 @@ is org-agnostic.
 | Qwen3 (MoE) | `mlx-community/Qwen3-30B-A3B-4bit` | 4-bit / fp16 |
 | Qwen3.5 (hybrid) | `mlx-community/Qwen3.5-0.8B-4bit` | 4-bit (text tower) |
 | Qwen3-VL (vision) | `mlx-community/Qwen3-VL-4B-Instruct-4bit` | 4-bit (image → text) |
+| AMALIA-9B (pt-PT) | `layerx-labs/AMALIA-9B-0626-DPO-MLX-4bit` | 4-bit |
 
 See [`doc/supported-models.md`](./doc/supported-models.md) for the full compatibility
 matrix, the per-family deltas, and how to add a new family.

apps/mlxforge_cli.cpp

@@ -110,7 +110,7 @@ LoadedModel load_for_inference(const std::string& spec, const std::string& rope_
     mlxforge::validate_rope_scaling(lm.cfg);
     lm.model = mlxforge::create_model(lm.cfg, mlxforge::load_weights(resolved, lm.cfg));
     lm.tok = mlxforge::Tokenizer::from_file(resolved + "/tokenizer.json", lm.cfg.bos_token_id,
-                                            mlxforge::chat_format_from_model_type(lm.cfg.model_type));
+                                            mlxforge::chat_format_for_model_dir(resolved, lm.cfg.model_type));
   }
   return lm;
 }

doc/supported-models.md

@@ -25,6 +25,7 @@ override — all selected automatically by `create_model` from `config.json`
 | Qwen3 (MoE, GGUF) | `Qwen/Qwen3-30B-A3B-GGUF` | Q4_0/Q4_1/Q8_0, Q4_K/Q5_K/Q6_K | ChatML (`<\|im_start\|>…`) | yes |
 | Qwen3.5 (hybrid) | `mlx-community/Qwen3.5-0.8B-4bit` | 4-bit (mixed bits ok) | ChatML (`<\|im_start\|>…`) | yes (text only) |
 | Qwen3-VL (dense) | `mlx-community/Qwen3-VL-4B-Instruct-4bit` | 4-bit (ViT in fp16) | ChatML + vision tokens | yes (image → text) |
+| AMALIA-9B (pt-PT) | `layerx-labs/AMALIA-9B-0626-DPO-MLX-4bit` | 4-bit, bf16 compute | ChatML + PT default system | yes |
 
 **Qwen3 dense** models (0.6B/1.7B/4B/8B/14B/32B) run end-to-end. They add three
 deltas over Llama-3.2, all handled automatically: per-head **QK-Norm** (an
@@ -111,6 +112,39 @@ is **not** bit-identical to HF's PIL bicubic, so results on non-aligned images m
 differ slightly from the reference. The **MoE-VL** variant (30B-A3B) is not yet a
 distinct model class.
 
+**AMALIA-9B** (European Portuguese, built on EuroLLM-9B) runs end-to-end. Its
+forward pass is the **stock Llama decoder** — plain `LlamaForCausalLM` (42
+layers, GQA with 8 KV heads, `rope_theta` 1e6, no rope scaling, untied
+embeddings), so `create_model` falls through to `LlamaModel` with no per-family
+hook — but it runs in **bfloat16, not the engine's usual fp16**: the family's
+residual stream overflows fp16's range from layer 9, so an fp16 forward is
+all-NaN logits (silent garbage — the fp16 mlx-lm reference is equally
+degenerate). `ModelConfig::bf16_compute` selects bf16 from the family's config
+fingerprint; KV-cache quantization and the prefix cache are rejected for
+bf16-compute models (no golden coverage yet). The remaining deltas are in the
+tokenizer and prompt format, all golden-gated against mlx-lm
+(`reference/fixtures_amalia/`):
+
+- **SentencePiece-BPE with Metaspace prepend**: the Metaspace pre-tokenizer uses
+  `prepend_scheme: "always"` (a `▁` is prepended to the input-leading text
+  segment before BPE — never after special tokens, and never doubled onto a
+  leading space; the exact semantics are pinned by the golden corpus), and the
+  decoder chain carries a `Strip` step that removes the resulting leading space
+  on decode. Both are parsed from `tokenizer.json`, not hard-coded (Gemma never
+  prepends).
+- **ChatML with a default Portuguese system prompt**: when the conversation has
+  no system turn, the template injects the "O teu nome é Amália…" preamble.
+  `model_type` is `"llama"`, so this format is detected from the checkpoint's
+  `chat_template.jinja` (via `chat_format_for_model_dir`), not from `model_type`.
+- **`<|im_start|>` (id 3) doubles as BOS**: the tokenizer's post-processor
+  prepends it on plain encode; `apply_chat_template` therefore encodes the
+  (fully rendered) template *without* the BOS prepend, mirroring HF's
+  `add_special_tokens=False` — prepending would double the opening tag. EOS is
+  `[4, 2]` (`<|im_end|>` / `</s>`).
+
+The SFT variant (`AMALIA-9B-0626-SFT-MLX-4bit`) shares the identical
+architecture, tokenizer, and template, so it runs unchanged.
+
 Other LLaMA-family models will be re-onboarded as needed; because the forward
 pass is shared, that work is mostly tokenizer/chat-format plus any small
 attention delta (see [Adding a new model family](#adding-a-new-model-family)).
@@ -253,15 +287,20 @@ constants:
 - **`model_type`** selects the chat format via `chat_format_from_model_type`: the
   Llama-3.2 header format, or Qwen ChatML (`qwen3` / `qwen3_moe` / `qwen2`,
   `qwen3_5`, and `qwen3_vl` — the last rendering vision blocks for attached images).
-  For a VLM, the nested `vision_config` and the M-RoPE / vision-token fields are read
-  too.
+  Formats that `model_type` cannot identify are recognized from the checkpoint's
+  on-disk template (`chat_template.jinja`, else the `chat_template` string in
+  `tokenizer_config.json`) by `chat_format_for_model_dir` — AMALIA is `model_type`
+  `"llama"` but ChatML. For a VLM, the nested `vision_config` and the M-RoPE /
+  vision-token fields are read too.
 - **`tokenizer.json`** drives encode/decode and supplies the special-token ids
   (`added_tokens[*].special`) that are skipped on decode — there are no hard-coded
   token ids. Two backends sit behind one `EncoderBackend` interface, picked by
   inspecting the blob: **byte-level BPE** (`ByteLevel` decoder; Llama-3.2 / Qwen)
-  and **SentencePiece-BPE** (`byte_fallback` + metaspace; Gemma). BOS is resolved
-  from the fast tokenizer's post-processor first (Gemma prepends `<bos>` despite
-  `add_bos_token: false`), then the `tokenizer_config.json` flag.
+  and **SentencePiece-BPE** (`byte_fallback` + metaspace; Gemma, AMALIA — the
+  Metaspace `prepend_scheme` and the decoder's `Strip` step are honored per
+  checkpoint). BOS is resolved from the fast tokenizer's post-processor first
+  (Gemma prepends `<bos>` despite `add_bos_token: false`; AMALIA prepends
+  `<|im_start|>`), then the `tokenizer_config.json` flag.
 - A separate `lm_head.weight` is used if present; otherwise the embedding is tied.
 
 ## What is and isn't implemented
@@ -283,9 +322,9 @@ GGUF `Q4_0`/`Q4_1`/`Q8_0` (group_size 32) all run; GGUF `Q4_K`/`Q5_K`/`Q6_K`
 - **Sliding-window attention.** Only plain causal attention is supported, so
   models that rely on a sliding window are not.
 - **Other tokenizer families / chat templates.** Byte-level BPE (Llama-3.2 /
-  Qwen) and SentencePiece-BPE (Gemma) are implemented; other families (e.g.
-  Unigram/WordPiece) make `Tokenizer::from_file` throw. Chat formats are limited
-  to the Llama-3.2 header format and Qwen ChatML.
+  Qwen) and SentencePiece-BPE (Gemma, AMALIA) are implemented; other families
+  (e.g. Unigram/WordPiece) make `Tokenizer::from_file` throw. Chat formats are
+  limited to the Llama-3.2 header format and ChatML (Qwen, AMALIA).
 - **Batched multimodal serving.** Qwen3-VL runs image → text, but multimodal
   requests are served **single-stream**; sharing a continuous-decode batch with
   text (per-row 3D M-RoPE in `BatchKVCache`) is not yet implemented. The **MoE-VL**

doc/tokenizer.md

@@ -125,12 +125,15 @@ than BPE mechanics:
   configured `bos_id` (default `128000` = `<|begin_of_text|>`; `-1` adds none),
   matching `mlx-lm`'s `tok.encode`. Decode pre-filters special ids symmetrically.
 - **Chat template.** `apply_chat_template` / `render_chat_template` render the
-  Llama-3.2 header format (`<|start_header_id|>role<|end_header_id|>\n\n…<|eot_id|>`)
-  and inject the default "Cutting Knowledge Date / Today Date" system preamble.
-  The format is selected from `config.json`'s `model_type` via
-  `chat_format_from_model_type` → `ChatFormat`, keeping the (shared) forward pass
-  decoupled from the (per-family) prompt formatting. Only `Llama3` exists today;
-  the `enum` is the seam for new families.
+  model's format: the Llama-3.2 header format
+  (`<|start_header_id|>role<|end_header_id|>\n\n…<|eot_id|>` with the default
+  "Cutting Knowledge Date / Today Date" system preamble), Qwen ChatML
+  (`Qwen3`/`Qwen35`, with the thinking toggle), or AMALIA ChatML (`Amalia`, with
+  the default Portuguese system prompt). The format is selected from
+  `config.json`'s `model_type` via `chat_format_from_model_type` → `ChatFormat`
+  — or, for formats `model_type` cannot identify (AMALIA is `"llama"`), from the
+  checkpoint's on-disk chat template via `chat_format_for_model_dir` — keeping
+  the (shared) forward pass decoupled from the (per-family) prompt formatting.
 - **Streaming detokenization.** `StreamingDetokenizer` is fed one new id at a
   time and returns only the text that has become **complete UTF-8** — it never
   emits a broken multi-byte character or a partial byte-BPE sequence mid-stream
@@ -140,10 +143,13 @@ than BPE mechanics:
 ## Supported families and the routing seam
 
 `BpeTokenizer::is_supported` returns `true` only for `model.type == "BPE"` **and**
-a `ByteLevel` decoder. `Tokenizer::from_file` calls it and **throws** otherwise —
-in particular a Metaspace / SentencePiece tokenizer is rejected rather than
-silently mistokenized, because the byte-level pipeline and the hand-rolled
-splitter are only correct for byte-level BPE.
+a `ByteLevel` decoder. A SentencePiece-style BPE (`byte_fallback` + a
+`ByteFallback` decoder — Gemma, AMALIA/EuroLLM) routes to `SpmBpeTokenizer`
+(`tokenizer/spm.{h,cpp}`) instead, which also honors the Metaspace
+pre-tokenizer's `prepend_scheme` ("always" prepends `▁` to every plain segment;
+"first" is rejected as unimplemented) and a `Strip` decode step.
+`Tokenizer::from_file` **throws** when no backend matches — an unsupported
+family (e.g. Unigram/WordPiece) is rejected rather than silently mistokenized.
 
 Re-onboarding a non-Llama family therefore means (a) routing its tokenizer to an
 appropriate backend behind this same `is_supported` check, and (b) adding its

reference/dump_ref.py

@@ -152,6 +152,39 @@
             "café ▁ leading metaspace",
         ],
     },
+    # AMALIA-9B (European Portuguese, EuroLLM-9B base) is a plain LlamaForCausalLM
+    # served by the stock LlamaModel; what it uniquely exercises is the tokenizer:
+    # SentencePiece-BPE with a Metaspace pre-tokenizer (prepend_scheme "always" —
+    # a U+2581 is prepended to the input-leading plain segment before BPE, unlike
+    # Gemma) and a Strip decoder (one leading space removed on decode), plus
+    # ChatML with <|im_start|> (id 3) doubling as BOS via the post-processor.
+    # The 4bit repo is the only published MLX conversion. Compute dtype is
+    # **bfloat16**, NOT the usual fp16 cast: this family's residual stream
+    # overflows fp16 from layer 9, so an fp16 forward is all-NaN logits (and an
+    # fp16 reference is equally degenerate — NaN goldens gate nothing). The C++
+    # engine mirrors this via ModelConfig::bf16_compute. The extra corpus pins
+    # the Metaspace prepend/Strip semantics where they are ambiguous: after
+    # special tokens, between adjacent specials, and around explicit spaces.
+    "amalia": {
+        "repo": "layerx-labs/AMALIA-9B-0626-DPO-MLX-4bit",
+        "fixtures": "fixtures_amalia",
+        "compute_dtype": mx.bfloat16,
+        # The AMALIA template injects a default Portuguese system message when the
+        # conversation has none — this chat prompt exercises exactly that path.
+        "chat_messages": [{"role": "user", "content": "Qual é a capital de Portugal?"}],
+        "thinking": False,
+        "extra_corpus": [
+            "<|im_start|>user\nOlá, como estás?<|im_end|>",
+            "<|im_start|>system\nÉs a Amália.<|im_end|>\n<|im_start|>user\nOlá<|im_end|>\n"
+            "<|im_start|>assistant\n",
+            "a<|im_end|><|im_end|>b",
+            "<|im_end|> depois de especial com espaço",
+            "<s>oi</s>",
+            "<extra_id_0>não especial",
+            "português: ação, coração, à noite, pêssego",
+            "Amália Rodrigues cantava fado.",
+        ],
+    },
 }
 
 # Fixed prompt set — committed so dumps are reproducible. Index 0 is the primary
@@ -567,7 +600,9 @@ def main():
 
     manifest = {
         "model_repo": MODEL_REPO,
-        "compute_dtype": "float16" if COMPUTE_DTYPE is mx.float16 else "quantized",
+        "compute_dtype": ("float16" if COMPUTE_DTYPE is mx.float16
+                          else "bfloat16" if COMPUTE_DTYPE is mx.bfloat16
+                          else "quantized"),
         "prompts": PROMPTS,
         "chat_messages": CHAT_MESSAGES,
         "greedy_max_new": GREEDY_MAX_NEW,
@@ -579,6 +614,10 @@ def main():
     def save(name, arr):
         """Eval an MLX array (or accept a numpy array) and write it as .npy."""
         if isinstance(arr, mx.array):
+            # numpy has no bfloat16: store bf16 dumps as float32 (exact upcast;
+            # the C++ compare harness casts both sides to float32 anyway).
+            if arr.dtype == mx.bfloat16:
+                arr = arr.astype(mx.float32)
             mx.eval(arr)
             np_arr = np.array(arr)
         else:
@@ -612,7 +651,8 @@ def save(name, arr):
     # byte-level BPE; tok.encode matches mlxforge::Tokenizer::encode for the family).
     # Dumped per-model so each tokenizer (Llama digit runs vs Qwen single digits) is
     # validated against its own oracle.
-    corpus = [{"text": s, "ids": [int(x) for x in tok.encode(s)]} for s in TOKENIZER_CORPUS]
+    corpus_texts = TOKENIZER_CORPUS + spec.get("extra_corpus", [])
+    corpus = [{"text": s, "ids": [int(x) for x in tok.encode(s)]} for s in corpus_texts]
     with open(os.path.join(FIXTURES_DIR, "tokenizer_corpus.json"), "w") as f:
         json.dump(corpus, f, ensure_ascii=False, indent=2)
     print(f"  wrote tokenizer_corpus.json ({len(corpus)} strings)")