fix: parse_messages_json drops content when key order is content-before-role#2
Open
ronaldw07 wants to merge 414 commits into
Open
fix: parse_messages_json drops content when key order is content-before-role#2ronaldw07 wants to merge 414 commits into
ronaldw07 wants to merge 414 commits into
Conversation
Signed-off-by: Neel Shejwalkar <nshejwalkar7@gmail.com>
…#485) Signed-off-by: jakmro <kubamroz124@gmail.com>
Signed-off-by: jakmro <kubamroz124@gmail.com> Co-authored-by: Henry Ndubuaku <ndubuakuhenry@gmail.com>
* fix: treat merged LoRA temp dir as local model path (cactus-compute#490) When `cactus convert` merges a LoRA adapter, it saves the merged model to a temp directory and tries to “download” it as if it were a Hugging Face repo id. The fix checks if the model_id is a local directory and passes the temp path directly into weight conversion by skipping the HF path Signed-off-by: Lennart <lvoelz@outlook.de> * fix: tokenizer detection failed for LoRA merged models - copy sentence piece model from LoRA directory (or base model in the HF cache as a fallback) and read the tokenizer config locally if possible - the model_id is a path in the merging case, which poisend the whole conversion chain -> replaced with a name variable for name-based detection (e.g. gemma special tokens) Signed-off-by: Lennart <lvoelz@outlook.de> * fix: copy tokenizer_config.json from LoRA adapter if available Signed-off-by: Lennart <lvoelz@outlook.de> --------- Signed-off-by: Lennart <lvoelz@outlook.de> Co-authored-by: Lennart <lvoelz@outlook.de>
* simplify and align sdks Signed-off-by: jakmro <kubamroz124@gmail.com> * Add fine-tuning guide and runtime compatibility documentation links to README Signed-off-by: jakmro <kubamroz124@gmail.com> --------- Signed-off-by: jakmro <kubamroz124@gmail.com>
…to_hf.yml (cactus-compute#495) Signed-off-by: jakmro <kubamroz124@gmail.com>
…actus-compute#496) Signed-off-by: jakmro <kubamroz124@gmail.com>
…nscribe (cactus-compute#499) Signed-off-by: jakmro <kubamroz124@gmail.com>
…nder rustc (cactus-compute#498) * Set MACOSX_DEPLOYMENT_TARGET=14.0 for SME2 linker symbols Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> * fix rust again Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> * fix Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> * fix Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> * fix Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> * fix Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> * fix Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> --------- Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com>
…cactus-compute#497) * Add FFI log control: cactus_log_set_level and cactus_log_set_callback Expose the existing C++ Logger::set_level() and Logger::set_callback() through the C FFI, allowing embedders to suppress or redirect log output instead of inheriting the default stderr behavior. Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> * Update cactus_ffi.h --------- Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com> Co-authored-by: Henry Ndubuaku <ndubuakuhenry@gmail.com>
* qwen 3.5 27B runs Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * optimized deltanet Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * removed unnecessary change Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * more cleanup Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * a little more cleanup Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * update readme Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> * removed kernel from graph ops to dedicated kernel file Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * cleanup Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * more cleanup Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * slower but more stable now with fp32 accumulation Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * added separate model enum for qwen3.5 Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * reverted a line of logic Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * simplified model detection Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * v1.10 Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> * Updates Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> --------- Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> Co-authored-by: HenryNdubuaku <ndubuakuhenry@gmail.com>
Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com>
* display RAM usage in chat stats Signed-off-by: Roman Shemet <shemet0roman@gmail.com> * add --system flag for system prompt injection Signed-off-by: Roman Shemet <shemet0roman@gmail.com> * add /image command and --image flag for VLM chat Signed-off-by: Roman Shemet <shemet0roman@gmail.com> * guard image features for non-VLM models Signed-off-by: Roman Shemet <shemet0roman@gmail.com> * clean up trailing whitespace in chat.cpp Signed-off-by: Roman Shemet <shemet0roman@gmail.com> * add cactus list command and centralize model registry - Add `cactus list` to show all supported models grouped by type - Add `--downloaded` flag to filter to local models only - Extract model registry from workflow YAML into shared models.json - Both CLI and publish workflow now read from single source of truth - Add `quantization=` field to config.txt for accurate precision display - Green ⬇ indicator, tags, disk size, and quantization for downloaded models Signed-off-by: Roman Shemet <shemet0roman@gmail.com> * add Qwen3.5-0.8B and Qwen3.5-2B to model registry Signed-off-by: Roman Shemet <shemet0roman@gmail.com> --------- Signed-off-by: Roman Shemet <shemet0roman@gmail.com>
…actus-compute#451) Signed-off-by: ammesatyajit <ammesatyajit@gmail.com> Co-authored-by: ammesatyajit <ammesatyajit@gmail.com>
* weight saving Signed-off-by: Noah Cylich <noahcylich@gmail.com> * WORKING! Signed-off-by: Noah Cylich <noahcylich@gmail.com> * making embeddings int8 for int4 Signed-off-by: Noah Cylich <noahcylich@gmail.com> * no rsqrt2 hard code float Signed-off-by: Noah Cylich <noahcylich@gmail.com> * sparse 16-group matmul Signed-off-by: Noah Cylich <noahcylich@gmail.com> * group size 32 kernel Signed-off-by: Noah Cylich <noahcylich@gmail.com> * Revert "group size 32 kernel" This reverts commit 3416df3. Signed-off-by: Noah Cylich <noahcylich@gmail.com> * Revert "sparse 16-group matmul" This reverts commit 941c2f8. Signed-off-by: Noah Cylich <noahcylich@gmail.com> * clearer/safer config params Signed-off-by: Noah Cylich <noahcylich@gmail.com> * small inaccuracy fixes and prefill optimization (40% skipped) Signed-off-by: Noah Cylich <noahcylich@gmail.com> * testing helpers Signed-off-by: Noah Cylich <noahcylich@gmail.com> * Revert "testing helpers" This reverts commit 2edf384. Signed-off-by: Noah Cylich <noahcylich@gmail.com> * finishing touches Signed-off-by: Noah Cylich <noahcylich@gmail.com> * adding gemma3n to readme Signed-off-by: Noah Cylich <noahcylich@gmail.com> * new approximation for exponent on (0,1) Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> * DEBUGGED AND WORKING GEMMA 3N Signed-off-by: Noah Cylich <noahcylich@gmail.com> * Removing all debugging files and debugging code from gemma 3n. Signed-off-by: Noah Cylich <noahcylich@gmail.com> * mode adaptive and safe gaussian topk filtering Signed-off-by: Noah Cylich <noahcylich@gmail.com> * fused altup Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> * Further optimisations Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> --------- Signed-off-by: Noah Cylich <noahcylich@gmail.com> Signed-off-by: Karen Mosoyan <karen.mossoyan@gmail.com> Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> Co-authored-by: Henry Ndubuaku <ndubuakuhenry@gmail.com>
…mple, fix tool response wrapping) (cactus-compute#502) Remove hardcoded “When you decide to call…” guidance and arg example from developer turn Fix FunctionGemma trigger string (no trailing period) and append tools declarations directly Wrap tool responses in a developer turn and allow stacking multiple tool responses Stop wrapping tool outputs in value:; pass through {...} payload as-is Close pending tool-response developer turn before next user/model turn to avoid malformed prompts Signed-off-by: Lennart <lvoelz@outlook.de> Co-authored-by: Lennart <lvoelz@outlook.de>
* added bias back to greedy sampling Signed-off-by: Noah Cylich <noahcylich@gmail.com> * fix: add early vocab_size==0 guard to both sampling functions Moves the empty-vocab guard to the top of cactus_sample_f32 and cactus_sample_f16 so it covers all paths (not just greedy), preventing UB from std::max_element on empty vectors. Signed-off-by: Noah Cylich <noahcylich@gmail.com> * small safety assert check Signed-off-by: Noah Cylich <noahcylich@gmail.com> * fix: replace assert with runtime guard in cactus_altup_predict_f16 assert(n <= 8) is compiled out in release builds, leaving the fixed-size float c[8] buffer unprotected. Replace with an early return. Signed-off-by: Noah Cylich <noahcylich@gmail.com> --------- Signed-off-by: Noah Cylich <noahcylich@gmail.com>
…#506) Signed-off-by: jakmro <kubamroz124@gmail.com>
* - Add strip_tag_blocks generic helper and refactored strip_thinking_block
that delegates to it, supporting multiple blocks and unclosed tags
- Add enable_thinking_if_supported param to parse_options_json and
thinking field to construct_response_json
- Refactor KV cache to store head_dim per-layer via LayerCache.head_dim,
get_layer_head_dim(), and vector-based init() signature
- Remove dim parameter from update_from_graph; derive per-layer inside loop
- Update set_window_size to iterate per-layer, skipping dim=0 layers
- Add KVCache::remove_token_range for surgical token removal from cache
- Add KVCache::compact_to_windows for per-layer compaction to target windows
- Add Model::remove_thinking_tokens (back-to-front multi-range removal)
- Add Model::compact_kv_cache virtual and get_kv_layer_dims virtual
- Add GemmaModel3n::get_kv_layer_dims override returning 0 for shared layers
- Add tool call tag auto-detection in gemma::parse_function_calls
supporting both <start_function_call> and <|tool_call> formats
- Add enable_thinking_if_supported param to format_chat_prompt and
format_qwen_style with thinking injection for Qwen models
- Add test_qwen_thinking.cpp with 6 tests for thinking option parsing,
tag stripping, response JSON, and prompt injection
- Update test_kv_cache.cpp for new init signature and fix pre-existing
bug where graph.input() returning node ID 0 caused layer skipping
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* removed qwen thinking test and updated existing qwen exhaustive test and added cli updates
---------
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* timestamps Signed-off-by: jakmro <kubamroz124@gmail.com> * timestamps 2 Signed-off-by: jakmro <kubamroz124@gmail.com> * timestamps 3 Signed-off-by: jakmro <kubamroz124@gmail.com> * sdk Signed-off-by: jakmro <kubamroz124@gmail.com> * Fix language option handling in transcribe process Signed-off-by: jakmro <kubamroz124@gmail.com> * Remove unused token handling in transcription process Signed-off-by: jakmro <kubamroz124@gmail.com> --------- Signed-off-by: jakmro <kubamroz124@gmail.com>
* Add versioned docs with quickstart and SDK chooser - Versioned docs via mike (deploy on release only) - Dynamic quickstart with snippet markers from SDK READMEs - SDK decision matrix page - Custom theme (Geist font, black bg, Cactus logo) - Version warning banner for old versions - Fix cactus_init signature in engine docs - Add vision note to Swift/Kotlin/Flutter/Rust READMEs Signed-off-by: test <test@test.com> * cleanup Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> --------- Signed-off-by: test <test@test.com> Signed-off-by: HenryNdubuaku <ndubuakuhenry@gmail.com> Co-authored-by: test <test@test.com> Co-authored-by: HenryNdubuaku <ndubuakuhenry@gmail.com>
Signed-off-by: Yujong Lee <yujonglee.dev@gmail.com>
* improve model weights discoverability Signed-off-by: jakmro <kubamroz124@gmail.com> * clean Signed-off-by: jakmro <kubamroz124@gmail.com> --------- Signed-off-by: jakmro <kubamroz124@gmail.com> Co-authored-by: Henry Ndubuaku <ndubuakuhenry@gmail.com>
…ute#519) Signed-off-by: Roman Shemet <shemet0roman@gmail.com>
Signed-off-by: jakmro <kubamroz124@gmail.com>
…te#736) The MoE layer's activation switch only handled GELU, GELU_ERF, RELU and SILU, with SIGMOID and TANH silently falling through to the SILU default. Both activations are valid `Activation` enum values accepted by the `moe_layer` builder and both kernels (cactus_sigmoid_f16 / cactus_tanh_f16) already exist, so a graph built with either produced wrong results. Add the two missing cases and a regression test that runs an identity-weight single-expert MoE layer for every activation and checks the output matches the corresponding standalone activation kernel. Signed-off-by: yus100 <90424373+yus100@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ANE (cactus-compute#734) The audio attention masked-fill constant (-1e9) overflows to -inf in the FLOAT16 NPU graph: fully-masked rows softmax to NaN and partial rows diverge on the Neural Engine. Clamp it to a finite fp16-safe -1e4 before export (soft-capped logits stay within +/-50, so masking is unchanged), and route the gemma4 audio encoder to CPU_AND_NE via the bundle manifest. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Yash sharma <trooper0018080@gmail.com>
* MOE * made lfm moe and gemma4 moe work * cleanup --------- Co-authored-by: Karen Mosoyan <karen.mossoyan@gmail.com>
- Removed the `on_token` callback from `generate_batch` method in Model class. - Introduced `supports_dynamic_batch` method to check for dynamic batch capabilities. - Replaced `batch_slot_capacity` with `set_decode_slots` to manage KV-cache slots dynamically. - Updated `decode_batch` and `generate_batch` methods to utilize dynamic slots. - Enhanced tests for batch processing to validate dynamic batch functionality. - Removed deprecated flags from CLI for model conversion and updated documentation accordingly. - Cleaned up unused tests related to batch scheduling and reshaping.
* Add cactus-code coding agent + integration Signed-off-by: jakmro <kubamroz124@gmail.com> * fix python workflow Signed-off-by: jakmro <kubamroz124@gmail.com> * parsing fixes --------- Signed-off-by: jakmro <kubamroz124@gmail.com>
…ompute#748) gemma_tools.h (reached transitively by the test binaries via utils.h) includes picojson.h from libs/, but that dir was a PRIVATE include of cactus_engine, so it was not propagated to targets that link the engine. Make libs a PUBLIC include. Signed-off-by: rshemet <shemet0roman@gmail.com>
) Reuse the Needle char-trie machinery for the Gemma tool-call path: fixes function-name prefix shadowing (e.g. send vs send_message) and enforces required arguments and enum values on a single forced call. Tokenizer-agnostic (validates decoded token strings); Needle path unchanged. Adds regression tests. Signed-off-by: rshemet <shemet0roman@gmail.com>
* gemma-metal Signed-off-by: jakmro <kubamroz124@gmail.com> * cleanup Signed-off-by: jakmro <kubamroz124@gmail.com> * cleanup v2 Signed-off-by: jakmro <kubamroz124@gmail.com> * faster Signed-off-by: jakmro <kubamroz124@gmail.com> * even faster Signed-off-by: jakmro <kubamroz124@gmail.com> * converter Signed-off-by: jakmro <kubamroz124@gmail.com> * Fix C++ test build: expose vendored libs include (picojson) (cactus-compute#748) gemma_tools.h (reached transitively by the test binaries via utils.h) includes picojson.h from libs/, but that dir was a PRIVATE include of cactus_engine, so it was not propagated to targets that link the engine. Make libs a PUBLIC include. Signed-off-by: rshemet <shemet0roman@gmail.com> Signed-off-by: jakmro <kubamroz124@gmail.com> * Trie-based grammar constraints for Gemma force_tools (cactus-compute#747) Reuse the Needle char-trie machinery for the Gemma tool-call path: fixes function-name prefix shadowing (e.g. send vs send_message) and enforces required arguments and enum values on a single forced call. Tokenizer-agnostic (validates decoded token strings); Needle path unchanged. Adds regression tests. Signed-off-by: rshemet <shemet0roman@gmail.com> Signed-off-by: jakmro <kubamroz124@gmail.com> * fixes + speedups Signed-off-by: jakmro <kubamroz124@gmail.com> * cq2/3 Signed-off-by: jakmro <kubamroz124@gmail.com> * cleanup Signed-off-by: jakmro <kubamroz124@gmail.com> * cleanup2 Signed-off-by: jakmro <kubamroz124@gmail.com> * Metal execution for vision and audio encoder towers Whisper audio embed 40.9ms vs CoreML 54.2ms, LFM2-VL vision embed 61.7ms vs CoreML 77.6ms, gemma-4 towers 3-5x faster than CPU. Adds dense fp16 GEMM via MPS with in-encoder fallback, flash attention for fp16 K/V with mask/GQA/logit-cap support, layer norm, softmax, GLU, clamp, conv1d_k3 and fp32 elementwise kernels; generic executor gains an fp16 retype pass for cast-bounded fp32 regions, GPU-dirty lazy sync, periodic mid-graph flush, and dense fp16 matmul/attention/layernorm/conv encode paths. Tower graphs stay resident on the Metal backend, vDSP FFT setups are cached, and lfm2 image embeddings resolve projector output names. Signed-off-by: jakmro <kubamroz124@gmail.com> * Generalize gemma decode into cactus-graph with automatic Metal fusion Replaces the hardcoded gemma-4-E2B execute_gpu_fused path with a general fusion-plan compiler in the graph executor: per-shape plans pattern-match the loaded graph once and fuse decomposed rope + q/k/v norms + KV append + cached attention into the fused attention kernel (owner and KV-sharing variants), sibling quantized gemvs into transform_batch/gemv_cat with consumer-ordering guards, residual rms-norm chains with folded layer scales into rms_norm_add_rms, gelu-swiglu into swiglu_transform+gemv, PLE pairs into transform_gemv, and the orthogonal lm_head with softcap into the GPU argmax/sampling tail. Plans add a liveness-scanned activation arena, a compact execution list, an embedding-fold prologue, and encode-failure rollback to the generic path. Decode matches the removed hand-fused path (119-121/111-114/95-100 tps vs 122/115/101) and prefill improves (L=16 205 vs 124 tps); tokens stay identical to CPU for cq4/cq3 and all models in the weights directory pass llm/stt/vlm suites on both backends. Renames graph backend identifiers from gpu to metal, adds ComputeBackend::METAL with Python binding, merges overlapping no-copy host buffer wraps, and runs batch generation at a fixed slot-count shape so every batch size executes identically. Signed-off-by: jakmro <kubamroz124@gmail.com> * Metal pipeline optimization: fix flushed-command-buffer sync hazard, batch-fuse unrolled tower subgraphs, MPS attention path - session_sync now waits on the last committed command buffer; previously flushed-but-unfinished GPU work was never awaited, corrupting CPU reads that raced it (output depended on flush cadence) - wrapHostPtr wraps whole VM regions, eliminating overlapping MTLBuffer wraps over the same memory - metal plan rule 8: CAT-of-matmuls over dense INDEX chains -> single batched gemm (fp16 or fp32 lhs, absorbed casts) - metal plan rule 9: CAT-of-depthwise-conv1d slices -> single depthwise conv kernel - metal plan rule 10: CAT-of-transposes at fixed source stride -> single strided copy - activation arena restricted to attention-clustered (decode) plans - MPS-based dense attention for large non-causal sequences with optional boolean mask folded via beta=1 - vectorized strided copy for contiguous inner rows; scalar precision casts execute on CPU when inputs are host-resident - gemma-3n audio tower 414ms -> 192ms, vision tower 440ms -> 376ms, outputs match CPU; llm/vlm/stt suites pass; decode tps unchanged Signed-off-by: jakmro <kubamroz124@gmail.com> * Metal encodes for the full elementwise, reduction, conv, norm, rope, and gather op set - scalar/unary/binary Metal kernels extended with exp, sqrt, cos, sin, log, abs, pow, not-equal, leaky-relu, gelu-erf, and sigmoid codes matching CPU semantics - new kernels: axis reductions (sum/mean/variance/min/max), cumsum, two-input concat, fp32-index gather, full rope (NeoX and GPT-J), maxpool1d, bilinear interpolation, generic conv1d, depthwise NLC conv1d (causal and same-k9), conv2d (dense/depthwise/pointwise), batchnorm, groupnorm, row bias add - conv1d_pointwise routed through the fp16 gemm path with fused bias pass - retype pass no longer retypes terminal nodes; their buffers are externally visible - padded cache snapshot/rollback made ring-aware for Metal-resident sliding-window caches, matching the kv_append_ring slot mapping; fixes batch-decode rollback corruption past the attention window - new test_metal_parity suite runs every newly covered op on CPU and Metal and compares outputs Signed-off-by: jakmro <kubamroz124@gmail.com> Signed-off-by: jakmro <kubamroz124@gmail.com> * Generic elementwise chain fusion for the Metal plan compiler Rule 11 fuses maximal single-consumer chains of activations, scalar ops, same-shape binaries, and clamps into one elemwise_chain_f16 dispatch with up to eight steps and three side inputs. Values round to fp16 between steps so fused output is bit-identical to the unfused per-node path. Specialized rules keep priority; chains only claim nodes no other rule clustered. gemma-3n audio tower 181ms -> 170ms, vision 376ms -> 354ms; decode unchanged; graph, parity, and engine suites pass. Signed-off-by: jakmro <kubamroz124@gmail.com> Signed-off-by: jakmro <kubamroz124@gmail.com> * Style and DRY pass over the Metal branch - remove dead code: rms_norm_rope encode path and kernel, unused gsc/smul locals, sampling_needs_adjust and stale step flags in metal_runtime - deduplicate kv-append encode variants behind shared static helpers and the persistent-activation allocation into one lambda in the executor - guard ATTENTION_CACHED plan matching against unmapped query input; add ctx/W guards to quant matmul encodes for consistency - free the ortho scratch buffer before regrowing; rename a local shadowing the buffer helper; drop an inverted empty branch in the CAT rules - update stale --backend help text that claimed gemma-only Metal support Signed-off-by: jakmro <kubamroz124@gmail.com> Signed-off-by: jakmro <kubamroz124@gmail.com> * Release Metal prefill dequant cache after each completion The prefill dense-gemm path caches fp16-dequantized weights (~3.8GB for gemma-3n-e2b cq4) that decode never reads. A guard in cactus_complete now trims the cache when the turn finishes: steady-state footprint after a completion drops from ~7.3GB to 3.5GB while benchmark prefill (2060 tps @256) and decode (110 tps) are unchanged; the next turn re-dequantizes lazily during its prefill. The MPS attention scores buffer is likewise released on invalidate_host_wraps instead of living in a function-local static. Signed-off-by: jakmro <kubamroz124@gmail.com> Signed-off-by: jakmro <kubamroz124@gmail.com> * Fix MatMul CQ staleness, pin retained outputs against fusion, PR review fixes - the tq expand cache added in 3811cbc keyed entries by the packed-weights pointer alone; remapping different weights at the same address (as the MatMul CQ test does per bit width) served stale expansions. Entries now carry a shape-and-content fingerprint and re-expand on mismatch. Fixes the unit-test failure present on main since June 21 - Metal quant matmul eligibility now requires the 4-row interleaved layout the kernels assume; synthetic flat-layout weights fall back to CPU - plan builder takes the retained/persistent node set and refuses to cover pinned nodes, so fusion clusters can no longer swallow graph outputs - run.py reads --backend via getattr like every other CLI entry point; cactus_set_backend exported from the package root; test bundle discovery prefers the highest-quality quant when several are present Signed-off-by: jakmro <kubamroz124@gmail.com> Signed-off-by: jakmro <kubamroz124@gmail.com> * Flash attention kernel and kernel-level bandwidth pass for the towers - attn_flash_f16: simdgroup-matrix flash attention (online softmax, 64-query blocks, diag-rescale with skip when the running max is stable, boolean mask support); replaces the MPS decomposition for fp16 non-causal D=64 attention. Vision attention 101ms -> 71ms with no 152MB scores buffer - rms_norm_simd_f16: simdgroup-per-row reduction, four rows per threadgroup; vision norms 42ms -> 9ms - elementwise chains now span precision-cast boundaries (per-step rounding matches unfused execution) and skip retype-claimed nodes; chain, flat binary/scalar/unary/clamp (f16 and f32) kernels process four elements per thread; broadcast binary gets a row variant without per-element div chains - tiled 32x32 transpose for trailing-dim swaps, vectorized strided scatter rows for CAT, conv2d computes four outputs per thread - gemma-3n towers: audio 168ms -> 142ms, vision 353ms -> 241ms, outputs match CPU; prefill 2225 tps @256, decode unchanged; llm/vlm/stt and all graph suites pass Signed-off-by: jakmro <kubamroz124@gmail.com> * Epilogue-style fusion: broadcast chain sides and fused residual-add rms_norm - elementwise chains accept broadcast side inputs (per-row and per-column) so post-matmul scale and bias passes fold into the surrounding chain; side buffer bindings are sized per side - rule 12 fuses ADD/ADD_CLIPPED feeding a multi-row RMS_NORM into one rms_norm_add_simd_f16 dispatch that writes both the residual sum (kept via the skip-but-preallocate action) and the normalized output, matching the unfused fp16 rounding exactly All graph, parity, and engine suites pass; towers audio 143ms vision 242ms, decode and prefill unchanged. Signed-off-by: jakmro <kubamroz124@gmail.com> * Review pass: close latent correctness holes and remove dead Metal paths Broadcast NOT_EQUAL now computes != in both bcast kernels instead of falling through to add. The disable-and-reexecute failure paths restore KV cache lengths so appends cannot double-apply. Flash attention edge tiles stage Q/O in an explicitly shared K/V pool instead of aliasing past the K array. The plan cache signature covers node count and op types, resets and the destructor invalidate plans and persistent activations, rule 12 gains the write-order guard rule 2 already had, rule 8 refuses retyped parents and anchors, and the rule 3 scale must be a graph input. Removed the dead rope_f16/lmhead_rotate encode chains, the unreachable executor branches, and leftover scaffolding; unified absent-bias binding on the dummy buffer and vectorized unary_f32 like its siblings. Parity suite grows a broadcast case and fails loudly on empty Metal output. Signed-off-by: jakmro <kubamroz124@gmail.com> * fixes Signed-off-by: jakmro <kubamroz124@gmail.com> * c Signed-off-by: jakmro <kubamroz124@gmail.com> * fixes * fixes * remove compilation warnings * fix * optimize gemma 4 26b * more optim * fixes * cleanup * more kernels * ram optimizations * update default model IDs to match existing configurations --------- Signed-off-by: jakmro <kubamroz124@gmail.com> Signed-off-by: rshemet <shemet0roman@gmail.com> Co-authored-by: rshemet <42696921+rshemet@users.noreply.github.com> Co-authored-by: HenryNdubuaku <ndubuakuhenry@gmail.com>
* fix testing infrastructure * add benchmark test * update benchamrks * add audio embed bench * refine benchmark descriptions and update table formatting --------- Co-authored-by: HenryNdubuaku <ndubuakuhenry@gmail.com>
- Updated README.md to include benchmark results for iPad M5. - Refactored test_embed.cpp to remove timing measurements and simplify output. - Removed benchmark functions from test_index.cpp to clean up test suite. - Simplified test_llm.cpp by removing redundant tests and improving output clarity. - Cleaned up test_rag.cpp by removing unnecessary initialization timing. - Streamlined test_stt.cpp by consolidating transcription tests and improving output. - Removed performance logging from test_utils.cpp and test_utils.h. - Cleaned up test_vlm.cpp by removing unused prefill tests. - Updated test.py to provide clearer instructions for enabling Developer Mode on iOS devices.
* remove coreml * fixes + cleanups * cleanups * fixes
Updated performance metrics for various models in README.
content_pos was searched starting from role_end instead of obj_start,
so a message object serializing content before role (e.g.
{"content":"hi","role":"user"}) would never match content within the
object bounds and silently drop it. Anchor on obj_start to match the
existing order-independent pattern already used for role, images,
audio, and name in the same function.
Adds test_json_parsing.cpp covering role-first, content-first, and
multi-message mixed ordering.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parse_messages_jsonsearched for"content"starting fromrole_endinstead of the object start, so a message object serializingcontentbeforerole(e.g.{"content":"hi","role":"user"}) would never matchcontentwithin the object's bounds and silently drop it.contentsearch onobj_start, matching the order-independent pattern already used forrole,images,audio, andnamein the same function.test_json_parsing.cppcovering role-first, content-first, and multi-message mixed key ordering.Fixes cactus-compute#757
Test plan
test_json_parsingvia CMake and confirmed all 3 cases pass with the fixtest_indexto confirm no regression from the header change