Add Qwen3-1.7B support for iOS

[massif-01]

Source link

https://huggingface.co/Qwen/Qwen3-1.7B/tree/70d244cc86ccca08cf5af4e1e306ecf908b1ad5e

Target platform

iOS

Use case

Qwen3-1.7B fills the capability and memory gap between the existing Qwen3-0.6B and Qwen3-4B iOS options. It is useful for private on-device language tasks such as structured extraction, instruction following, short conversation, and retrieval over user-provided context without sending that content to a server.

Preferred precision / compression

W8 per-tensor K-means-palettized transformer projections, separate INT8 per-tensor quantization for the tied embedding table, FP16 compute, and a fixed-size FP16 KV cache with a 4,096-token maximum context.

Additional context

We prepared and validated a standalone Qwen3-1.7B iOS onboarding:

• Related community conversions exist, but they do not make this request a duplicate: one publishes a dynamic INT4 GPU bundle for h18p after its static ANE path failed to invoke, while another publishes a third-party INT8 macOS-path asset with no recorded AOT target and numeric accuracy reported as not run. This request is for an Apple-main preset, recipe, metadata, tests, and a validated static h16p path rather than another hosted compiled model.
• The preset is explicitly experimental and requires --experimental.
• The reference patch adds model metadata, the frozen compression recipe, registry entry, documentation, focused tests, and the iOS conversion matrix entry.
• The patch applies cleanly to Apple coreai-models main commit 04a3fd6cfe9bfae9cf05b1f246cf915d930d1c0a.
• 28 focused unit tests and Ruff pass on that revision, and 42 related iOS conversion tests collect successfully.
• Clean export and h16p AOT compilation completed with 34 functions.
• An iPhone 15 Pro running iOS 27 completed four full six-case suites, including a 3,790-token long-context case and a reasoning case.
• The frozen W8 holdout produced mean cosine 0.996598, minimum cosine 0.959625, 98.37% top-1 agreement, and mean NLL delta 0.003167.
• Multiple W4 and mixed W4/W8 candidates were rejected before export, so the published W8 mechanism is the result of a frozen quality gate rather than a successful single-prompt smoke test.
• Apple Neural Engine participation was observed in a Core AI Instruments trace; exclusive ANE execution is not claimed.
• The public reproduction repository contains no model weights, tokenizer, compiled model, signed app, device identifier, local path, private prompt, or product source.

These measurements establish conversion fidelity for the frozen holdout, not a general downstream benchmark ranking. Full WikiText-2 perplexity is not included in the current evidence and is not represented as complete. The repository provides the recipe, reference patch, reproduction procedure, sanitized evidence, and resource-free companion app.

Related public artifacts, the ANE/GPU distinction, and the limits of cross-project comparison are documented in RELATED_WORK.md.

[massif-01]

Update: reproducible INT4/GPU artifact and paired W8/ANE evaluation

Since opening this model request, I completed a second, independent Core AI path for Qwen3-1.7B and published the full model artifact, sanitized physical-device benchmark apps, deterministic scoring tools, and aggregate comparison evidence.

Public artifacts

• Downloadable INT4/GPU model: https://huggingface.co/massif/Qwen3-1.7B-CoreAI-GPU-INT4-4K-h16p
• Reproduction and evaluation repository: https://github.com/massif-01/qwen3-1.7b-coreai-reproduction

The tested INT4 configuration uses:

• Qwen/Qwen3-1.7B at revision 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
• Apple coreai-models at 85e2f2d8c6433825e64219a32693ed1a4aed519f
• block-32 INT4 weights with FP16 compute
• growing/dynamic FP16 KV cache
• 4,096-token context
• iOS 27, h16p, GPU-preferred AOT compilation

It was validated on a physical iPhone 15 Pro running iOS 27 with Xcode 27 Beta. Core AI selected the coreai-pipelined GPU path, and the artifact completed dynamic-KV generation on device.

Paired quality comparison

I ran both configurations on the same frozen 300-example subset of the CMRC2018 validation split (100 short, 100 medium, and 100 long unique contexts):

Metric	W8 / static KV / ANE	INT4 / dynamic KV / GPU
Exact Match	59.33	57.67
F1	81.70	81.42
Byte-identical predictions	—	187 / 300

For the paired comparison, W8-minus-INT4 F1 was +0.28 points with a 95% bootstrap interval of -2.44 to +3.00 and a paired sign-test p = 0.614. The EM difference was +1.67 points with a 95% interval of -3.00 to +6.33 and p = 0.576.

This 300-example experiment did not establish a statistically detectable quality difference, but it also does not establish equivalence.

The protocol froze a separate heuristic gate before the run: INT4-minus-W8 had to be at least -1.0 point on both EM and F1. INT4 passed that rule on F1 (-0.28) but failed it on EM (-1.67), so the overall heuristic gate is reported as failed. I am not claiming quality parity.

Size, memory, and speed

Measurement	W8 / ANE	INT4 / GPU
Compiled bundle	1,714.4 MiB	924.6 MiB
Peak process RSS	2,865.0 MiB	1,995.3 MiB
161 input / 60 output, total	3.259 s	2.592 s
3,790 input / 10 output, total	6.116 s	13.717 s
120 input / 256 output, total	13.292 s	7.159 s

There is no single performance winner across workloads:

• INT4/GPU substantially reduces install size and memory pressure and is faster for sustained decoding.
• W8/ANE has much faster time to first token and long-prompt prefill, and it was faster on the short-answer CMRC workload.
• W8 has the slightly higher aggregate quality point estimate, with the uncertainty and failed heuristic gate reported above.

The W8 measurements remain tied to the explicitly recorded device-validated artifact identity. The current downloadable payload on the related W8 Hub page is not presented as byte-identical to that A/B artifact; this boundary is documented in the reproduction repository rather than silently substituting binaries.

Runtime observation: Qwen3 no-thinking mode

For this runtime, Qwen3 no-thinking mode had to be applied at chat-template time:

additionalContext: ["enable_thinking": false]

Appending /no_think to the prompt was not equivalent. The pinned high-level runtime did not expose this as a per-request generation option, so the benchmark used a narrowly scoped patch at the tokenizer.applyChatTemplate call. The exact diff and rationale are published in the reproduction repository. A production API would ideally expose thinking mode per request rather than hard-code it globally.

Request

These results strengthen the original request for official Qwen3-1.7B iOS support. The model fills a useful capability and memory gap between Qwen3-0.6B and Qwen3-4B, while the two validated runtime profiles serve different product workloads.

Please consider adding an official Qwen3-1.7B preset, conversion recipe, metadata, and test coverage, with the static-KV/ANE and dynamic-KV/GPU tradeoffs represented explicitly rather than treating either configuration as universally best.

I am happy to provide additional reproduction details or focused evidence if useful for triage.