feat(app): embed isolated Hermes profile sessions - #220
Open
Cyb3rb1ade wants to merge 240 commits into
Open
Conversation
Adds an AppKit-level integration test target (MTPLXAppHostTests) that drives
the real ComposerInputTextView.Coordinator against a live NSTextView,
exercising the actual IME mechanism (setMarkedText / insertText).
It exists only to make the fix verifiable in-tree and is reverted by the
final commit of this PR, so the net diff is the fix alone:
* Check out THIS commit and run
cd apps/MTPLXApp && swift test --filter ComposerIMEIntegrationTests
-> testPreeditStaysOutOfBindingUntilCommitted FAILS: the composer has no
marked-text guard, so IME preedit leaks into the SwiftUI binding.
* Check out the next commit (the fix) and run the same
-> it PASSES.
Shippable regression coverage lives in the fix commit as pure-logic unit
tests (MTPLXAppCoreTests/ComposerTextSyncTests); this harness is the
on-machine proof, not a permanent addition to the test surface.
The composer is an NSViewRepresentable bridging a SwiftUI @binding<String> to an AppKit NSTextView. During IME composition — any input method that builds a character from intermediate keystrokes: CJK (pinyin, kana-kanji, hangul), dead-key accents (´ + e -> é), and more — the text view holds uncommitted "marked text" (the preedit) that mutates every keystroke, while SwiftUI binding propagation lags a render cycle. Two paths then race: - textDidChange published the provisional preedit up into the binding. - updateNSView wrote a now-stale binding value back into textView.string, which tears down the live marked-text session and drops the characters being composed — so non-ASCII input "disappears" in the composer. Gate both crossings on NSTextView.hasMarkedText(): never publish provisional preedit, and never overwrite the text view while a composition is in flight. The committed text arrives in a later textDidChange with no marked text. The decision logic is extracted into ComposerTextSync in MTPLXAppCore so it is unit-tested without a live NSTextView, matching the existing test strategy (logic in Core, views kept thin). ASCII typing is unaffected: it never enters a marked-text phase.
… this PR)" This reverts commit 752c2c0.
…-ime-composition Chat composer: preserve IME composition instead of dropping input
…penalties Add presence_penalty / frequency_penalty (per-position, vLLM-exact) — fixes youssofal#102
… Penalty dial Completes the presence/frequency penalty feature that PR youssofal#120 started. PR youssofal#120 wired server-default flags into the samplers but typed request fields were still silently dropped (issue youssofal#102): ChatCompletionRequest / CompletionRequest use extra="allow", so client-sent presence_penalty and frequency_penalty vanished without effect or error. - Typed presence_penalty/frequency_penalty fields on ChatCompletionRequest and CompletionRequest (range-validated -2..2). - Threaded through all 11 chat/completions dispatch sites into _run_generation -> _generation_params with precedence: request value > server default > 0.0 (explicit 0 beats a non-zero server default). Gated by client_controls_allowed exactly like temperature; observability (request_/effective_presence_penalty) and ignored-client-field reporting extended to match. - _run_generation solo lane gained the params (youssofal#120 covered only the AR-batch lane). - Live settings: presence_penalty/frequency_penalty accepted by /v1/mtplx/settings (MTPLXSettingsUpdate, DASHBOARD_MUTABLE_SETTINGS_KEYS, _coerce_setting float coercion + range check -> 400 out of range), applied onto state.args.default_*_penalty, reported by GET settings. - Web chat UI: "Presence Penalty" slider (0-2, step 0.05, "off" at 0) in the Sampling sidebar, synced to /v1/mtplx/settings like the other dials; server-side default_settings extended. - tests/test_penalty_request_wiring.py: 10 tests covering precedence, typed parsing, end-to-end capture through TestClient, controls gating, live settings update, range rejection, draft-sampler isolation. QA (2026-07-02, fans pinned+verified for every generation): - temp-0 seed-0 litmus: penalties-unset byte-identical to explicit-0; presence 2.0 diverges. Server-owned settings path (no client headers) proves the app lane too. - bench tune 192: AR 29.8 / D1 50.8 / D2 55.9 / D3 60.16 BEST vs baseline 59.31 (flat-or-better), peak memory identical 15.11 GiB. - quick suite: all contract lanes flat-or-better vs wave-0 baseline; long-tool-history FAIL is the pre-existing unknown-test bug on main. - pytest 1593 passed / 4 skipped; ruff: only the 4 pre-existing errors.
… start serve/quickstart accept the flags since PR youssofal#120, but mtplx start (the documented first-run path) neither exposed nor forwarded them, so a server default set at start was silently lost. Adds the two flags to the start parser and carries them through _with_server_policy_args into the spawned server.
LM Studio-style "Presence Penalty" slider (0-2, step 0.05, default 0, help text with the Qwen guidance: keep 0 for coding, raise for anti-repetition/creative) under SAMPLING next to Top P/Top K. presencePenalty flows end-to-end: MutableSettings (presence_penalty JSON key) -> MTPLXBackendStore live-settings patch/persist/merge paths -> daemon /v1/mtplx/settings round-trip -> persisted in MTPLXAppConfiguration; MTPLXChatClient.ChatRequest carries the field per-request as well. QA: swift tests 437 pass; built app + app-owned daemon exercised live - dial steps propagated to daemon settings (0.2 -> 0.3 observed via GET /v1/mtplx/settings) and persisted to settings.json; in-app chat with penalty active streamed at 50.7 tok/s.
Adds presence_penalty to MutableSettings and a NumberField dial (0-2, step 0.05, with description) in ControlsSidebar; rebuilt mtplx/dashboard/_static (bun run build).
README server section documents the sampler dials incl. presence penalty defaults; TROUBLESHOOTING gains "Model Repeats Itself / Loops" with the Qwen guidance (0 for coding, ~0.5-1.5 for anti-repetition).
…ied, held through postcommit (youssofal#127) Root causes of the reported fan flakiness (issue youssofal#127 + founder reports of fans not ramping until output generation): 1. The ramp was issued at generation dispatch — after routing, transcript canonicalization, prompt encoding, and queueing — so long prefills ran their lead-in on silent fans. 2. SmartFanController.begin_request ran set_thermal_profile synchronously under the controller lock: serial subprocess probes with 15 s timeouts on the request path, failures swallowed with no verification or retry. 3. Fans restored 0.2 s after the HTTP request finished while the idle postcommit re-prefilled the whole conversation at 100% GPU — the "fans stop while the Mac is still cooking" report. QA measured this phase at 5.5 minutes for a 47k-token conversation. 4. Switching the server to Max mode called smart_fans.restore_now(wait =False) BEFORE pinning max: the delayed async smart restore then fired AFTER the max pin and silently dropped fans back to auto while the UI showed Max — the "max released early" report. The overhaul: - SmartFanController is now a desired-state machine driven by one dedicated worker thread. begin_request/end_request never block and never run a subprocess on the caller thread; the lock only guards state. Ramp latency, target verification, actual-RPM verification, and attempt counts are tracked and exposed via status(). - After commanding max the worker verifies the daemon accepted the target RPM (fan_summary) and retries the command once on failure, then polls until the physical ramp is visible (30 s bound). A failed ramp logs one actionable line and does not hammer the daemon until a new lease generation arrives. wait_for_ramp() gives bench/QA/test code a synchronization point. - New _SmartFanArrivalMiddleware leases the fans the moment a generation POST (/v1/chat/completions, /v1/completions, /v1/messages) arrives — before body parsing and prompt encoding — and releases when the full response (including the stream body) has been sent. Registered inside the auth middleware so unauthorized requests never ramp. Open WebUI background task probes are skipped. - The idle postcommit now holds a fan lease from schedule time until the job resolves (try/finally, plus release-on-submit-failure), so fans stay ramped through the post-generation GPU phase. - Restore debounce raised 0.2 s -> 2 s so agent tool loops reuse the ramp instead of flapping fans between calls. - Max-mode switch now uses the new smart_fans.detach() (drop leases and pending work WITHOUT touching hardware) so the max pin cannot be raced back to auto; default-mode switch drains the smart controller synchronously before the verified restore. - /health gains smart_fan_target_verified / smart_fan_actual_ramp_ verified / smart_fan_ramp_latency_s / smart_fan_actual_ramp_latency_s. - TROUBLESHOOTING: new section explaining the post-generation postcommit GPU phase and the health fields (answers the youssofal#127 reporter's question directly). QA (live server, thermalforge status sampled every 0.4 s): - 186k-char prompt: target RPM flipped to max 0.66 s after request arrival (ramp_latency_s 0.19, target verified attempt 1), actual RPM 7636 by 1.5 s; first token at 229 s — the entire prefill ran on max fans (previously the lead-in ran silent). - Fans held at max through the 5.5-minute idle postcommit after the stream closed; restored to auto only after it stored (leases 1 -> 0). - 3-turn back-to-back session: fans stayed manual across all turns (no flap), restored after the session drained. - Smart->Max switch with active smart state: pin still manual/7826 5 s later (old code dropped to auto). Max->default restores verified. SIGTERM shutdown restores fans to auto. - tests/test_thermal.py: 31 pass (5 new: non-blocking begin, retry-once, failure-without-hammering, detach-no-hardware, updated lease test). - Full pytest 1597 passed / 4 skipped; ruff clean on changed files.
…t-party coercion (youssofal#57) Users loading third-party or legacy artifacts were shown a first-party canonical id: issue youssofal#57 (Qwen3.6-27B-MTPLX-Optimized reported as mtplx-qwen36-27b-optimized-speed) and the PR youssofal#77 report (a samuelfaj 35B build served as mtplx-qwen36-27b-optimized-quality). Four fuzzy inference lanes in default_models.py could each claim a non-first-party artifact as first-party: - artifact_role substring matching ("quality"/"speed"/"gdn8"/"fp16") — third-party builds made with mtplx forge carry these roles too; - verified_on.model substring inference (same problem); - precision_variant=fp16 coercion to the first-party FP16 id; - quantization-layout inference (Q4 vs Flat8 "upgrading" the legacy artifact's identity — the exact youssofal#57 report); - loose family-name coercion (any "qwen3.6-35b-a3b"+"mtplx" string claimed as the first-party 35B artifact). New contract: a canonical mtplx-* id requires a true first-party match — an explicit public_model_id/served_model_id/model_id in mtplx_runtime.json, or an exact first-party name (public id, HF repo id, released folder name; the loose 9B/35B family matches are now exact released-name matches, including the CyanKiwi CleanRecipe local build of the released 35B). Everything else serves under its sanitized actual artifact name. Regression matrix added to tests/test_default_models.py: nom666 Qwopus 4bit-Speed/8bit-Quality (real repos from the June triage), the samuelfaj 35B case folded in from PR youssofal#77 (credit wwadge for the report), a 35B family remix, artifact_role/precision/verified_on non-coercion, the youssofal#57 legacy-name quantization case, and a first-party matrix across all released ids. QA: served the real nom666--Qwopus3.6-27B-Coder-MTPLX-4bit-Speed artifact — startup banner, /health, /v1/models, /v1/mtplx/settings, and the chat completion "model" field all report nom666-qwopus3.6-27b-coder-mtplx-4bit-speed (previously mislabeled lanes). First-party flagship continues to serve mtplx-qwen36-27b-optimized-speed (same session, waves 2-3 QA servers). Full pytest 1609 passed / 4 skipped.
…pad embeddings (youssofal#103) Investigation result first (greedy A/B, temp 0, seed 0, cache bypass, two near-identical dashboard screenshots, 200- and 600-token runs): MTP greedy output is byte-identical across D1/D2/D3 with images, both before and after this change, and the model correctly identifies the single planted difference. Verify-side exactness with images was never broken — the depth-dependent hallucinations reported in youssofal#103 at temperature are sampling variance (different depths consume the RNG differently, drawing different samples from the same exact distribution), not distribution corruption. A text-only control also showed the chunk-size output sensitivity is bf16 chunked-prefill non-associativity, not vision-specific. Consequently the depth-1 image gate contemplated in the plan is NOT shipped: exactness is proven, and capping depth would only slow vision requests down. The real (draft-side) defect fixed here: _append_mtp_history passed raw token ids to the MTP head, so the committed history embedded image-pad tokens where the trunk prefill saw spliced vision rows. The draft head's history context over image spans was therefore built from meaningless pad embeddings. Verify authority means this could never corrupt output, but it degrades draft/trunk alignment over image spans. - vision/splice.py: new cursor-free spliced_embeddings_for_window (the history stream pairs hidden t with token t+1, so its embedding window is shifted one token right of the trunk chunk; rows are read at an explicit offset via pad prefix counts, leaving the trunk's sequential cursor untouched). Shared row-splice helper extracted. - generation.py: both prompt-history append paths (sustained streaming chunk loop + non-sustained full-sequence path) now build the shifted spliced window and thread it through _append_mtp_history (input_embeddings). Zero change when no vision splice is present. - runtime.update_mtp_cache / mtp_patch _mtp_core + mtp_update_cache: optional input_embeddings overrides embed_tokens(next_token_ids); runtime raises instead of silently dropping the rows on MTP backends that don't accept them. - tests: 4 new splice-window tests (row alignment vs the trunk lane, offset reads, overflow, no-pad fast path); existing sustained-history test stubs extended with the new parameter. Measured effect (D3 greedy, deterministic per lane, fans pinned): - exhaustive single-image describe: mean accept prob by depth [0.862, 0.720, 0.593] -> [0.903, 0.719, 0.622], accepted 411/567 -> 415/555 (fewer drafts wasted). - two-image compare: [0.969, 0.901, 0.851] -> [0.951, 0.896, 0.840] (slightly down, within the fluctuation of a changed-history regime). - outputs byte-identical pre/post in all lanes, as exactness demands. Full pytest 1613 passed / 4 skipped; ruff clean on changed files.
…ions/actions/setup-python-6.3.0 Bump actions/setup-python from 6.2.0 to 6.3.0
…raphy-48.0.1 Bump cryptography from 48.0.0 to 48.0.1
Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.2 to 7.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6.0.2...v7.0.0) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [idna](https://github.com/kjd/idna) from 3.13 to 3.15. - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.md) - [Commits](kjd/idna@v3.13...v3.15) --- updated-dependencies: - dependency-name: idna dependency-version: '3.15' dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…te-1.3.1 Bump starlette from 1.0.0 to 1.3.1
…ions/actions/checkout-7.0.0 build(deps): bump actions/checkout from 6.0.2 to 7.0.0
build(deps): bump idna from 3.13 to 3.15
youssofal#134 by @Jonathangadeaharder, applied locally for v2) Qwen3_5MoeForConditionalGeneration multimodal checkpoints (Ornith-1.0-35B) ship the vision tower under model.visual.*; the vision module only knew mlx-vlm's vision_tower.* layout, so vision_spec_for_model_dir returned None and image requests got HTTP 400 despite a complete tower in the index. resolve_vision_prefix() picks the active prefix; text-only repos still resolve to None. Mirrors the trunk loader's existing _mlx_key remap. Verified locally: test_vision_tower 22 passed (2 new), compressed_tensors + artifacts suites green.
…oussofal#133 by @shiningliang, applied locally for v2) _quickstart_print_dashboard_handoff does `from mtplx.ui import pretty_path`, but the helper only existed as onboarding._pretty_path — the live-dashboard lane of `mtplx start` crashed with ImportError right after the model check. Lazy module __getattr__ re-export keeps mtplx.ui import-cheap. Verified locally: import crash reproduced on main, gone after; PR's two regression tests green (conflict with the neighboring Hermes-config tests resolved by keeping both).
…rMarioYL, applied locally for v2) Key the finished set by request_id (O(1) dedup vs the O(n) identity scan), cap retained RequestState objects at 4096, and report the exact cumulative count via finished_total (snapshot()["finished"] keeps its meaning; new finished_retained gauge). Long-running servers no longer accumulate every finished request forever. Verified locally: test_batching_foundation + test_model_scheduler green (14 passed), including the new bounded-retention and idempotency pins.
Session-cache v2: boundary-true GDN restores, O(1) RAM restores, SSD cold tier that survives daemon restarts, store-on-prefill keyed to the request's token prefix so tool-call turns chain warm instead of re-prefilling whole transcripts. Turbo profile promoted to the default for the quantized 27B flagships: NAX verify kernels (vk_k / vk-q8) plus context-routed compiled verify with a per-model quantization gate. Long-context kernel wave: packed-GQA verify attention and commit-first KV donation in compiled verify (64k decode +12%, 128k 17 -> 20+ tok/s, peak memory -8..-16 GB). Stability: the app health watchdog treats liveness as transport truth and no longer terminates a healthy daemon on an undecodable /health payload (youssofal#105); transformers pinned <5.13 after 5.13.0 broke mlx-lm imports on every fresh install (youssofal#135, youssofal#136, community PR youssofal#137 by @davidtai). Agent protocol pass: OpenCode plan->build no longer breaks the cache or hides tools, prefix-stable transcripts, per-request presence and frequency penalties end to end, served model identity is contract-match-only (youssofal#57), Hermes config merge-preserved (youssofal#131). Vision under MTP consumes spliced vision rows in the draft history (youssofal#103). Smart fan control ramps at request arrival, verifies RPM, holds through postcommit (youssofal#127). App chat: live streaming markdown, per-turn activity strips, sources footer, IME composition fix (community PR youssofal#119 by @penta2himajin). Startup ready in ~2s with silent background warmup. All profiles run stock PyPI MLX; the vestigial fork metadata is gone (youssofal#129). Full notes: docs/releases/v2.0.0.md and CHANGELOG.md.
The v2 turbo default now covers every dense catalog model on every Apple Silicon generation: - 27B Optimized-Speed-FP16 (the M1/M2 routing target) defaults to turbo: 19-31% faster decode than the 2.0.0 default across 0.5k-32k context, true-AR multiplier 1.34x to ~2x, exactness gated on the real weights. - New 6-bit affine verify kernels (split-K hexpack family): the 6-bit 9B tier gains 33-62% decode and 43% 2k-prefill under turbo. Qwen 3.5 9B and 9B FP16 now default to turbo. - New model: Qwen3.6-27B-MTPLX-Optimized-Quality-FP16, the missing M1/M2 quality artifact, published and wired into the app picker, CLI catalog, and chip-aware routing (a Quality pick on M1/M2 resolves the FP16 sibling, like the speed lane). Includes the reusable fp16-sibling converter script. - Load-time kernel self-validation: every turbo lane checks itself against stock MLX in the model's exact dtype/quantization at boot; a mismatching lane falls back to the stock path for the session and the verdicts surface as kernel_selfcheck in /health. Worst case on unusual silicon is 2.0.0 speed, never wrong output. - MTPLX_FORCE_GPU_FAMILY_FALLBACK=1 rehearses the exact M1-M4 code path on newer machines; new kernel-matrix CI workflow runs kernel exactness plus a live turbo smoke on real M1 runners. - Honest exclusions: 35B A3B MoE and Gemma 4 keep sustained (their architectures bypass these kernels); the 4B keeps sustained (turbo measured slightly slower at matched depth); compiled verify stays off for 6-bit models. Full pillar regression against 2.0.0 (decode 0.5k-128k, prefill, warm TTFT, peak memory) flat or better on every unchanged model.
… tag split (youssofal#149) The streaming reasoning splitter held back only 7 trailing bytes (sized to </think>) while the alias set recognizes tags up to </reasoning>; a longer tag split across a streaming chunk boundary leaked the reasoning block and raw markup into user-visible content. Hold back the full 16-byte tag window, mirroring the disabled-splitter path.
…#148) The quickstart onboarding screens printed hardcoded http://127.0.0.1:8000 Web UI and dashboard URLs regardless of --host/--port. Thread host and port through the onboarding flow and render wildcard binds with the shared bind-vs-connect resolver.
Multi-turn agent sessions now render reasoning history on Qwen's trained contract, which fixes the plan-execution repetition marathons reported on the 27B models: - Scoped reasoning history (default on Qwen 3.6/3.5 templates): the chat template's own rolling checkpoint governs multi-turn rendering. Completed turns render without think scaffolds; the active round keeps its full reasoning, now including the structured reasoning_content agent clients send (previously dropped). The captured looping session goes from 3/4 repetition marathons to 4/4 immediate healthy tool calls. --preserve-thinking on/off/scoped pins the behavior explicitly; templates without the rolling checkpoint (Gemma 4, custom) are untouched, and explicit on/off keeps cache identity byte-for-byte. - Root-cause receipts: the loop reproduces on full bf16 weights under the old history rendering and vanishes on clean context at every precision level — context construction, not quantization damage and not the model alone. - Serve on any host and port from the app (youssofal#109): wildcard binds resolve to connectable addresses, the port preflight tests the address family the daemon will bind, an API-key mismatch reads as live-but-unauthorized instead of lost, and LAN serving surfaces its API-key requirement before launch. - Warm prefix reuse for every agent client (youssofal#138): the block-prefix restore lane engages for all clients under boundary-true restores, not just OpenCode's tool contract. - Settings Off means Off (youssofal#140): the app passes the SSD session cache mode explicitly, including off, so session-bank stops growing back. The runtime venv self-heals after app updates (youssofal#139). Ctrl-C returns the terminal within a bounded drain under open SSE streams (youssofal#124). - Opt-in Loop Guard (MTPLX_LOOP_GUARD=1) for repetition-damaged third-party quants: loop-armed steering that is bit-exact until a real verbatim loop is detected and never touches tool-call content. Default OFF — MTPLX does not alter sampling unless asked. Community fixes in this release: streaming reasoning-tag leak fix (youssofal#149, Osamaali313) and quickstart host/port rendering (youssofal#148, Takeshi HASEGAWA).
The Hermes CLI writes base_url: '' for provider profiles without a custom endpoint (for example openai-codex). The routing parser rejected the quoted empty scalar, failed the whole model block, and marked the profile unavailable — disabling it in the picker and hiding its sessions. A quoted empty string is now a valid empty scalar and an empty base_url is treated the same as an omitted one. Regression test pins the openai-codex profile shape.
Qwen3.8-27B day-one prep. The vLLM reference contract reads the MTP layer count generically (N stacked full-attention draft layers); MTPLX's tensor gate and draft forward were frozen at the depth-1 template. A checkpoint declaring mtp_num_hidden_layers > 1 died at load with invalid-mtp-tensor-layout, and _mtp_core only ever ran mtp.layers[0]. - constants.expand_mtp_layer_keys(): every named expected-key set stays the canonical depth-1 template; expansion replicates the per-layer keys across the declared count (identity at N=1, so depth-1 behavior is byte-identical). - artifacts._mtp_expected_key_set() + onboarding._expected_embedded_mtp_keys() + mtp_patch._mtp_contract_for_weight_keys() now expand by the config's declared layer count (numbered-expert MoE path was already N-aware). - _mtp_core runs all draft layers in sequence, one KV cache per layer (make_mtp_cache was already per-layer); cache-length mismatch fails loud instead of silently truncating. Tests: tests/test_mtp_depth_n.py — expansion identity at N=1, N=3 replication, tensor gate pass/fail at N=2, prequantized contract detection at N=2, and a live two-layer inject + mtp_forward on a tiny real qwen3_5 TextModel with donor-harvested weights (lockstep cache offsets pinned). Regression: test_mtp_patch/test_artifacts/test_onboarding/test_forge_cli 237 passed, 0 failed (exit 0, captured log).
…oussofal#175 Scratch-venv proof (2026-08-02): transformers 5.14.1 + mlx-lm 0.31.3 — import clean (the 5.13.0 AutoTokenizer.register crash class does not fire), real Optimized-Speed tokenizer encode/decode round-trip, and tool-bearing chat template fingerprints byte-identical to the 5.8.0 baseline.
Field A/B in youssofal#227 (M5 Pro, 48GB): under bursty agent load the SoC soaks past 90C during a burst, the smart lease restores auto after its 2s idle debounce, and Apple's auto curve (~5,000 rpm) never drains the soak in the gaps — so every following turn runs throttled (-51% decode, recovering only after 100s of forced max fans). Fix: after the idle debounce, the worker now probes the die temperature (new soc_temperature_c(): hottest TC* sensor from ThermalForge status JSON, MTPLX_SMART_FAN_SOAK_SENSOR to pin a key) and holds max fans until the die cools below MTPLX_SMART_FAN_SOAK_RELEASE_C (default 75C), re-probing every 5s. Fails open: no readable die temperature -> legacy instant restore; and MTPLX_SMART_FAN_SOAK_HOLD_CAP_S (default 180s) bounds the pin regardless of sensor state so an idle machine always gets its fans back (the fans-left- pinned lesson). Explicit synchronous restores (wait_for_restore, restore_now, detach) bypass the hold. Hold state is surfaced in status()/health (soak_holding, soak_last_temp_c, soak_release_reason). Tests: 9 new (hold-until-cooled, hold-cap bound, probe-failure fallback, env disable, bench-lane bypass, sensor selection/sentinel/pinning/absence); thermal suite 46 passed, 0 failed, ruff clean. Live thermal A/B replicating the issue's arms is deferred to the next controlled-fans benchmark session per tonight's no-benchmarks constraint.
…gated, bit-exact) — davidtai
…l (PR youssofal#209 review) Two review edits on the merged PR: (1) an explicit opt-in via MTPLX_LINEAR_GDN_TAPE_IMPL=headquarter no longer silently degrades to the incumbent when the kernel module fails to import — narrowed to ImportError with a one-time warning; (2) the bit-exactness test clears the env var so a stray headquarter setting in the invoking shell cannot turn the reference arm into headquarter-vs-headquarter and pass vacuously.
…nes (o-LoRA routes, adaptive width, wide-M3, attention island) — davidtai
…, portability, test hygiene Four required edits on the merged PR: 1. runtime.py: canonical_mixed_route now binds only on the explicit MTPLX_DSV4_O_LORA=gather_qmm opt-in. The default "cached" load takes the per-module dense route (bit-identical on the canonical artifact per the PR's own test_cached_dequant_is_bit_identical) instead of hard-validating the exact DeepSeek-V4-Flash topology — which refused every non-canonical DSV4 MTP artifact (8-bit/bf16 user conversions, other group sizes) that loads fine on v2.4.2. Also restores lazy wo_a dequant on default loads (~2.7 GiB eager materialization avoided). 2. test_deepseek_v4_attention_island.py: module-level mx.set_default_device (the PR youssofal#216 landmine — leaks CPU process-wide at pytest collection) replaced with the autouse save/restore fixture from the PR's own o-LoRA test file. 3. Island bench pair parameterized like the adaptive-width pair (MTPLX_DSV4_PYTHON/BENCH_DIR/MODEL_PATH/PROMPT_FILE/QUALITY_PLIST*/ EXPECTED_WIRED_LIMIT_MB env): no more /Users/davidtai venv, bench, model, LaunchAgent paths; wired-limit gate is operator-pinnable and 0-disable. 4. Frozen per-file SHA256 SOURCE_MANIFEST removed from the arms script and its live-tree assert loop from the bracket test — it duplicated the exact-commit + clean-worktree gates and broke on every legitimate commit to runtime.py/deepseek_v4.py (including this train's). Plus the reviewer's hygiene suggestion: the published no-developer-absolute- paths gate now covers all three DSV4 bench pairs, not just adaptive width. Receipts: 179 DSV4 lane tests green (island module in its own pytest process), ruff clean. K<=3 ruling verified respected by review; bf16 lanes remain opt-in and unpromoted.
… lane, D1 fusion + async scheduling receipts (davidtai)
… portable scratchpads, hygiene Four review edits on the merged port: 1. alt_prefill_forward: enabled-but-unwired prefill flags (P1-P4) now raise NotImplementedError instead of silently running stock — same anti-fake-win contract the decode lane already enforces, so an A/B can never measure a "win" against a no-op arm. 2. Seven docs bench scratchpads dropped their hardwired /Users/davidtai worktree/model/bench paths for repo-relative sys.path roots plus MTPLX_LAGUNA_MODEL_DIR / MTPLX_LAGUNA_BENCH_DIR env overrides. 3. Unused imports removed from laguna_alt_step (dataclasses.field, FULL, StepGeometry). 4. test_metal_kernel_matches_reference restores the default device in a try/finally like its CPU siblings instead of leaking the gpu pin into the rest of the session. Receipts: 28/28 alt-lane + steel-attn tests green (digest-equality and fail-loud contracts included), product files ruff-clean. Notes for the maintainer: the +5.8% decode receipt is an unwired benchmark-lane result — wiring D1+S1 into the reference lane is follow-up work; 15 of 19 ported kernels are receipt-only (unreachable from product code) and currently ship in the wheel — packaging policy call deferred.
…iptor - HY_V3_MTP_DESCRIPTOR (backend_id hy_v3_mtp): official Tencent sampler defaults temp 0.9 / top_p 1.0 / top_k off (generation_config.json of tencent/Hy3), qwen3-style reasoning codec, pre_norm hidden, contract-gated. Registered in DESCRIPTORS_BY_BACKEND_ID so the public serve wrapper, the in-server defaults, and /health all resolve it (previously fell back to the Qwen 0.6/0.95/20 coding sampler). - Server: _model_declared_sampler_defaults() — hy_v3 artifacts' own generation_config.json applied at parse time when flags don't override (covers direct python -m mtplx.server.openai launches too). Deliberately scoped to hy_v3: flipping Qwen/Gemma defaults from artifact metadata would change shipped behavior and needs its own A/B. - Reasoning codecs: Hy3 renames chat control tokens with an :opensource suffix at the same ids (<think:opensource>...</think:opensource>). Close-tag regexes now tolerate suffixed spellings (open/control already did), and the streaming holdback window grows to cover the longest suffixed spelling — 16 chars could split </think:opensource> across emits, permanently missing the close and classifying the whole tail as reasoning. - tests: un-skip the hy_v3 suite (the vendored class makes it runnable on released mlx-lm) + descriptor/sampler/split/stream regression tests. 7/7; reasoning stream suites 22/22. (cherry picked from commit 7885e882ad06e9af6a926e3fcecabcbde618156b) (cherry picked from commit 2c02204fc8d2af5eea34ef03aedc0dc8b1bfcd25)
The first real HY3 OpenCode run exited after printing <tool_calls:opensource> as visible text and created no files. MTPLX only understood Qwen-style tool envelopes, while the official HY3 tokenizer emits suffixed tool-call, separator, argument-key, and argument-value tokens. Add a schema-aware suffix-token parser and streaming adapter, preserve JSON-shaped string arguments, validate emitted calls at the OpenAI boundary, suppress native control markup, and cover character-split parallel calls. The native app now launches HY3 with its tokenizer profile in both the command and environment instead of carrying the Qwen profile label. Verified with the complete tool-stream translator, oMLX bridge, and server test modules plus a focused Swift command-builder test. This is a local checkpoint before rebuilding the experimental bundle and repeating the same OpenCode project. (cherry picked from commit dbab059483cc4b8d8eda13e0864ccd00b8dbf82a)
…v3 at load No released mlx-lm ships a hy_v3 class (ml-explore/mlx-lm#1211 open; the MTP surface only exists in the #1485 stack). The 2.1.0 backend was therefore inert: is_hy_v3_mtp_config dispatched, but mlx_lm.utils.load raised on the missing module for every real checkpoint. - mtplx/vendored_hy_v3.py: the #1211+#1485 reference implementation (kernelpool + eauchs lineage, as shipped by ox-ox), imports made absolute. Keeps and uses the MTP layer (MTPBlock + predict_next_tokens + return_hidden_states) instead of stripping it. - install_hy_v3_model_shim(): registers the vendored module as mlx_lm.models.hy_v3 before mlx_lm.utils.load resolves the model type. Prefers a future upstream module IF it exposes predict_next_tokens; a base-only upstream (which strips MTP in sanitize) is overridden. - runtime.load: install the shim for any hy_v3 config (with or without head), next to the qwen3_5_mtp trunk-shim precedent. Verified: synthetic hy_v3 (real 120832 tokenizer, mixed-quant recipe shapes, depth-1 MTP) loads through mtplx.runtime.load, AR forward + return_hidden + mtp_forward + make_cache/make_mtp_cache all pass; generate_ar and generate_mtp1 produce tokens with verify_calls/bonus/correction counters live. tests: test_hy_v3_mtp_backend.py + test_artifacts.py 76/76. (cherry picked from commit 0fc4bfe7a66ee7247d4737fd103b87f2f37fcbc3) (cherry picked from commit a7908fc2a93f13cb0c653d11bf5683872682902a)
…res class
Fire-drill finding (Qwen3.8 day-one prep): a checkpoint with a fresh
model_type string but a known schema — the exact Qwen3.6 precedent
(shipped as model_type qwen3_5) that Qwen3.8 is expected to repeat — passed
`mtplx inspect` as verified (detection matches the architectures string)
but hard-failed at load, because mlx_lm.utils.load resolves the model class
from model_type alone: "Model type X not supported".
Fix: before trunk load, when model_type has no mlx-lm module but the
config's own `architectures` names a class in the verified table
(Qwen3_5[Moe]{ForConditionalGeneration,ForCausalLM,TextForCausalLM}),
register a loud sys.modules alias to the implementing module — the same
mechanism transformers uses for class resolution, and the same shim
precedent as qwen3_5_mtp/hy_v3. Unknown model_type + unknown architecture
keeps the fail-loud behavior.
Receipt: renamed-config fire drill (4B Speed clone, model_type
qwen3_8_drill) now loads through the alias, injects the MTP head, and
generates in MTP mode (depth 3) end to end via the public `mtplx run` path
— previously ValueError at load. 5 new unit tests pin the contract
(alias/fail-loud/native-untouched/text_config-nesting/idempotence).
… the parser default Real-QA finding: `mtplx start --dry-run` advertised "profile: sustained / mode: Sustained MTP" for the 27B Optimized-Speed flagship even though the actual launch resolves turbo via _apply_model_default_profile — the 2026-07-16 stale-display bug class on one more surface (pre-existing on v2.4.2, reproduced on the shipped build). Benchmarkers reading the dry-run would pin the slow profile. The payload now uses _resolved_default_profile_name(args, model); explicit --profile flags are respected unchanged. Receipts: dry-run flagship now prints turbo, --profile sustained still prints sustained, test_onboarding + test_public_cli suites green.
…faces 2026-08-03 parity audit findings, all three real drifts fixed: 1. CLI OpenCode lane gains the four MTPLX_VLLM_METAL_PAGED_GQA_SDPA_* keys (long-context decode route) the app and the CLI hermes lane already set. 2. App codingAgentRuntimeEnvironment gains MTPLX_LAZY_TARGET_DISTRIBUTIONS=1 (was CLI-only). 3. CLI `start pi` now composes exactly like the app: the shared coding-agent block (session bank, SDPA route, postcommit wait, frontier flags, tool prompt, template profile) + the Pi history-budget overrides. Previously a CLI Pi user ran a bare engine with none of that, and three history values (96/16/150) diverged from the app-lane numbers (72/8/120) every app Pi user already runs — unified to the app values. 4. The 35B Speed FP16 sibling gets the measured launch defaults (depth 1, target_prefix, draft 0.6/0.95/20) on the CLI exact-id gate too; the app's substring detection already applied them. New tests/test_app_cli_env_parity.py parses MTPLXCommandBuilder.swift (the test_model_catalog.py approach) and asserts key- and value-parity for the shared block and the Pi composition, so this drift class now fails in CI. Receipts: parity tests 2/2, public_cli+onboarding+model_catalog suites green, ruff clean, swift build clean. Deliberate non-changes: hermes block left duplicated (in-sync today); legacy 27B turbo one-sidedness and the vestigial MTPLX_DISABLE_FAST_MLX_AUTODISCOVERY key documented for follow-up.
…ies; repair near-miss tool argument keys (youssofal#196, youssofal#197) Two root causes from the youssofal#196/youssofal#197 reports, both proven by red->green unit tests (no engine, sampler, or wire-protocol surface touched). 1. youssofal#196 hard error "malformed tool_call: unterminated stream" is MTPLX's own stream hidden-tool guard (STREAM_HIDDEN_TOOL_GUARD_TOKENS=2048 / _S=30), not a serializer bug. The guard stands down while the Qwen-XML stream parser is inside a <parameter=> value (tool_argument_in_progress), but the youssofal#170 JSON-dialect function body (<function=write>{"filePath": ..., "content": "..."}) waits in the find_parameter stage, so in_known_tool_parameter stayed False for the entire body. Any write payload >= 2048 hidden tokens streamed over >= 30 s (every multi-KB code/HTML file on slower hardware; the reporter runs the FP16 27B on an M2 Max 32GB) crossed the budget and generation was CANCELLED mid-call with the 422 - the exact reported symptom including the size correlation ("small prose writes succeed; code writes fail") and the exact error string. Fix: in_known_tool_parameter now also covers the JSON-dialect body wait state (known tool + find_parameter stage + object-body buffer), so the guard stands down over argument payload by construction - the same rule the loop-guard incident established (guards must treat tool-call payload spans as legitimate, not threshold-tuned). Unknown-tool bodies stay guarded. 2. youssofal#197 corrupted argument keys (offsets for offset; "offset "; "offset >") pass schema validation whenever the real parameter is optional (OpenCode's read tool declares offset/limit optional), so the argument was silently dropped client-side -> 13+ repeated no-op reads, context blowup. New _repair_tool_argument_keys_for_schema in the normalize path (shared by the streaming parser, the suffixed parser, and the final parser) renames a key only on an unambiguous mapping: not itself a schema property; resolves via trim (whitespace / trailing '>'), letter case, or a single trailing 's' to exactly one schema property; target not already supplied; no two keys collapsing onto one target. Anything ambiguous passes through verbatim (unknown-tool pass-through stays the client's contract). Repair runs before value decoding, so a repaired key also gets its schema-typed value. Receipts: tests/test_tool_call_hidden_guard_and_key_repair.py (8 new tests) mirrors the reporters' exact corruption shapes. Pre-fix behavior reproduced on this tree before the change: tool_argument_in_progress=False mid-JSON-body and {"offsets": 45} emitted verbatim; post-fix True / {"offset": 45}. Suites green: 102 passed (tool/stream translator suites incl. the new file), 359 passed (test_server_openai.py + test_openai_bridge.py). The remaining youssofal#196 layer (rare engine-side early stop mid-reasoning, agent-context-only) stays open pending a captured failing turn via the 2.4.2 request log + MTPLX_REQUEST_CAPTURE_DIR replay; this commit removes the self-inflicted mid-call cancellation lane from that investigation.
…alized MTP head Full-suite gate caught a lineage collision between tonight's vendored-class cherry-pick and the PR youssofal#208 graft lane already on main: the vendored model constructs a native MTPBlock unconditionally, so inject_hy_v3_mtp_support's native branch trusted it and an AR-only checkpoint (no draft tensors on disk) returned mtp_enabled=True with a RANDOM head — silent acceptance collapse instead of the clear AR-only error the graft tests pin. Fix: the native branch now verifies the checkpoint actually carries draft tensors (appended model.layers.{N}.* or model.mtp.* — key-name scan via index json / safetensors headers, no tensor materialization) and raises the same clear AR-only error otherwise. test_quantized_overrides_are_honored is scoped explicitly to the graft lane it pins (drops the constructed native block first), since the vendored class's real flow loads+quantizes via mlx_lm.load_model. Receipts: hy3 backend + graft pair 15/15 green together (previously 2 failed under cross-module shim registration); the 4 ruff E402s in the graft file pre-exist on the shipped tree (importorskip pattern).
Promote the 2026-08-03 integration train as a capability release rather than a patch. Version every canonical package surface at 2.5.0 and add user-facing notes covering multi-layer MTP and architecture-alias readiness, first-class HY3 support, and the coding-agent bridge fixes proven through real OpenCode, Pi, Hermes, and protocol QA. Credit David Tai explicitly for the DeepSeek V4, Laguna S-2.1, and GDN contributions while preserving his original commits and authorship. Keep each performance lane opt-in and document the measured DeepSeek speed, memory envelope, and agent-quality caveat honestly so the release advances capability without changing established defaults. The release gate remains the rollback boundary: separate-install Qwen V2 A/B showed no consistent decode, prefill, or memory regression; the full Python suite and Swift suite are green before packaging.
…es-agent-selection-pr
Owner
|
Review queued for the next cycle. The isolated sidecar ( |
youssofal
requested changes
Aug 17, 2026
youssofal
left a comment
youssofal
left a comment
Owner
There was a problem hiding this comment.
This is titled as an app feature but ships an engine policy change: _session_keep_live_refs_for_request hard-returns False for any Hermes client. 2.8.0's session-cache fix routes oversized snapshots through the live-ref lease that this disables, so merging as is reinstates the ~38k committed-frontier freeze for exactly the client class in #86. CI also never ran on the head. Split the app feature from the engine change and the app half can go in.
Contributor
|
Superseded by #373. The replacement is rebuilt from current Validation: 145 focused Hermes tests, 774 full Swift tests with 1 existing skip, and |
youssofal
force-pushed
the
main
branch
2 times, most recently
from
September 1, 2026 08:07
2382dfd to
8bc4d88
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
hermes -p <profile> serve --isolatedsidecar for the selected profileIsolation and lifecycle guarantees
config.yamland.envbyte-identicallyValidation
swift test --scratch-path /tmp/mtplx-hermes-textfield-tests --filter Hermes: 134 passed, 0 failedswift test --scratch-path /tmp/mtplx-hermes-textfield-tests: 663 executed, 1 skipped, 0 failedswift build --scratch-path /tmp/mtplx-hermes-textfield-build --product MTPLXApp: passed with Xcode-betauv build --wheel --out-dir dist: passedscripts/fresh_venv_smoke.sh: passedgit diff --check upstream/main...HEAD: passedupstream/mainuses Ornith; this PR changes no Python or Python-test filesScope
This branch is rebuilt directly on current
upstream/mainand contains only the Hermes app integration, its Swift tests, and the corresponding design/plan documents. Retrieval, embeddings, reranking, packaged-runtime, and local build-environment commits are deliberately excluded.