From 3270c6f25828e5c596641d1b8f50d5b261ccec88 Mon Sep 17 00:00:00 2001 From: pretyflaco Date: Sat, 15 Aug 2026 13:53:08 +0300 Subject: [PATCH] docs: add Qwen3.8-27B local-model eval (EN/TR/DE); reconcile recommendations Records the Qwen3.8-27B evaluation run through millet's real two-pass code path (not a harness): matches/exceeds the cloud baseline on topic/action/ question coverage across English, Turkish, and German meetings, 5/5 format, localized non-English headers, no hallucinated speakers, ~55-100s/meeting. Requires the v0.15.1 Pass-1 output-reserve fix or it silently truncates. Supersedes the earlier 'gpt-oss:20b is best local' conclusion; reconciles the stale default-model and 'Sonnet is the only model' claims. Metrics only - no transcript content. --- docs/local-model-evaluation.md | 138 +++++++++++++++++++++++++++++++-- 1 file changed, 131 insertions(+), 7 deletions(-) diff --git a/docs/local-model-evaluation.md b/docs/local-model-evaluation.md index 0117cf3..3b7fe01 100644 --- a/docs/local-model-evaluation.md +++ b/docs/local-model-evaluation.md @@ -1,13 +1,23 @@ # MeetScribe Local Model Improvement: Double-Pass Approach & Prompt Engineering Created: 2026-04-24 +Last updated: 2026-08-15 — added Qwen3.8-27B evaluation (EN/TR/DE, via millet's real two-pass code path); reconciled recommendations. Context: Research session evaluating local models as potential replacements for Sonnet 4.6 in meetscribe's meeting transcript summarization pipeline. +> **TL;DR (2026-08-15):** With the two-pass extraction-truncation fix in +> v0.15.1, **`qwen3.8:27b` is the strongest local model** for this task — +> it matches or exceeds the Sonnet baseline on topic/action/question +> coverage across English, Turkish, and German meetings, with 5/5 format +> compliance and localized section headers, at ~55–100 s/meeting on an +> RTX 3090. It supersedes the earlier "gpt-oss:20b is the best local" +> conclusion below. Caveats: ~2× slower than Sonnet; a mild tendency to +> over-count "decisions"; validated on 3 EN + 1 TR + 1 DE meetings. + --- ## Background -MeetScribe uses an LLM to summarize meeting transcripts into a structured 5-section Markdown format (Overview, Topics, Actions, Decisions, Open Questions). The current production backend is **Sonnet 4.6** via claudemax proxy, with **Ollama** as fallback. The Ollama default was `qwen3.5:9b` (now changed to `gpt-oss:20b`). +MeetScribe uses an LLM to summarize meeting transcripts into a structured 5-section Markdown format (Overview, Topics, Actions, Decisions, Open Questions). The current production backend is **Sonnet 4.6** via claudemax proxy, with **Ollama** as fallback. The hardcoded Ollama default is `qwen3.5:9b`; per-machine model choice is set via the `MILLET_SUMMARY_MODEL` env var (e.g. `qwen3.8:27b` on GPUs with ≥18 GB VRAM). The default is deliberately conservative so it runs on small GPUs — the hardware, not the codebase, dictates which model to prefer. The goal is to determine whether local models can match Sonnet 4.6 quality for fully offline summarization. @@ -18,7 +28,7 @@ The goal is to determine whether local models can match Sonnet 4.6 quality for f - **User prompt (non-English):** `meet/prompts/summarize_user_lang.md` - **Summarization code:** `meet/summarize.py` (4 backends: claudemax, openrouter, ollama, openai) - **Language headers:** `meet/languages.py` (section header translations for DE, FR, ES, TR, FA) -- **Default Ollama model:** `gpt-oss:20b` (changed from `qwen3.5:9b` during this session, line 33 of `summarize.py`) +- **Default Ollama model:** `qwen3.5:9b` (hardcoded floor; override per-machine with `MILLET_SUMMARY_MODEL`) ### Typical meeting characteristics @@ -47,6 +57,7 @@ The goal is to determine whether local models can match Sonnet 4.6 quality for f | Model | Ollama ID | Size | Context | Architecture | Speed (tg128) | |-------|-----------|------|---------|--------------|---------------| | Sonnet 4.6 | N/A (cloud) | N/A | 200K | Anthropic | N/A | +| **Qwen3.8 27B** | `qwen3.8:27b` | 18 GB | 262K | Qwen3.5/DeltaNet hybrid + vision | 41.9 t/s | | GPT-OSS 20B | `gpt-oss:20b` | 13 GB | 128K | OpenAI MoE, MXFP4 | ~50 t/s | | Qwen3.6 27B | `qwen3.6:27b` | 17 GB | 262K | Qwen3.5/DeltaNet hybrid | 36 t/s | | Qwen3.5 27B | `qwen3.5:27b` | 17 GB | 262K | Qwen3.5 | ~8 t/s (partial CPU) | @@ -367,14 +378,24 @@ Organize the following extracted meeting data into the required format: | Sonnet 4.6 single-pass | 5/5 | 11 topics, 13 actions, 6 decisions, 8 questions | 0 | 64s | Gold standard | | gpt-oss single-pass (original prompt) | 4/5 | 5 topics, 4 actions | 3 | 65s | Usable but poor | | gpt-oss single-pass (improved prompt) | 0/5 | 6 topics | 2 | 24s | Regression | -| **gpt-oss double-pass** | **5/5** | **5 topics, 6 actions, 4 decisions** | **1** | **94s** | **Best local** | +| **gpt-oss double-pass** | **5/5** | **5 topics, 6 actions, 4 decisions** | **1** | **94s** | **Best local (Apr 2026)** | | qwen3.6 single-pass | 0/5 | 0 (refused) | N/A | 147s | Unusable | | **qwen3.6 double-pass** | **5/5** | **5 topics, 5 actions, 3 decisions** | **2** | **353s** | **Good but slow** | | qwen3.5 single-pass | 0/5 | 1 topic (tunneled) | 0 | 98s | Unusable | | gemma4 single-pass | 0/5 | 3 topics (context-limited) | 0 | 23s | Context-limited | +> **Note (2026-08-15):** Meeting D predates Qwen3.8 and wasn't re-run on it. +> On a comparable set (3 EN + 1 TR + 1 DE), **Qwen3.8-27B double-pass reached +> full cloud-baseline coverage** — a clear step above gpt-oss's ~50% here. +> See "Qwen3.8-27B Evaluation (2026-08-15)" below for the current best local. + ### Remaining quality gap vs Sonnet 4.6 (even with double-pass) +> These gaps describe the **20B-class models (gpt-oss, qwen3.6)** as of +> April 2026. **Qwen3.8-27B (2026-08-15) closes most of them** — see its +> section below: it reaches baseline coverage and produces non-empty +> open-questions sections. The items below still hold for the older models. + 1. **Content coverage (~50%):** Local models extract ~5 topics where Sonnet finds 11. This is a fundamental attention/capability limitation of 20B-class models processing 20K tokens. 2. **Open questions (0 vs 8):** No local model identified any open questions. This requires understanding the difference between "agreed upon" and "discussed but unresolved" — a nuanced distinction that smaller models miss. 3. **Action item specificity:** Sonnet extracts 13 specific action items with exact owners and context. Local models find 4-6, often vague. @@ -384,7 +405,12 @@ Organize the following extracted meeting data into the required format: ## Implementation Plan -### Phase 1: Implement double-pass in summarize.py (recommended) +### Phase 1: Implement double-pass in summarize.py ✅ DONE + +**Shipped.** `_summarize_ollama_twopass()` is implemented and is the default +for the ollama backend (opt out with `MILLET_OLLAMA_SINGLEPASS=1`). The +v0.15.1 fix widened Pass-1's output reserve so thinking-heavy models +(qwen3.8) don't truncate mid-extraction. The original design notes follow. Add a `_summarize_ollama_twopass()` function to `summarize.py` that: @@ -422,13 +448,111 @@ The content coverage gap (~50%) is a model capability issue. Options to close it --- +## Qwen3.8-27B Evaluation (2026-08-15) + +Qwen3.8-27B was released 2026-08-14. Same hybrid DeltaNet architecture as +Qwen3.6 (`qwen3_5`, 64 layers) plus native vision. This evaluation was run +**through millet's real two-pass code path** (`summarize()` → +`_summarize_ollama_twopass()`), not an ad-hoc harness — the numbers reflect +what production actually produces. + +### Requirements & setup + +- **Ollama ≥ 0.32.12** — older versions cannot run the architecture + (`qwen3next: layer 64 missing attn projections`). +- Official model: `ollama pull qwen3.8:27b` (18 GB, 256K context, vision + + tools + thinking). Do **not** hand-register the Unsloth GGUF — Ollama's + engine can't run it; only the official pull works. +- Runs fully on a 24 GB GPU (17 GB resident). **Will not fit a 12 GB GPU** + (e.g. RTX 5070) — pick a smaller model there via `MILLET_SUMMARY_MODEL`. + +### Prerequisite fix: Pass-1 truncation (v0.15.1) + +The first run through millet **under-performed a throwaway harness** by ~2×. +Root cause: `_dynamic_num_ctx` reserved only 4096 output tokens, so +Qwen3.8's long exhaustive Pass-1 extraction hit the context window and was +silently cut off (`done_reason: length`). Raising the Pass-1 reserve to +16384 (v0.15.1) fixed it — extraction on a 44 KB meeting grew ~1.8 KB → +~5.6 KB, and `done_reason` flipped `length` → `stop`. **All results below +require v0.15.1 or later.** `think: False` is already set by the two-pass +flow and prevents the separate "empty content" failure mode. + +### Throughput (llama-bench, pp512/tg128, RTX 3090) + +| Model | Prompt (pp512) | Generation (tg128) | +|-------|:-:|:-:| +| Qwen3.8-27B Q4_K_M | 1405 t/s | 41.9 t/s | +| Qwen3.6-27B Q4_K_M | 1421 t/s | 41.3 t/s | +| SuperGemma4-26B Q4_K_M | 4646 t/s | 161.8 t/s | + +Marginally faster than Qwen3.6 on generation; still ~4× slower than +SuperGemma4 (which is unusable here for other reasons — format + context). + +### Double-pass results vs cloud baseline (via millet, v0.15.1) + +Counts are section item counts (**T**opics / **A**ctions / **D**ecisions / +open **Q**uestions). Baselines are the existing production summaries +(claudemax / Sonnet-class). Meetings are anonymized; only metrics are shown. + +| Meeting | Lang | ~Tokens | Qwen3.8 (T/A/D/Q) | Baseline (T/A/D) | Format | Time | +|---------|:-:|:-:|:-:|:-:|:-:|:-:| +| E | EN | 11K | 20 / 8 / 4 / 7 | 11 / 6 / 3 | 5/5 | 84 s | +| F | EN | 8K | 10 / 24 / 5 / 10 | 18 / 20 / 4 | 5/5 | 102 s | +| G | EN | 11K | 20 / 16 / 6 / 10 | 10 / 12 / 4 | 5/5 | 102 s | +| H | **TR** | 1.5K | 10 / 1 / 3 / 7 | 6 / 1 / 0 | 5/5 | 53 s | +| I | **DE** | 17K | 15 / 4 / 3 / 8 | 15 / 3 / 0 | 5/5 | 93 s | + +**Findings:** + +- **Coverage matches or exceeds the baseline** on every meeting. The model + is more granular (often more topics/questions than the baseline) without + dropping content. +- **Format compliance 5/5** everywhere, including a genuine open-questions + section — historically the section all other local models left empty. +- **Non-English works.** Turkish and German summaries are written *in the + target language* with **localized section headers** (`Toplantı Özeti` / + `Besprechungsübersicht`, etc.) via `languages.py`. No language leakage. +- **No hallucinated speakers.** Names in the TR/DE outputs trace to the + transcript (proper nouns that look unusual — e.g. a vendor founder's + name — were genuinely present). +- **Known weakness — decisions over-inclusion.** Qwen3.8 tends to list + more "decisions" than the baseline (e.g. 4 vs 3), occasionally promoting + a strong suggestion or an action to a decision. Not fabrication, but + looser than Sonnet's stricter "explicitly agreed" bar. +- **Speed:** ~55–100 s/meeting (pass1 + pass2), roughly 2× the cloud + latency but fine for a batch/offline job. + +### Verdict + +**Qwen3.8-27B (double-pass, v0.15.1+) is the best local model for meetscribe +summarization to date** — the first to reach cloud-baseline coverage across +English *and* non-English (TR/DE) meetings with correct format and no +hallucinated speakers. It supersedes gpt-oss:20b as the recommended local +model. Sonnet remains more concise/polished and stricter on decisions, and +is ~2× faster, so it stays the production default; but for a fully offline +deployment on a ≥18 GB GPU, Qwen3.8 is now a credible replacement. + +Remaining gaps to close before a stronger claim: larger non-English sample +(1 TR + 1 DE so far), and tightening the decisions-vs-suggestions boundary +(prompt-level). + +--- + ## Appendix: Model-Specific Notes -### gpt-oss:20b (recommended local model for meetscribe) +### qwen3.8:27b (recommended local model, 2026-08-15) +- 18 GB, fits a 24 GB GPU; needs Ollama ≥ 0.32.12; vision + tools + thinking. +- Best local results to date: matches/exceeds cloud baseline coverage on + EN/TR/DE via the two-pass flow; 5/5 format; localized non-English headers. +- **Requires the v0.15.1 Pass-1 reserve fix** or it silently truncates. +- Weaknesses: ~2× Sonnet latency; mild over-inclusion of "decisions". +- Won't fit a 12 GB GPU — choose a smaller model there. + +### gpt-oss:20b (prior best local; now a fallback) - 13 GB, fits comfortably in 24 GB VRAM - 128K context — handles all meeting sizes without truncation - MoE architecture with MXFP4 native quantization (OpenAI) -- Best local model for this task: follows format (especially in double-pass), reasonable speed +- Was the best local model here until Qwen3.8 (2026-08-15); follows format well in double-pass, reasonable speed. Now a fallback / small-GPU option (fits 13 GB where Qwen3.8's 18 GB does not). - Persistent weakness: hallucinated proper nouns (entity names, dates) - Ollama ID: `gpt-oss:20b` (9M pulls, well-tested) @@ -457,4 +581,4 @@ The content coverage gap (~50%) is a model capability issue. Options to close it - Comprehensive topic coverage (11+ topics on complex meetings) - No hallucinations observed - 40-90s per meeting via claudemax proxy -- Remains the only model that reliably produces publication-ready summaries +- Most concise/polished output and the strictest "explicitly agreed" bar for decisions; ~2× faster than local Qwen3.8. Remains the production default. As of 2026-08-15, **Qwen3.8-27B (double-pass, v0.15.1+) matches it on coverage** and is a credible fully-offline replacement on a ≥18 GB GPU (see the Qwen3.8 section above).