Add support for GPT-OSS Harmony-style chat templates - #24
Open
AndrewFasano wants to merge 1 commit into
Open
Conversation
AndrewFasano
force-pushed
the
feat/gpt_oss_thinking
branch
from
December 5, 2025 07:22
bd31b7b to
a337fa1
Compare
Harmony models use a two-channel response format where the model generates both analysis (thinking) and final answer sections using <|channel|> tags. This enables the model to show its reasoning process alongside the answer. When reconstructing multi-turn conversations, we now: - Detect Harmony templates by checking for <|channel|> and 'thinking' markers - Parse saved responses to extract analysis and final sections separately - Build chat messages with a 'thinking' field containing the analysis - This ensures the tokenizer's chat template receives properly structured data The implementation adds two helper functions: - _tokenizer_uses_harmony(): Detects Harmony-style chat templates - _split_harmony_message(): Parses responses into analysis and final sections During streaming, the full response (including all tags) is displayed to the user and saved automatically by the llm framework. The parsing only happens when reconstructing conversation history for multi-turn interactions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
AndrewFasano
force-pushed
the
feat/gpt_oss_thinking
branch
from
December 5, 2025 07:22
a337fa1 to
da813b0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Harmony-based models (such as GPT-OSS variants) use a structured two-channel chat format: an internal analysis segment and the user-visible final answer, each wrapped in
<|channel|>tags. When the Hugging Face Harmony chat template processes history, it requires:contentthinkingfield<|channel|>tags inside either fieldBefore this PR,
llm-mlxstored the entire raw model output (including both channels and tags) as the assistant message. On subsequent turns, when the history was passed back toapply_chat_template(), GPT-OSS models would raise:This PR detects Harmony chat templates and:
contentthinking(not shown to the user)<|channel|>markers from historyThis enables correct multi-turn chat with GPT-OSS / Harmony models using
llm chat.A similar upstream change was proposed in mlx-lm (see ml-explore/mlx-lm#365), but maintainers stated that the application layer is responsible for providing well-formed chat messages — so I think that means this behavior should be implemented here instead.