Skip to content

server: improve user message detection and create checkpoints at every user message#24176

Draft
aldehir wants to merge 7 commits into
ggml-org:masterfrom
aldehir:improve-messages-spans
Draft

server: improve user message detection and create checkpoints at every user message#24176
aldehir wants to merge 7 commits into
ggml-org:masterfrom
aldehir:improve-messages-spans

Conversation

@aldehir
Copy link
Copy Markdown
Contributor

@aldehir aldehir commented Jun 5, 2026

Overview

Scan for user message boundaries directly on the tokens and avoid doing a second pass that translates from byte offsets.

Create checkpoints at the start of every user message, as opposed to only the last message.

cc. @ggerganov @jacekpoplawski

Additional information

The checkpoint logic makes sense to me, but I haven't thoroughly tested it for edge cases.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: Yes - used to update the chat params for every model. Also used to review the checkpoint changes.

@aldehir
Copy link
Copy Markdown
Contributor Author

aldehir commented Jun 5, 2026

Looks like rebasing onto master broke something...

Nevermind, looks like an unrelated regression on Windows, will investigate later.

@ggerganov ggerganov self-assigned this Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples server testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants