Skip to content

Add chat template for gemma models#2183

Open
C00kieFact0ry wants to merge 1 commit intoabetlen:mainfrom
C00kieFact0ry:feat/add-gemma-vision-chat-handler
Open

Add chat template for gemma models#2183
C00kieFact0ry wants to merge 1 commit intoabetlen:mainfrom
C00kieFact0ry:feat/add-gemma-vision-chat-handler

Conversation

@C00kieFact0ry
Copy link
Copy Markdown

Problem

Using the standard Llava15ChatHandler with multimodal Gemma models fail.
Gemma models expect <start_of_turn>user / <start_of_turn>model / <end_of_turn> tokens instead. There is no vision-capable handler subclass for Gemma models, even though the library already follows the pattern of subclassing Llava15ChatHandler for other model families (e.g., ObsidianChatHandler, MoondreamChatHandler, Llava16ChatHandler) in llama_chat_format.py.

Although there is a format_gemma for text based Gemma models, this is bypassed by the chat handler

Solution

For multimodal models, we need to change the user/assistant roles and add the proper start and end tokens using a GemmaChatHandler

For multimodal models, we need to change the user/assistant roles and add the proper start and end tokens
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant