Skip to content

Commit ea13a50

Browse files
authored
[Bug] Fix chat_completion() for Llama3 (#1353)
The LLama3 chat template also needs the BOS token, so add that in when rendering the template.
1 parent 4dcbc70 commit ea13a50

1 file changed

Lines changed: 7 additions & 1 deletion

File tree

guidance/models/_engine/_engine.py

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -458,11 +458,17 @@ def chat_completion_streaming(
458458
with a Lark grammar.
459459
It is very much experimental in nature, and the API is subject to change.
460460
"""
461+
# Get the tokens which might be needed by the chat template
462+
tokens = {
463+
"eos_token": self.tokenizer.eos_token.decode("utf-8"),
464+
"bos_token": self.tokenizer.bos_token.decode("utf-8")
465+
}
466+
461467
# Render the messages
462468
chat_template = self.get_chat_template().template_str
463469
rtemplate = Environment(loader=BaseLoader).from_string(chat_template)
464470
rendered_prompt = rtemplate.render(
465-
messages=messages, eos_token=self.tokenizer.eos_token.decode("utf-8"), tools=tools
471+
messages=messages, tools=tools, **tokens
466472
)
467473
rendered_prompt += self.get_chat_template().get_role_start("assistant")
468474

0 commit comments

Comments
 (0)