Qwen-Coder结束符是应该使用<endoftext>还是<im_end>? #1104
Unanswered
m-maoyanyu
asked this question in
Q&A
Replies: 1 comment
|
base 用 <|endoftext|>;instruct 用 <|im_end|> |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
请问,Qwen-Coder的config.json与generation.json中eos_token_id均为151643:"<|endoftext|>",但加载Qwen-Coder的tokenizer后,eos_token_id为151645:"<|endoftext|>" :"<|im_end|>"。直接使用Qwen-Coder推理时,也是输出"<|im_end|>"结尾。请问要以哪个作为结束符更好。


All reactions