Wrong tokenizer config?
#6
by
avoroshilov
- opened
Hello!
It seems like there is a mismatch between the tokenizer chat template, which output <end_of_turn>
token at the end of the turn, and tokenizer config which states "eos_token": "<eos>"
. This causes tokenizer to not properly skip the EOS token. Probably should be set to "eos_token": "<end_of_turn>"
.
This applies to both 27B and 1B models that I glanced over.