Wrong tokenizer config?

#6
by avoroshilov - opened

Hello!

It seems like there is a mismatch between the tokenizer chat template, which output <end_of_turn> token at the end of the turn, and tokenizer config which states "eos_token": "<eos>". This causes tokenizer to not properly skip the EOS token. Probably should be set to "eos_token": "<end_of_turn>".

This applies to both 27B and 1B models that I glanced over.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment