tokenizer configuration
#9
by
TingchenFu
- opened
When running the demo code, it warns that some special tokens are not set. Not sure do we need to set the special tokens:
Using unk_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using mask_token, but it is not set yet.
Update the transformers version and tokenizer version can solve the issue. closed.
TingchenFu
changed discussion status to
closed