File size: 323 Bytes
9d55e9d |
1 2 |
It is a sentencepice tokenizer trained using the Oscar Turkish Corpus from scratch. The tokenizer can be used for bigbird models and can be callled either typing
AutoTokenizer.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer") or BigBirdTokenizerFast.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer"). |