vukrosic
/

essential-web-16k-tokenizer

Model card Files Files and versions Community

essential-web-16k-tokenizer

Ctrl+K

Ctrl+K

1 contributor

History: 4 commits

vukrosic's picture

Add essential_web_500k_text.txt - 500K characters of raw text from Essential-Web dataset

eadb605 verified about 1 month ago

.gitattributes

1.52 kB

initial commit about 1 month ago
README.md

4.6 kB

Add tokenizer usage documentation about 1 month ago
bpe_tokenizer_16k_n1000000.pkl
Pickle imports
- No problematic imports detected
What is a pickle import?
194 kB
LFS

Upload bpe_tokenizer_16k_n1000000.pkl about 1 month ago
essential_web_500k_text.txt

504 kB

Add essential_web_500k_text.txt - 500K characters of raw text from Essential-Web dataset about 1 month ago