Models

dotan1111/BioTokenizer-BFD-UNI-400

Updated Sep 13, 2023

dotan1111/BioTokenizer-BFD-UNI-800

Updated Sep 13, 2023

dotan1111/BioTokenizer-BFD-UNI-1600

Updated Sep 13, 2023

dotan1111/BioTokenizer-BFD-UNI-3200

Updated Sep 13, 2023

dotan1111/BioTokenizer-BFD-UNI-6400

Updated Sep 13, 2023

dotan1111/BioTokenizer-BFD-UNI-12800

Updated Sep 13, 2023

dotan1111/BioTokenizer-BFD-UNI-25600

Updated Sep 13, 2023

dotan1111/BioTokenizer-BFD-UNI-51200

Updated Sep 13, 2023

kardosdrur/en_20-newsgroups_wordlevel_glove-50

Updated Oct 2, 2023

baebee/nuph-tokenizer

Updated Oct 11, 2023

RaymondLi/gpt-4-tokenizer

Updated Oct 11, 2023 • 1

dustalov/wikitext-wordlevel

Updated Jan 22, 2024 • 1

Xenova/claude-tokenizer

Updated Apr 30, 2024 • 17

Xenova/c4ai-command-r-v01-tokenizer

Updated Mar 22, 2024 • 3

Xenova/grok-1-tokenizer

Updated Apr 30, 2024 • 17

Xenova/dbrx-instruct-tokenizer

Updated Mar 28, 2024 • 5

LnL-AI/dbrx-instruct-tokenizer

Updated Apr 3, 2024

LnL-AI/dbrx-base-tokenizer

Updated Apr 4, 2024

nsarrazin/c4ai-command-r-v01-tokenizer

Updated Jul 16, 2024

Xenova/gpt-4o

Updated May 13, 2024 • 63

Saghen/c4ai-command-r-v01-tokenizer

Updated May 22, 2024

Xenova/Nemotron-4-340B-Instruct-Tokenizer

Updated Jun 15, 2024 • 6

mekaneeky/jw1

Text Generation • 7B • Updated Jul 23, 2024 • 8

IsmaelMousa/arabic-bpe-tokenizer

Summarization • Updated Sep 14, 2024 • 1

Sakalti/kanide.ai

Updated Sep 24, 2024 • 6

alea-institute/kl3m-001-32k

Updated Sep 29, 2024

alea-institute/kl3m-003-64k

Updated Oct 9, 2024

deepnet/SN29-C00-llama-HK13-1

Text Generation • 7B • Updated Nov 5, 2024 • 11

deepnet/SN29-C00-llama-HK14-1

Text Generation • 7B • Updated Oct 30, 2024 • 24

deepnet/SN9-C4-llama-HK5-1

Text Generation • 14B • Updated Oct 28, 2024 • 102