Spaces:

lucalp
/

blt-entropy-patcher

Running on Zero

App Files Files Community

blt-entropy-patcher / bytelatent /tokenizers

Ctrl+K

Ctrl+K

12 contributors

History: 3 commits

par-meta's picture

Remove byte tokenizer and add config args to switch between byte/patch packing (#68)

aeb95f1 unverified 3 months ago

__init__.py

53 Bytes

Initial commit 5 months ago
abstract_tokenizer.py

601 Bytes

Add vocab and seq len abstract fields (#66) 3 months ago
blt_tokenizer.py

4.77 kB

Add vocab and seq len abstract fields (#66) 3 months ago
build_tokenizer.py

1.88 kB

Remove byte tokenizer and add config args to switch between byte/patch packing (#68) 3 months ago
constants.py

185 Bytes

Initial commit 5 months ago
sentence_piece_tokenizer.py

1.99 kB

Add vocab and seq len abstract fields (#66) 3 months ago
test_blt_tokenizer.py

1.26 kB

Initial commit 5 months ago
tiktoken_tokenizer.py

3.15 kB

Add vocab and seq len abstract fields (#66) 3 months ago