JetBrains

/

Mellum-4b-base

like 222

Follow

JetBrains 366

Gracias muy buen desarrollo y expandirlo a la comunidad..

Hi, we've released the tokenizer along with the model, which you can find under the Files and versions page on the Hugging Face Hub. Below is a helpful snippet to directly access the tokenizer.

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('JetBrains/Mellum-4b-base')
>>> tokenizer.encode('def main() -> None:')
[904, 2614, 403, 1295, 2374, 52]
>>> tokenizer.decode([904, 2614, 403, 1295, 2374, 52])
'def main() -> None:'