Aranizer | Arabic Tokenization with SentencePiece & PBE - a riotu-lab Collection

riotu-lab 's Collections

SAND: Large-Scale Synthetic Arabic OCR Dataset

Aranizer | Arabic Tokenization with SentencePiece & PBE

ArabianLLM Series | Native Arabic Large Language Models

Aranizer | Arabic Tokenization with SentencePiece & PBE

updated Aug 25, 2024

Collection of Arabic Tokenizers with different sizes based on SentencePiece & PBE Encodings suitable for training LLMs