Slightly modified version of cl100k_base that supports Dolma 1.x special tokens (|||PHONE_NUMBER|||, |||EMAIL_ADDRESS|||, |||IP_ADDRESS|||) as well as adds extra tokens to fill gaps in tiktoken cl100k_base version.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model is not currently available via any of the supported third-party Inference Providers, and HF Inference API was unable to determine this model’s pipeline type.