Hugging Face's logo

yuchenxie
/
arlowgpt-tokenizer-v2

Transformers
English
Inference Endpoints
Model card Files Community
arlowgpt-tokenizer-v2 / README.md
yuchenxie's picture
yuchenxie
Update README.md
fc0e3e6 verified
preview code
|
raw
history blame contribute delete
920 Bytes
metadata
license: apache-2.0
datasets:
  - HuggingFaceFW/fineweb
language:
  - en
base_model:
  - yuchenxie/ArlowGPT-Tokenizer
library_name: transformers

ArlowGPT Tokenizer

This repository contains a custom-trained BPE tokenizer for ArlowGPT, created by Yuchen Xie.

Tokenizer Details

  • Type: BPE (Byte-Pair Encoding)
  • Vocabulary Size: 131,072 tokens
  • Special Tokens:
    • Start of Text: <|startoftext|>
    • End of Text: <|endoftext|>
    • Padding: <|pad|>
    • Unknown: <|unk|>
    • Mask: <|mask|>
    • Message Start: <|im_start|>
    • Message End: <|im_end|>

Usage 

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("yuchenxie/arlowgpt-tokenizer-v2")

Training Details

This tokenizer was trained on the 10B GPT-2 randomly shuffled tokens under a custom script composed by Yuchen Xie. This tokenizer is compatible with HuggingFace Transformer's Auto Tokenizer class.