README.md

---
datasets:
- phonemetransformers/IPA-BabyLM
language:
- en
---

# From Babble to Words: Tokenizers

Tokenizers trained for [From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes](https://arxiv.org/abs/2410.22906).

This repository contains the eight tokenizers trained for the project, covering the combinations of the three transformations:

- Character-based tokenization (`CHAR`) vs. subword tokenization (`BPE`)
- Tokenizer for phonemic data (`PHON`) vs. orthographic data (`TXT`)
- Tokenizer removes whitespace (`SPACELESS`) vs. keeps whitespace
  
To load a tokenizer:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('phonemetransformers/babble-tokenizers', subfolder='BABYLM-TOKENIZER-CHAR-TXT')
```