Create README.md

README.md

@@ -0,0 +1,16 @@
+---
+datasets:
+- phonemetransformers/IPA-BabyLM
+language:
+- en
+---
+
+# From Babble to Words: Tokenizers
+
+Tokenizers trained for [From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes](https://arxiv.org/abs/2410.22906).
+
+This repository contains the eight tokenizers trained for the project, covering the combinations of the three transformations:
+
+- Character-based tokenization (`CHAR`) vs. subword tokenization (`BPE`)
+- Tokenizer for phonemic data (`PHON`) vs. orthographic data (`TXT`)
+- Tokenizer removes whitespace (`SPACELESS`) vs. keeps whitespace