Upload folder using huggingface_hub

README.md

@@ -36,18 +36,36 @@ Trained on bilingual Japanese-English story data with masked loss on Japanese pr
 ## Usage
 
 ```python
-from transformers import LlamaForCausalLM, LlamaTokenizer
+from transformers import LlamaForCausalLM, AutoTokenizer
 
 model = LlamaForCausalLM.from_pretrained("one-way-polyglot-8m-tied")
-tokenizer = LlamaTokenizer.from_pretrained("one-way-polyglot-8m-tied")
+tokenizer = AutoTokenizer.from_pretrained("one-way-polyglot-8m-tied")
 
-# Japanese input, English output
+# Japanese input → English output (primary use case)
 prompt = "昔々、赤い傘を持った少女がいました。"
 inputs = tokenizer(prompt, return_tensors="pt")
 outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.7)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+
+# Mixed-language name transliteration
+prompt = "太郎は公園で花子と遊んでいました。After playing, Taro told Hanako that"
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=30, temperature=0.7)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+
+# English text (works perfectly with case folding)
+prompt = "Hello World"  # Automatically normalized to lowercase
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=30, temperature=0.7)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
 
+### Tokenizer Features
+- **✅ Case Folding**: "Hello", "hello", and "HELLO" produce identical tokenization
+- **✅ Japanese Support**: Full Japanese text support with proper normalization  
+- **✅ No UNK Tokens**: Proper handling of uppercase/lowercase English text
+- **✅ SentencePiece Compatibility**: Built using proper Unigram model with normalization
+
 ## Model Variants
 
 This is part of a series exploring one-way polyglot capabilities:

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "[BOS]",
+  "eos_token": "[EOS]",
+  "unk_token": "[UNK]",
+  "pad_token": "[PAD]"
+}

tokenizer_config.json

@@ -1,12 +1,16 @@
 {
-  "tokenizer_class": "LlamaTokenizer",
+  "tokenizer_class": "PreTrainedTokenizerFast",
   "vocab_size": 16384,
   "model_max_length": 512,
   "pad_token": "[PAD]",
   "bos_token": "[BOS]",
   "eos_token": "[EOS]",
   "unk_token": "[UNK]",
-  "add_bos_token": true,
+  "pad_token_id": 3,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "unk_token_id": 0,
+  "add_bos_token": false,
   "add_eos_token": false,
   "clean_up_tokenization_spaces": false
 }