Hugging Face's logo

vidula123
/
phi2-API-finetune

Safetensors
English
Model card Files Community
vidula123 commited on
Commit
2d57567
·
verified ·
1 Parent(s): 3a4e9e2

Upload tokenizer

Browse files
Files changed (2) hide show
  1. tokenizer.json +0 -0
  2. tokenizer_config.json +1 -0
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
@@ -317,6 +317,7 @@
317
  "bos_token": "<|endoftext|>",
318
  "clean_up_tokenization_spaces": true,
319
  "eos_token": "<|endoftext|>",
 
320
  "model_max_length": 2048,
321
  "return_token_type_ids": false,
322
  "tokenizer_class": "CodeGenTokenizer",
 
317
  "bos_token": "<|endoftext|>",
318
  "clean_up_tokenization_spaces": true,
319
  "eos_token": "<|endoftext|>",
320
+ "extra_special_tokens": {},
321
  "model_max_length": 2048,
322
  "return_token_type_ids": false,
323
  "tokenizer_class": "CodeGenTokenizer",