metascroy commited on
Commit
742853f
·
verified ·
1 Parent(s): bbd8708

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -27
README.md CHANGED
@@ -186,35 +186,12 @@ python -m executorch.examples.models.llama.export_llama \
186
  --metadata '{"get_bos_id":199999, "get_eos_ids":[200020,199999]}' \
187
  --output_name="phi4-mini-8dq4w.pte"
188
  ```
189
-
190
- ## Run model with pybindings
191
- ```
192
- export TOKENIZER="/path/to/tokenizer.json"
193
- export TOKENIZER_CONFIG="/path/to/tokenizer_config.json"
194
- export PROMPT="<|system|><|end|><|user|>Hey, are you conscious? Can you talk to me?<|end|><|assistant|>"
195
- python -m executorch.examples.models.llama.runner.native \
196
- --model phi_4_mini \
197
- --pte phi4-mini-8dq4w.pte \
198
- -kv \
199
- --tokenizer ${TOKENIZER} \
200
- --tokenizer_config ${TOKENIZER_CONFIG} \
201
- --prompt "${PROMPT}" \
202
- --params "${PARAMS}" \
203
- --max_len 128 \
204
- --temperature 0
205
- ```
206
-
207
- The output is:
208
-
209
- ```
210
- Hello! I am Phi, an AI developed by Microsoft. I am not conscious in the way humans are, but I am here to help and converse with you. How can I assist you today?Hello! I am Phi, an AI developed by Microsoft. I am not conscious in the way humans are, but I am here to help and converse with you. How can I assist you today?Hello! I am Phi, an AI developed by Microsoft. I am not conscious in the way humans are, but I am here to
211
- ```
212
-
213
- Note: the runner does not currently recongize the stop token from Phi 4 Mini, so it generates text beyond when it should stop.
214
 
215
  ## Running in a mobile app
216
- The model can be run in a mobile app. See [instructions](https://pytorch.org/executorch/main/llm/llama-demo-ios.html) for doing this on iOS.
217
- TODO: add perf numbers, memory numbers, and screenshot from app
 
 
218
 
219
  # Disclaimer
220
  PyTorch has not performed safety evaluations or red teamed the quantized models. Performance characteristics, outputs, and behaviors may differ from the original models. Users are solely responsible for selecting appropriate use cases, evaluating and mitigating for accuracy, safety, and fairness, ensuring security, and complying with all applicable laws and regulations.
 
186
  --metadata '{"get_bos_id":199999, "get_eos_ids":[200020,199999]}' \
187
  --output_name="phi4-mini-8dq4w.pte"
188
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
189
 
190
  ## Running in a mobile app
191
+ The model can be run in a mobile app. See [instructions](https://pytorch.org/executorch/main/llm/llama-demo-ios.html) for doing this in iOS.
192
+ On iPhone 15 Pro, the model runs at 17.3 tokens/sec and uses 3206 Mb of memory.
193
+
194
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/66049fc71116cebd1d3bdcf4/hn3HzR-DF-OUweIEE3HzC.png)
195
 
196
  # Disclaimer
197
  PyTorch has not performed safety evaluations or red teamed the quantized models. Performance characteristics, outputs, and behaviors may differ from the original models. Users are solely responsible for selecting appropriate use cases, evaluating and mitigating for accuracy, safety, and fairness, ensuring security, and complying with all applicable laws and regulations.