File size: 1,161 Bytes
00a155f 72c5460 00a155f ebca2d0 00a155f ebca2d0 00a155f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
---
license: mit
---
## **DeepSeek-R1-Distill-Qwen-7B-INT4-ONNX**
<b><span style="text-decoration:underline">Note: This is unoffical version,just for test and dev.</span></b>
### **Sample**
```python
import onnxruntime_genai as og
import numpy as np
import os
model_folder = "Your DeepSeek-R1-Distill-Qwen-7B-INT4-ONNX location"
model = og.Model(model_folder)
tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()
search_options = {}
search_options['max_length'] = 2048
search_options['past_present_share_buffer'] = False
chat_template = "<|im_start|><|user|>{input}<|im_end|>\n<|assistant|>"
text = """ Explain 'The goat grazing problem'"""
prompt = f'{chat_template.format(input=text)}'
input_tokens = tokenizer.encode(prompt)
params = og.GeneratorParams(model)
params.set_search_options(**search_options)
params.input_ids = input_tokens
generator = og.Generator(model, params)
while not generator.is_done():
generator.compute_logits()
generator.generate_next_token()
new_token = generator.get_next_tokens()[0]
print(tokenizer_stream.decode(new_token), end='', flush=True)
``` |