Ziggazr/Qwen2.5-7B-Instruct-1M-W8A8-rkllm

Model Card for Model ID

Model Details

Model Description

• Developed by: JiangTao
• Model type: rknn
• License: MIT

Uses

Direct Use

uv venv --python=3.12
source .venv/bin/activate
uv pip install flask Werkzeug
git clone https://github.com/airockchip/rknn-llm
cd rknn-llm/examples/rkllm_server_demo
python3 flask_server.py --rkllm_model_path /path/to/model/Qwen2.5-7B-Instruct-1M_W8A8_RK3588.rkllm --target_platform rk3588

Export Pipeline

# init rkllm export environment
cd ~/autodl-tmp 
git clone https://github.com/airockchip/rknn-llm.git
conda init
conda create -n rkllm python=3.10
conda activate rkllm
cd rknn-llm
pip install rkllm-toolkit/rkllm_toolkit-1.1.4-cp310-cp310-linux_x86_64.whl

# processing
cd examples/DeepSeek-R1-Distill-Qwen-1.5B_Demo/export
# /root/autodl-tmp/models/Qwen2.5-7B-Instruct-1M
python generate_data_quant.py -m /root/autodl-tmp/models/Qwen2.5-7B-Instruct-1M
# modify `modelpath` in `export_rkllm.py`
python export_rkllm.py

Out-of-Scope Use

• Not supported for rk3576, rk3562

Environmental Impact

• Cloud Provider: AutoDL

Technical Specifications [optional]

Software

https://github.com/airockchip/rknn-llm

Model Card Contact

[email protected]