jamescallander/MediPhi-Instruct_w8a8_g128_rk3588.rkllm

MediPhi-Instruct — RKLLM build for RK3588 boards

Author: @jamescallander
Source model: MediPhi-Instruct

Target: Rockchip RK3588 NPU via RKNN-LLM Runtime

This repository hosts a conversion of MediPhi-Instruct for use on Rockchip RK3588 single-board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using the RKNN-LLM toolkit

Conversion details

• RKLLM-Toolkit version: v1.2.1
• NPU driver: v0.9.8
• Python: 3.12
• Quantization: w8a8_g128
• Output: single-file .rkllm artifact
• Tokenizer: not required at runtime (UI handles prompt I/O)

⚠️ Safety disclaimer

🛑 This model is not a substitute for professional advice, diagnosis, or treatment.

• It is intended for research, educational, and experimental purposes only.
• Do not rely on its outputs for decisions related to health, safety, or legal/financial matters.
• Always consult a qualified professional for real-world guidance in these domains.
• Use responsibly and in compliance with the source model’s license and Acceptable Use Policy.

Intended use

• On-device deployment of medical-domain instruction-tuned LLMs for research, educational exploration, and proof-of-concept projects.
• MediPhi-Instruct is tuned for healthcare/clinical contexts, making it useful for experimentation in edge AI scenarios on RK3588 SBCs.

Limitations

• Requires 5.5GB free memory
• Requires ~8–12GB of free system memory depending on board and workload.
• Quantization (w8a8_g128) may slightly impact model fidelity vs. upstream FP16.
• Tested on Radxa Rock 5B+; other devices may require different toolkit/runtime versions.

Quick start (RK3588)

1) Install runtime

• The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from airockchip's github page.
• Download and install the required packages as per the toolkit's instructions.

2) Simple Flask server deployment

The simplest way the deploy the .rkllm converted model is using an example script provided in the toolkit in this directory: rknn-llm/examples/rkllm_server_demo

python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
  --rkllm_model_path <MODEL_PATH>/MediPhi-Instruct_w8a8_g128_rk3588.rkllm \
  --target_platform rk3588

3) Sending a request

A basic format for message request is:

{
    "model":"MediPhi-Instruct",
    "messages":[{
        "role":"user",
        "content":"<YOUR_PROMPT_HERE>"}],
    "stream":false
}

Example request using curl:

curl -s -X POST <SERVER_IP_ADDRESS>:8080/rkllm_chat \
    -H 'Content-Type: application/json' \
    -d '{"model":"MediPhi-Instruct","messages":[{"role":"user","content":"What is the role of protein folding in disease?"}],"stream":false}'

The response is formated in the following way:

{
    "choices":[{
        "finish_reason":"stop",
        "index":0,
        "logprobs":null,
        "message":{
            "content":"<MODEL_REPLY_HERE">,
            "role":"assistant"}}],
        "created":null,
        "id":"rkllm_chat",
        "object":"rkllm_chat",
        "usage":{
            "completion_tokens":null,
            "prompt_tokens":null,
            "total_tokens":null}
}

Example response:

{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":" Protein misfolding and aggregation are central to many diseases, including neurodegenerative disorders like Alzheimer's Disease (AD), Parkinson\u2019s Disease (PD), Huntington\u2019s Disease (HD) etc.\n\nIn these conditions, proteins fail to fold into their correct three-dimensional structures and instead aggregate or misfold in the cellular environment leading to formation of insoluble protein aggregates known as amyloid fibrils which are highly resistant to degradation by lysosomal enzymes.\n\nThese aberrant proteins can disrupt normal physiological processes, interfere with neuronal communication and eventually lead to cell death or apoptosis in the affected tissues/organs thereby causing disease symptoms associated with these conditions like memory loss (in Alzheimer\u2019s), movement difficulties etc.\n\nTherefore understanding protein folding pathways is crucial for developing therapeutic strategies aimed at preventing misfolding or promoting proper refolding of aberrant proteins back into their native conformations thereby restoring normal cellular function and alleviating disease symptoms associated with these conditions.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}

4) UI compatibility

This server exposes an OpenAI-compatible Chat Completions API.

You can connect it to any OpenAI-compatible client or UI (for example: Open WebUI)

• Configure your client with the API base: http://<SERVER_IP_ADDRESS>:8080 and use the endpoint: /rkllm_chat
• Make sure the model field matches the converted model’s name, for example:

{
 "model": "MediPhi-Instruct",
 "messages": [{"role":"user","content":"Hello!"}],
 "stream": false
}

License

This conversion follows the license of the source model: mit license