exp: update model weights, tools, and README

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+app-release.apk filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1 +1,102 @@
-In this repository we host the compiled MLC Android App, MLC Model w/ weights, and tool scripts for on-device performance test.
+# Quick Test
+
+1. Download the Hugging Face model from [Tinytron/MLC-Tinytron at main](https://huggingface.co/Tinytron/MLC-Tinytron/tree/main).
+
+2. Execute the Python script:
+
+```bash
+python bundle_weight.py --apk-path ./app-release.apk
+```
+
+# Full Reproduction
+
+## Compile mlc-llm
+
+1. Clone the repository: [Tinytron/mlc-llm: mlc-llm modified by tinytron](https://github.com/Tinytron/mlc-llm).
+
+```bash
+git clone --recursive https://github.com/Tinytron/mlc-llm.git
+```
+
+2. Compile mlc_llm:
+
+```bash
+cd mlc-llm/
+mkdir -p build && cd build
+python ../cmake/gen_cmake_config.py
+cmake .. && cmake --build . --parallel $(nproc) && cd ..
+```
+
+3. Install the Python package:
+
+```bash
+export MLC_LLM_SOURCE_DIR=/path-to-mlc-llm
+export PYTHONPATH=$MLC_LLM_SOURCE_DIR/python:$PYTHONPATH
+alias mlc_llm="python -m mlc_llm"
+```
+
+## Build Android Application
+
+1. **Weight Conversion and Config File Generation**
+
+   Modify `build_mlc_android.sh`, changing `MODEL_PATH` to the corresponding Hugging Face model path, and then execute the script. The converted weights and config files will be generated in the `bundle` directory under the current path.
+
+2. **Package and Compile the Model**
+
+   Navigate to `mlc_llm/android/MLCChat`, edit `mlc-package-config.json`, and modify it according to the following example. Change the `model` field to the directory of the weights generated for each model conversion.
+
+```json
+{
+    "device": "android",
+    "model_list": [
+     {
+            "model": "path-to-qwen",
+            "estimated_vram_bytes": 4250586449,
+            "model_id": "Qwen2-7B-Instruct-Tinytron-MLC",
+         "bundle_weight": true
+     },
+     {
+            "model": "path-to-llama",
+            "estimated_vram_bytes": 4250586449,
+            "model_id": "Llama3.1-8B-iInstruct-Tinytron-MLC",
+         "bundle_weight": true
+     },
+     {
+            "model": "path-to-phi2",
+            "estimated_vram_bytes": 4250586449,
+            "model_id": "Phi-2-Tinytron-preview-MLC",
+         "bundle_weight": true
+     },
+     {
+            "model": "path-to-cauchy",
+            "estimated_vram_bytes": 4250586449,
+            "model_id": "Cauchy-3B-preview-MLC",
+         "bundle_weight": true
+     }
+     ]
+}
+```
+
+   Then execute:
+
+```bash
+mlc_llm package
+```
+
+3. **Generate APK**
+
+   Use Android Studio to generate the APK, and then execute the installation script:
+
+```bash
+python bundle_weight.py --apk-path ./app/release/app-release.apk
+```
+# Successful Run Screenshots
+We successfully ran the models phi2 and cauchy on a P70 phone with 12GB of RAM, and all models on an iQOO 12 Pro phone with 16GB of RAM. Below are the screenshots of the successful runs.
+## P70
+![phi_12G.jpg#455px #998px](screenshot/phi_12G.jpg) 
+![cauchy_12G.jpg#455px #998px](screenshot/cauchy_12G.jpg)
+## IQOO 12 Pro
+![qwen_16G.jpg#455px #998px](screenshot/qwen_16G.jpg)
+![llama_16G.jpg#455px #998px](screenshot/llama_16G.jpg)
+![phi_16G.jpg#455px #998px](screenshot/phi_16G.jpg)
+![cauchy_16G.jpg#455px #998px](screenshot/cauchy_16G.jpg)

app-release.apk

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:baf76c7df91436cfa2fe1ddd305f49c8638d8efd7cab769a71846416bd7cc528
+size 118016033

build_mlc_android.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# 定义变量
+MODEL_OUTPUT_PATH="./bundle"
+QUANTIZATION="q0f16"
+
+MODEL_PATH="Llama3.1-8B-Instruct-Tinytron"
+MODEL_NAME="Llama3.1-8B-Instruct-Tinytron-MLC"
+# 转换权重
+mlc_llm convert_weight --device cpu ${MODEL_PATH}/ --quantization ${QUANTIZATION} -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+
+# 生成配置
+mlc_llm gen_config ${MODEL_PATH}  --conv-template llama-3_1 --quantization ${QUANTIZATION} --context-window-size 512 --prefill-chunk-size=16 --max-batch-size=1 -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+
+
+# 切换模型
+MODEL_PATH="Qwen2-7B-Instruct-Tinytron"
+MODEL_NAME="Qwen2-7B-Instruct-Tinytron-MLC"
+# 转换权重
+mlc_llm convert_weight --device cpu ${MODEL_PATH} --quantization ${QUANTIZATION} -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+
+# 生成配置
+mlc_llm gen_config ${MODEL_PATH}  --conv-template chatml --quantization ${QUANTIZATION} --context-window-size 512 --prefill-chunk-size=16 --max-batch-size=1 -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+
+# 切换模型
+MODEL_PATH="Phi-2-Tinytron-preview"
+MODEL_NAME="Phi-2-Tinytron-preview-MLC"
+# 转换权重
+mlc_llm convert_weight --device cpu ${MODEL_PATH} --quantization ${QUANTIZATION} -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+
+# 生成配置
+mlc_llm gen_config ${MODEL_PATH}  --conv-template phi-2 --quantization ${QUANTIZATION} --context-window-size 512 --prefill-chunk-size=16 --max-batch-size=1 -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+
+MODEL_PATH="Cauchy-3B-preview"
+MODEL_NAME="Cauchy-3B-preview-MLC"
+# 转换权重
+mlc_llm convert_weight --model-type cauchy  --device cpu ${MODEL_PATH} --quantization ${QUANTIZATION} -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+
+# 生成配置
+mlc_llm gen_config ${MODEL_PATH} --model-type cauchy --conv-template chatml --quantization ${QUANTIZATION} --context-window-size 512 --prefill-chunk-size=16 --max-batch-size=1 -o ${MODEL_OUTPUT_PATH}/${MODEL_NAME}/
+

{Cauchy-3B-preview-MLC → bundle/Cauchy-3B-preview-MLC}/mlc-chat-config.json

@@ -3,7 +3,7 @@
   "model_type": "cauchy",
   "quantization": "q0f16",
   "model_config": {
-    "vocab_size": 151646,
+    "vocab_size": 151680,
     "hidden_size": 2304,
     "num_hidden_layers": 52,
     "num_attention_heads": 36,
@@ -29,7 +29,7 @@
     "num_experts_per_tok": 0,
     "num_experts": 0
   },
-  "vocab_size": 151646,
+  "vocab_size": 151680,
   "context_window_size": 512,
   "sliding_window_size": -1,
   "prefill_chunk_size": 16,

{Cauchy-3B-preview-MLC → bundle/Cauchy-3B-preview-MLC}/ndarray-cache.json

@@ -1,28 +1,29 @@
 {
     "metadata": {
         "ParamSize": 470,
-        "ParamBytes": 6220393984.0,
-        "BitsPerParam": 14.384120906723469
+        "ParamBytes": 6220550656.0,
+        "BitsPerParam": 16.0
     },
     "records": [
         {
             "dataPath": "params_shard_0.bin",
             "format": "raw-shard",
-            "nbytes": 698784768,
+            "nbytes": 698941440,
             "records": [
                 {
                     "name": "model.embed_tokens.weight",
                     "shape": [
+                        1185,
                         2304,
-                        151646
+                        128
                     ],
                     "dtype": "float16",
                     "format": "f32-to-bf16",
-                    "nbytes": 698784768,
+                    "nbytes": 698941440,
                     "byteOffset": 0
                 }
             ],
-            "md5sum": "c0c4bfb94521a8c77a60e9e08388b491"
+            "md5sum": "10ca727fa119834a674e3b558aa9b47d"
         },
         {
             "dataPath": "params_shard_1.bin",

{Cauchy-3B-preview-MLC → bundle/Cauchy-3B-preview-MLC}/params_shard_0.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2bdda8b016f1e38efc0302c534dd477b3c50350ddc366739dd285f26cfcafbec
-size 698784768
+oid sha256:a274b1587fcf032ba30828071061d0e4bd5f6a3992b03fc186f566b6c91791a1
+size 698941440