bupalinyu commited on
Commit
a1a7d92
·
verified ·
1 Parent(s): 139d3f2

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +18 -17
README.md CHANGED
@@ -33,6 +33,24 @@ pipeline_tag: text-generation
33
  **Edge0-35b-a3b** — a 35B MoE LLM that runs at viable speed on portable devices in under **2.9 GiB of active memory** (1/8 of its 23 GB weight footprint),
34
  via the [edge0](https://github.com/Edge0-AI/edge0) streaming inference framework.
35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  Three mechanisms make this work:
37
 
38
  - **SSD expert offload**: expert weights are streamed from storage on
@@ -48,10 +66,6 @@ Three mechanisms make this work:
48
  quantization loss at 4-bit (see Quality below). Adapters stay
49
  unmerged: one read-only base serves multiple adapter sets.
50
 
51
- > **Preview status:** this is an early preview release of the edge0
52
- > pipeline. The checkpoint ships as int4 quantization plus LoRA and
53
- > prerouter adapters trained for this framework.
54
-
55
  ## Model summary
56
 
57
  | | |
@@ -96,19 +110,6 @@ Measured with `examples/bench.py` on a Mac mini M4 Pro, 24 GB:
96
  *Short contexts; long contexts add KV cache. Expert weights stream from
97
  SSD on demand and are not resident.
98
 
99
- ## Highlights
100
-
101
- - **Runs in phone-class memory**: the full 4-bit checkpoint stays on
102
- storage and experts are streamed on demand, so only the active
103
- weights are in RAM — under **2.9 GiB**, with no sharding and no
104
- upfront download of the weights into memory.
105
- - **Fast enough for interactive use**: 14.9–17.7 tok/s decode on a
106
- Mac mini M4 Pro; long prompts fill in at 113–140 tok/s.
107
- - **Quality kept after quantization**: Recover-LoRA distillation keeps
108
- the int4 model within **3.9 points** of its fp16 base.
109
- - **Works out of the box**: base, LoRA and prerouter adapters ship
110
- together and load automatically via `edge0`.
111
-
112
  ## Use cases
113
 
114
  - Edge / on-device inference where GPU VRAM is scarce and storage is
 
33
  **Edge0-35b-a3b** — a 35B MoE LLM that runs at viable speed on portable devices in under **2.9 GiB of active memory** (1/8 of its 23 GB weight footprint),
34
  via the [edge0](https://github.com/Edge0-AI/edge0) streaming inference framework.
35
 
36
+ > **Preview status:** this is an early preview release of the edge0
37
+ > pipeline. The checkpoint ships as int4 quantization plus LoRA and
38
+ > prerouter adapters trained for this framework.
39
+
40
+ ## Highlights
41
+
42
+ - **Runs in phone-class memory**: the full 4-bit checkpoint stays on
43
+ storage and experts are streamed on demand, so only the active
44
+ weights are in RAM — under **2.9 GiB**, with no sharding and no
45
+ upfront download of the weights into memory.
46
+ - **Fast enough for interactive use**: 14.9–17.7 tok/s decode on a
47
+ Mac mini M4 Pro; long prompts fill in at 113–140 tok/s.
48
+ - **Quality kept after quantization**: Recover-LoRA distillation keeps
49
+ the int4 model within **3.9 points** of its fp16 base.
50
+ - **Works out of the box**: base, LoRA and prerouter adapters ship
51
+ together and load automatically via `edge0`.
52
+
53
+
54
  Three mechanisms make this work:
55
 
56
  - **SSD expert offload**: expert weights are streamed from storage on
 
66
  quantization loss at 4-bit (see Quality below). Adapters stay
67
  unmerged: one read-only base serves multiple adapter sets.
68
 
 
 
 
 
69
  ## Model summary
70
 
71
  | | |
 
110
  *Short contexts; long contexts add KV cache. Expert weights stream from
111
  SSD on demand and are not resident.
112
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
  ## Use cases
114
 
115
  - Edge / on-device inference where GPU VRAM is scarce and storage is