neuralmagic
/

granite-3.1-2b-instruct-quantized.w4a16

@@ -66,7 +66,9 @@ vLLM also supports OpenAI-compatible serving. See the [documentation](https://do
 This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.
 ```bash
 python quantize.py --model_path ibm-granite/granite-3.1-2b-instruct --quant_path "output_dir/granite-3.1-2b-instruct-quantized.w4a16" --calib_size 1024 --dampening_frac 0.01 --observer mse --group_size 64
 ```
@@ -145,6 +147,7 @@ oneshot(
 model.save_pretrained(quant_path, save_compressed=True)
 tokenizer.save_pretrained(quant_path)
 ```
 ## Evaluation
@@ -203,37 +206,114 @@ evalplus.evaluate \
 ### Accuracy
-#### OpenLLM Leaderboard V1 evaluation scores
-| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 |
-|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| ARC-Challenge (Acc-Norm, 25-shot)       | 55.63                             | 54.18                                      |
-| GSM8K (Strict-Match, 5-shot)            | 60.96                             | 62.85                                       |
-| HellaSwag (Acc-Norm, 10-shot)           | 75.21                             | 73.36                                       |
-| MMLU (Acc, 5-shot)                      | 54.38                             | 52.17                                        |
-| TruthfulQA (MC2, 0-shot)                | 55.93                             | 56.83                                        |
-| Winogrande (Acc, 5-shot)                | 69.67                             | 69.85                                        |
-| **Average Score**                       | **61.98**                         | **61.54**                                   |
-| **Recovery**                            | **100.00**                        | **99.29**                                   |
-#### OpenLLM Leaderboard V2 evaluation scores
-| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 |
-|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| IFEval (Inst Level Strict Acc, 0-shot)| 67.99                           | 67.63                                          |
-| BBH (Acc-Norm, 3-shot)            | 44.11                             | 43.22                                        |
-| Math-Hard (Exact-Match, 4-shot)   | 8.66                            | 8.77                                       |
-| GPQA (Acc-Norm, 0-shot)           | 28.30                             | 28.56                                        |
-| MUSR (Acc-Norm, 0-shot)           | 35.12                             | 35.26                                         |
-| MMLU-Pro (Acc, 5-shot)            | 26.87                             | 27.27                                        |
-| **Average Score**                 | **35.17**                         | **35.12**                                    |
-| **Recovery**                      | **100.00**                         | **99.84**                                    |
-#### HumanEval pass@1 scores
-| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 |
-|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| HumanEval Pass@1                        | 53.40                             |  52.30                                     |
 ## Inference Performance

 This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.
+<details>
+  <summary>Model Creation Code</summary>
 ```bash
 python quantize.py --model_path ibm-granite/granite-3.1-2b-instruct --quant_path "output_dir/granite-3.1-2b-instruct-quantized.w4a16" --calib_size 1024 --dampening_frac 0.01 --observer mse --group_size 64
 ```
 model.save_pretrained(quant_path, save_compressed=True)
 tokenizer.save_pretrained(quant_path)
 ```
+</details>
 ## Evaluation
 ### Accuracy
+<table>
+  <thead>
+    <tr>
+      <th>Category</th>
+      <th>Metric</th>
+      <th>ibm-granite/granite-3.1-2b-instruct</th>
+      <th>neuralmagic/granite-3.1-2b-instruct-quantized.w4a16</th>
+      <th>Recovery (%)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td rowspan="7"><b>OpenLLM v1</b></td>
+      <td>ARC-Challenge (Acc-Norm, 25-shot)</td>
+      <td>55.63</td>
+      <td>54.18</td>
+      <td>97.39</td>
+    </tr>
+    <tr>
+      <td>GSM8K (Strict-Match, 5-shot)</td>
+      <td>60.96</td>
+      <td>62.85</td>
+      <td>103.10</td>
+    </tr>
+    <tr>
+      <td>HellaSwag (Acc-Norm, 10-shot)</td>
+      <td>75.21</td>
+      <td>73.36</td>
+      <td>97.54</td>
+    </tr>
+    <tr>
+      <td>MMLU (Acc, 5-shot)</td>
+      <td>54.38</td>
+      <td>52.17</td>
+      <td>95.93</td>
+    </tr>
+    <tr>
+      <td>TruthfulQA (MC2, 0-shot)</td>
+      <td>55.93</td>
+      <td>56.83</td>
+      <td>101.61</td>
+    </tr>
+    <tr>
+      <td>Winogrande (Acc, 5-shot)</td>
+      <td>69.67</td>
+      <td>69.85</td>
+      <td>100.26</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>61.98</b></td>
+      <td><b>61.54</b></td>
+      <td><b>99.29</b></td>
+    </tr>
+    <tr>
+      <td rowspan="7"><b>OpenLLM v2</b></td>
+      <td>IFEval (Inst Level Strict Acc, 0-shot)</td>
+      <td>67.99</td>
+      <td>67.63</td>
+      <td>99.47</td>
+    </tr>
+    <tr>
+      <td>BBH (Acc-Norm, 3-shot)</td>
+      <td>44.11</td>
+      <td>43.22</td>
+      <td>97.98</td>
+    </tr>
+    <tr>
+      <td>Math-Hard (Exact-Match, 4-shot)</td>
+      <td>8.66</td>
+      <td>8.77</td>
+      <td>101.27</td>
+    </tr>
+    <tr>
+      <td>GPQA (Acc-Norm, 0-shot)</td>
+      <td>28.30</td>
+      <td>28.56</td>
+      <td>100.92</td>
+    </tr>
+    <tr>
+      <td>MUSR (Acc-Norm, 0-shot)</td>
+      <td>35.12</td>
+      <td>35.26</td>
+      <td>100.40</td>
+    </tr>
+    <tr>
+      <td>MMLU-Pro (Acc, 5-shot)</td>
+      <td>26.87</td>
+      <td>27.27</td>
+      <td>101.49</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>35.17</b></td>
+      <td><b>35.12</b></td>
+      <td><b>99.84</b></td>
+    </tr>
+    <tr>
+      <td rowspan="2"><b>HumanEval</b></td>
+      <td>HumanEval Pass@1</td>
+      <td>53.40</td>
+      <td>52.30</td>
+      <td>97.94</td>
+    </tr>
+  </tbody>
+</table>
 ## Inference Performance