Update README.md

README.md

@@ -120,20 +120,94 @@ vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://do
 
 ## Evaluation
 
-The model was evaluated on the OpenLLM leaderboard tasks (version 1), using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and [vLLM](https://docs.vllm.ai/en/stable/).
+The model was evaluated on the OpenLLM leaderboard tasks (versions 1 and 2), using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), and on reasoning tasks using [lighteval](https://github.com/neuralmagic/lighteval/tree/reasoning).
+[vLLM](https://docs.vllm.ai/en/stable/) was used for all evaluations.
 
 <details>
   <summary>Evaluation details</summary>
 
+  **lm-evaluation-harness**
   ```
   lm_eval \
     --model vllm \
-    --model_args pretrained="RedHatAI/Qwen3-1.7B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=2 \
+    --model_args pretrained="RedHatAI/Qwen3-1.7B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=1 \
     --tasks openllm \
     --apply_chat_template\
     --fewshot_as_multiturn \
     --batch_size auto
   ```
+
+  ```
+  lm_eval \
+    --model vllm \
+    --model_args pretrained="RedHatAI/Qwen3-1.7B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=1 \
+    --tasks mgsm \
+    --apply_chat_template\
+    --batch_size auto
+  ```
+
+  ```
+  lm_eval \
+    --model vllm \
+    --model_args pretrained="RedHatAI/Qwen3-1.7B-FP8-dynamic",dtype=auto,gpu_memory_utilization=0.5,max_model_len=16384,enable_chunk_prefill=True,tensor_parallel_size=1 \
+    --tasks leaderboard \
+    --apply_chat_template\
+    --fewshot_as_multiturn \
+    --batch_size auto
+  ```
+
+  **lighteval**
+  
+  lighteval_model_arguments.yaml
+  ```yaml 
+  model_parameters:
+    model_name: RedHatAI/Qwen3-1.7B-FP8-dynamic
+    dtype: auto
+    gpu_memory_utilization: 0.9
+    max_model_length: 40960
+    generation_parameters:
+      temperature: 0.6
+      top_k: 20
+      min_p: 0.0
+      top_p: 0.95
+      max_new_tokens: 32768
+  ```
+
+  ```
+  lighteval vllm \
+    --model_args lighteval_model_arguments.yaml \
+    --tasks lighteval|aime24|0|0 \
+    --use_chat_template = true
+  ```
+
+  ```
+  lighteval vllm \
+    --model_args lighteval_model_arguments.yaml \
+    --tasks lighteval|aime25|0|0 \
+    --use_chat_template = true
+  ```
+
+  ```
+  lighteval vllm \
+    --model_args lighteval_model_arguments.yaml \
+    --tasks lighteval|math_500|0|0 \
+    --use_chat_template = true
+  ```
+
+  ```
+  lighteval vllm \
+    --model_args lighteval_model_arguments.yaml \
+    --tasks lighteval|gpqa:diamond|0|0 \
+    --use_chat_template = true
+  ```
+
+  ```
+  lighteval vllm \
+    --model_args lighteval_model_arguments.yaml \
+    --tasks extended|lcb:codegeneration \
+    --use_chat_template = true
+  ```
+
 </details>
 
 ### Accuracy
@@ -223,4 +297,140 @@ The model was evaluated on the OpenLLM leaderboard tasks (version 1), using [lm-
    <td><strong>98.6%</strong>
    </td>
   </tr>
+  <tr>
+   <td rowspan="7" ><strong>OpenLLM v2</strong>
+   </td>
+   <td>MMLU-Pro (5-shot)
+   </td>
+   <td>23.45
+   </td>
+   <td>21.38
+   </td>
+   <td>91.1%
+   </td>
+  </tr>
+  <tr>
+   <td>IFEval (0-shot)
+   </td>
+   <td>71.08
+   </td>
+   <td>70.93
+   </td>
+   <td>99.8%
+   </td>
+  </tr>
+  <tr>
+   <td>BBH (3-shot)
+   </td>
+   <td>7.13
+   </td>
+   <td>5.41
+   </td>
+   <td>---
+   </td>
+  </tr>
+  <tr>
+   <td>Math-lvl-5 (4-shot)
+   </td>
+   <td>35.91
+   </td>
+   <td>34.71
+   </td>
+   <td>96.7%
+   </td>
+  </tr>
+  <tr>
+   <td>GPQA (0-shot)
+   </td>
+   <td>0.11
+   </td>
+   <td>0.00
+   </td>
+   <td>---
+   </td>
+  </tr>
+  <tr>
+   <td>MuSR (0-shot)
+   </td>
+   <td>7.97
+   </td>
+   <td>7.18
+   </td>
+   <td>---
+   </td>
+  </tr>
+  <tr>
+   <td><strong>Average</strong>
+   </td>
+   <td><strong>24.28</strong>
+   </td>
+   <td><strong>23.27</strong>
+   </td>
+   <td><strong>95.8%</strong>
+   </td>
+  </tr>
+  <tr>
+   <td><strong>Multilingual</strong>
+   </td>
+   <td>MGSM (0-shot)
+   </td>
+   <td>22.10
+   </td>
+   <td>
+   </td>
+   <td>
+   </td>
+  </tr>
+  <tr>
+   <td rowspan="6" ><strong>Reasoning<br>(generation)</strong>
+   </td>
+   <td>AIME 2024
+   </td>
+   <td>43.96
+   </td>
+   <td>40.10
+   </td>
+   <td>91.2%
+   </td>
+  </tr>
+  <tr>
+   <td>AIME 2025
+   </td>
+   <td>32.29
+   </td>
+   <td>32.29
+   </td>
+   <td>100.0%
+   </td>
+  </tr>
+  <tr>
+   <td>GPQA diamond
+   </td>
+   <td>38.38
+   </td>
+   <td>38.89
+   </td>
+   <td>101.3%
+   </td>
+  </tr>
+  <tr>
+   <td>Math-lvl-5
+   </td>
+   <td>89.00
+   </td>
+   <td>88.80
+   </td>
+   <td>99.8%
+   </td>
+  </tr>
+  <tr>
+   <td>LiveCodeBench
+   </td>
+   <td>33.44
+   </td>
+   <td>
+   </td>
+   <td>
+   </td>
+  </tr>
 </table>