Update README.md

README.md

@@ -43,7 +43,7 @@ from transformers import AutoTokenizer
 from vllm import LLM, SamplingParams
 
 max_model_len, tp_size = 4096, 1
-model_name = "neuralmagic-ent/granite-3.1-2b-instruct-quantized.w4a16"
+model_name = "neuralmagic/granite-3.1-2b-instruct-quantized.w4a16"
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True)
 sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])
@@ -148,13 +148,16 @@ tokenizer.save_pretrained(quant_path)
 
 ## Evaluation
 
-The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
+The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard), OpenLLM Leaderboard [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
+
+<details>
+<summary>Evaluation Commands</summary>
 
 OpenLLM Leaderboard V1:
 ```
 lm_eval \
   --model vllm \
-  --model_args pretrained="neuralmagic-ent/granite-3.1-2b-instruct-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --model_args pretrained="neuralmagic/granite-3.1-2b-instruct-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
   --tasks openllm \
   --write_out \
   --batch_size auto \
@@ -162,11 +165,23 @@ lm_eval \
   --show_config
 ```
 
+OpenLLM Leaderboard V2:
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/granite-3.1-2b-instruct-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --tasks leaderboard \
+  --write_out \
+  --batch_size auto \
+  --output_path output_dir \
+  --show_config
+```
+
 #### HumanEval
 ##### Generation
 ```
 python3 codegen/generate.py \
-  --model neuralmagic-ent/granite-3.1-2b-instruct-quantized.w4a16 \
+  --model neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 \
   --bs 16 \
   --temperature 0.2 \
   --n_samples 50 \
@@ -176,20 +191,21 @@ python3 codegen/generate.py \
 ##### Sanitization
 ```
 python3 evalplus/sanitize.py \
-  humaneval/neuralmagic-ent--granite-3.1-2b-instruct-quantized.w4a16_vllm_temp_0.2
+  humaneval/neuralmagic--granite-3.1-2b-instruct-quantized.w4a16_vllm_temp_0.2
 ```
 ##### Evaluation
 ```
 evalplus.evaluate \
   --dataset humaneval \
-  --samples humaneval/neuralmagic-ent--granite-3.1-2b-instruct-quantized.w4a16_vllm_temp_0.2-sanitized
+  --samples humaneval/neuralmagic--granite-3.1-2b-instruct-quantized.w4a16_vllm_temp_0.2-sanitized
 ```
+</details>
 
 ### Accuracy
 
 #### OpenLLM Leaderboard V1 evaluation scores
 
-| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic-ent/granite-3.1-2b-instruct-quantized.w4a16 |
+| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 |
 |-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
 | ARC-Challenge (Acc-Norm, 25-shot)       | 55.63                             | 54.18                                      |
 | GSM8K (Strict-Match, 5-shot)            | 60.96                             | 62.85                                       |
@@ -201,7 +217,7 @@ evalplus.evaluate \
 | **Recovery**                            | **100.00**                        | **99.29**                                   |
 
 #### OpenLLM Leaderboard V2 evaluation scores
-| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic-ent/granite-3.1-2b-instruct-quantized.w4a16 |
+| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 |
 |-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
 | IFEval (Inst Level Strict Acc, 0-shot)| 67.99                           | 67.63                                          |
 | BBH (Acc-Norm, 3-shot)            | 44.11                             | 43.22                                        |
@@ -214,7 +230,7 @@ evalplus.evaluate \
 
 
 #### HumanEval pass@1 scores
-| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic-ent/granite-3.1-2b-instruct-quantized.w4a16 |
+| Metric                                  | ibm-granite/granite-3.1-2b-instruct             | neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 |
 |-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
 | HumanEval Pass@1                        | 53.40                             |  52.30                                     |