Update README.md

README.md

@@ -7,35 +7,30 @@ language:
 tags:
 - falcon3
 base_model: tiiuae/Falcon3-7B-Base
-license: other
-license_name: falcon-llm-license
+license: other 
+license_name: falcon-llm-license 
 license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
-library_name: transformers
 ---
 
-<div align="center">
-    <img src="https://huggingface.co/datasets/tiiuae/documentation-images/resolve/main/general/falco3-logo.png" alt="drawing" width="500"/>
-</div>
-
 # Falcon3-7B-Instruct
 
-**Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B.
+**Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B parameters.
 
-This repository contains the **Falcon3-7B-Instruct**. It achieves state of art results (at the time of release) on reasoning, language understanding, instruction following, code and mathematics tasks.
-Falcon3-7B-Instruct supports 4 languages (english, french, spanish, portuguese) and a context length up to 32K.
+This repository contains the **Falcon3-7B-Instruct**. It achieves state-of-the-art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks.
+Falcon3-7B-Instruct supports 4 languages (English, French, Spanish, Portuguese) and a context length of up to 32K.
 
 ## Model Details
 - Architecture
-  - Transformer based causal decoder only architecture
+  - Transformer-based causal decoder-only architecture
   - 28 decoder blocks
-  - Grouped query attention (GQA) for faster inference: 12 query heads and 4 key value heads
+  - Grouped Query Attention (GQA) for faster inference: 12 query heads and 4 key-value heads
   - Wider head dimension: 256
   - High RoPE value to support long context understanding: 1000042
   - Uses SwiGLU and RMSNorm
   - 32K context length
   - 131K vocab size
-- Pretrained on 14 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 1024 H100 GPU chips
-- Postrained on 1.2 million samples of STEM, conversations, code, safety and function call data
+- Pretrained on 14 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 2048 H100 GPU chips
+- Posttrained on 1.2 million samples of STEM, conversational, code, safety and function call data
 - Supports EN, FR, ES, PT
 - Developed by [Technology Innovation Institute](https://www.tii.ae)
 - License: TII Falcon-LLM License 2.0
@@ -91,66 +86,7 @@ print(response)
 <br>
 
 ## Benchmarks
-We report the official HuggingFace leaderboard normalized evaluations [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) in the following table.
-<table border="1" style="width: 100%; text-align: center; border-collapse: collapse;">
-    <colgroup>
-        <col style="width: 10%;">
-        <col style="width: 7%;">
-        <col style="width: 7%;">
-        <col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;">
-    </colgroup>
-    <thead>
-        <tr>
-            <th>Benchmark</th>
-            <th>Llama-3.1-8B-Instruct</th>
-            <th>Qwen2.5-7B-Instruct</th>
-            <th>Falcon3-7B-Instruct</th>
-        </tr>
-    </thead>
-    <tbody>
-        <tr>
-            <td>IFEval</td>
-            <td><b>78.56</b></td>
-            <td>75.85</td>
-            <td>76.12</td>
-        </tr>
-        <tr>
-            <td>BBH (3-shot)</td>
-            <td>29.89</td>
-            <td>34.89</td>
-            <td><b>37.92</b></td>
-        </tr>
-        <tr>
-            <td>MATH Lvl-5 (4-shot)</td>
-            <td>19.34</td>
-            <td>0.00</td>
-            <td><b>31.87</b></td>
-        </tr>              
-        <tr>
-            <td>GPQA (0-shot)</td>
-            <td>2.35</td>
-            <td>5.48</td>
-            <td><b>8.05</b></td>
-        </tr>
-        <tr>
-            <td>MUSR (0-shot)</td>
-            <td>8.41</td>
-            <td>8.45</td>
-            <td><b>21.17</b></td>
-        </tr>
-        <tr>
-            <td>MMLU-PRO (5-shot)</td>
-            <td>30.68</td>
-            <td><b>36.52</b></td>
-            <td>34.30</td>
-        </tr>        
-    </tbody>
-</table>
-
-Also, we report in the following table our internal pipeline benchmarks.
- - We use [lm-evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness).
- - We report **raw scores** obtained by applying chat template and fewshot_as_multiturn.
- - We use same batch-size across all models.
+We report in the following table our internal pipeline benchmarks:
 
 <table border="1" style="width: 100%; text-align: center; border-collapse: collapse;">
     <colgroup>
@@ -173,15 +109,15 @@ Also, we report in the following table our internal pipeline benchmarks.
         <tr>
             <td rowspan="3">General</td>
             <td>MMLU (5-shot)</td>
-            <td>68.2</td>
-            <td><b>73.5</b></td>
-            <td>70.5</td>
+            <td>55.9</td>
+            <td><b>72.4</b></td>
+            <td>68</td>
         </tr>
         <tr>
             <td>MMLU-PRO (5-shot)</td>
-            <td>36.4</td>
-            <td><b>43.1</b></td>
-            <td>40.7</td>
+            <td>21.8</td>
+            <td>35.8</td>
+            <td><b>40.7</b></td>
         </tr>
         <tr>
             <td>IFEval</td>
@@ -192,34 +128,34 @@ Also, we report in the following table our internal pipeline benchmarks.
         <tr>
             <td rowspan="3">Math</td>
             <td>GSM8K (5-shot)</td>
-            <td><b>82.6</b></td>
-            <td>72.0</td>
-            <td>81.4</td>
+            <td>78.1</td>
+            <td>77.5</td>
+            <td><b>79.1</b></td>
         </tr>
         <tr>
             <td>GSM8K (8-shot, COT)</td>
-            <td><b>85.4</b></td>
-            <td>76.6</td>
-            <td>79.7</td>
+            <td>79.8</td>
+            <td>72.7</td>
+            <td><b>80.9</b></td>
         </tr>
         <tr>
             <td>MATH Lvl-5 (4-shot)</td>
-            <td>15.4</td>
-            <td>-</td>
-            <td><b>29.4</b></td>
+            <td>10.4</td>
+            <td>26</td>
+            <td><b>33.1</b></td>
         </tr>
         <tr>
             <td rowspan="5">Reasoning</td>
             <td>Arc Challenge (25-shot)</td>
-            <td>58.6</td>
-            <td>57.8</td>
-            <td><b>62.6</b></td>
+            <td>46.6</td>
+            <td>55.7</td>
+            <td><b>65.9</b></td>
         </tr>
         <tr>
             <td>GPQA (0-shot)</td>
-            <td><b>33.5</b></td>
-            <td>32</td>
+            <td><b>33.6</b></td>
             <td>31.9</td>
+            <td>32</td>
         </tr>
         <tr>
             <td>GPQA (0-shot, COT)</td>
@@ -230,13 +166,13 @@ Also, we report in the following table our internal pipeline benchmarks.
         <tr>
             <td>MUSR (0-shot)</td>
             <td>38.6</td>
-            <td>41</td>
+            <td>40.7</td>
             <td><b>46.4</b></td>
         </tr>
         <tr>
             <td>BBH (3-shot)</td>
-            <td>48.6</td>
-            <td><b>54.1</b></td>
+            <td>43.7</td>
+            <td><b>53.9</b></td>
             <td>52.4</td>
         </tr>
         <tr>
@@ -267,30 +203,26 @@ Also, we report in the following table our internal pipeline benchmarks.
         <tr>
             <td rowspan="2">Instructions following</td>
             <td>MT-Bench (avg)</td>
-            <td>7.9</td>
-            <td><b>8.5</b></td>
-            <td>8.4</td>
+            <td>7.86</td>
+            <td><b>8.54</b></td>
+            <td>8.36</td>
         </tr>
         <tr>
-            <td>Alpaca (WC)</td>
-            <td>26.6</td>
+            <td>Alapaca (WC)</td>
+            <td>26.57</td>
             <td><b>31.5</b></td>
-            <td>26.1</td>
+            <td>26.13</td>
         </tr>
         <tr>
             <td>Tool use</td>
             <td>BFCL AST (avg)</td>
             <td>90.6</td>
             <td><b>91.4</b></td>
-            <td>89.5</td>
+            <td>72.3</td>
         </tr>
     </tbody>
 </table>
 
-## Useful links
-- View our [release blogpost](https://huggingface.co/blog/falcon3).
-- Feel free to join [our discord server](https://discord.gg/fwXpMyGc) if you have any questions or to interact with our researchers and developers.
-
 ## Technical Report
 Coming soon....
 

tokenizer_config.json

@@ -16219,7 +16219,7 @@
     ">>PASSWORD<<",
     ">>KEY<<"
   ],
-  "chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n  {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n  {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}{{ '<|system|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'user' %}{{ '<|user|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'assistant' %}{% if not loop.last %}{{ '<|assistant|>\n'  + message['content'] + eos_token + '\n' }}{% else %}{{ '<|assistant|>\n'  + message['content'] + eos_token }}{% endif %}{% endif %}{% if loop.last and add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}{% endfor %}",
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
   "extra_special_tokens": {},