Update README.md
Browse files
README.md
CHANGED
|
@@ -18,7 +18,8 @@ model-index:
|
|
| 18 |
value: 73.04
|
| 19 |
name: normalized accuracy
|
| 20 |
source:
|
| 21 |
-
url:
|
|
|
|
| 22 |
name: Open LLM Leaderboard
|
| 23 |
- task:
|
| 24 |
type: text-generation
|
|
@@ -34,7 +35,8 @@ model-index:
|
|
| 34 |
value: 88.61
|
| 35 |
name: normalized accuracy
|
| 36 |
source:
|
| 37 |
-
url:
|
|
|
|
| 38 |
name: Open LLM Leaderboard
|
| 39 |
- task:
|
| 40 |
type: text-generation
|
|
@@ -51,7 +53,8 @@ model-index:
|
|
| 51 |
value: 75.49
|
| 52 |
name: accuracy
|
| 53 |
source:
|
| 54 |
-
url:
|
|
|
|
| 55 |
name: Open LLM Leaderboard
|
| 56 |
- task:
|
| 57 |
type: text-generation
|
|
@@ -67,7 +70,8 @@ model-index:
|
|
| 67 |
- type: mc2
|
| 68 |
value: 69.38
|
| 69 |
source:
|
| 70 |
-
url:
|
|
|
|
| 71 |
name: Open LLM Leaderboard
|
| 72 |
- task:
|
| 73 |
type: text-generation
|
|
@@ -84,7 +88,8 @@ model-index:
|
|
| 84 |
value: 85.32
|
| 85 |
name: accuracy
|
| 86 |
source:
|
| 87 |
-
url:
|
|
|
|
| 88 |
name: Open LLM Leaderboard
|
| 89 |
- task:
|
| 90 |
type: text-generation
|
|
@@ -101,8 +106,11 @@ model-index:
|
|
| 101 |
value: 67.7
|
| 102 |
name: accuracy
|
| 103 |
source:
|
| 104 |
-
url:
|
|
|
|
| 105 |
name: Open LLM Leaderboard
|
|
|
|
|
|
|
| 106 |
---
|
| 107 |
this is [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b), dequantised from q5 to f16 && transposed to pytorch. shapes have been rotated less wrongly than in [alpindale/miqu-1-70b-pytorch](https://huggingface.co/alpindale/miqu-1-70b-pytorch/tree/main)
|
| 108 |
|
|
@@ -110,7 +118,7 @@ usage
|
|
| 110 |
```python
|
| 111 |
from transformers import LlamaForCausalLM as LLM, LlamaTokenizer as LT
|
| 112 |
|
| 113 |
-
lt = LT.from_pretrained("
|
| 114 |
t = lt("[INST] eloquent high camp prose about a cute catgirl [/INST]", return_tensors='pt').input_ids.cuda()
|
| 115 |
|
| 116 |
llm = LLM.from_pretrained("152334H/miqu-1-70b-sf", device_map='auto') # note: you may need many gpus for this
|
|
@@ -144,7 +152,7 @@ So let us raise our teacups in honor of this fabulous feline, this queen of camp
|
|
| 144 |
|
| 145 |

|
| 146 |
|
| 147 |
-
some benchmarks
|
| 148 |
|
| 149 |
```
|
| 150 |
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|
|
@@ -245,5 +253,4 @@ Detailed results can be found [here](https://huggingface.co/datasets/open-llm-le
|
|
| 245 |
|MMLU (5-Shot) |75.49|
|
| 246 |
|TruthfulQA (0-shot) |69.38|
|
| 247 |
|Winogrande (5-shot) |85.32|
|
| 248 |
-
|GSM8k (5-shot) |67.70|
|
| 249 |
-
|
|
|
|
| 18 |
value: 73.04
|
| 19 |
name: normalized accuracy
|
| 20 |
source:
|
| 21 |
+
url: >-
|
| 22 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
| 23 |
name: Open LLM Leaderboard
|
| 24 |
- task:
|
| 25 |
type: text-generation
|
|
|
|
| 35 |
value: 88.61
|
| 36 |
name: normalized accuracy
|
| 37 |
source:
|
| 38 |
+
url: >-
|
| 39 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
| 40 |
name: Open LLM Leaderboard
|
| 41 |
- task:
|
| 42 |
type: text-generation
|
|
|
|
| 53 |
value: 75.49
|
| 54 |
name: accuracy
|
| 55 |
source:
|
| 56 |
+
url: >-
|
| 57 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
| 58 |
name: Open LLM Leaderboard
|
| 59 |
- task:
|
| 60 |
type: text-generation
|
|
|
|
| 70 |
- type: mc2
|
| 71 |
value: 69.38
|
| 72 |
source:
|
| 73 |
+
url: >-
|
| 74 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
| 75 |
name: Open LLM Leaderboard
|
| 76 |
- task:
|
| 77 |
type: text-generation
|
|
|
|
| 88 |
value: 85.32
|
| 89 |
name: accuracy
|
| 90 |
source:
|
| 91 |
+
url: >-
|
| 92 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
| 93 |
name: Open LLM Leaderboard
|
| 94 |
- task:
|
| 95 |
type: text-generation
|
|
|
|
| 106 |
value: 67.7
|
| 107 |
name: accuracy
|
| 108 |
source:
|
| 109 |
+
url: >-
|
| 110 |
+
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
|
| 111 |
name: Open LLM Leaderboard
|
| 112 |
+
language:
|
| 113 |
+
- en
|
| 114 |
---
|
| 115 |
this is [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b), dequantised from q5 to f16 && transposed to pytorch. shapes have been rotated less wrongly than in [alpindale/miqu-1-70b-pytorch](https://huggingface.co/alpindale/miqu-1-70b-pytorch/tree/main)
|
| 116 |
|
|
|
|
| 118 |
```python
|
| 119 |
from transformers import LlamaForCausalLM as LLM, LlamaTokenizer as LT
|
| 120 |
|
| 121 |
+
lt = LT.from_pretrained("152334H/miqu-1-70b-sf")
|
| 122 |
t = lt("[INST] eloquent high camp prose about a cute catgirl [/INST]", return_tensors='pt').input_ids.cuda()
|
| 123 |
|
| 124 |
llm = LLM.from_pretrained("152334H/miqu-1-70b-sf", device_map='auto') # note: you may need many gpus for this
|
|
|
|
| 152 |
|
| 153 |

|
| 154 |
|
| 155 |
+
## some benchmarks
|
| 156 |
|
| 157 |
```
|
| 158 |
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|
|
|
|
| 253 |
|MMLU (5-Shot) |75.49|
|
| 254 |
|TruthfulQA (0-shot) |69.38|
|
| 255 |
|Winogrande (5-shot) |85.32|
|
| 256 |
+
|GSM8k (5-shot) |67.70|
|
|
|