metadata
library_name: transformers
tags:
- mergekit
- merge
base_model:
- Solshine/reflection-llama-3.1-8B
- Solshine/Meta-Llama-3.1-8B-Instruct-Python-Coder
- mlabonne/Hermes-3-Llama-3.1-8B-lorablated
model-index:
- name: Llama-3-1-big-thoughtful-passthrough-merge-2
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 25.47
name: strict accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Solshine/Llama-3-1-big-thoughtful-passthrough-merge-2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 5.01
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Solshine/Llama-3-1-big-thoughtful-passthrough-merge-2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 0.15
name: exact match
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Solshine/Llama-3-1-big-thoughtful-passthrough-merge-2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 1.23
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Solshine/Llama-3-1-big-thoughtful-passthrough-merge-2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 6.75
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Solshine/Llama-3-1-big-thoughtful-passthrough-merge-2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 2.06
name: accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Solshine/Llama-3-1-big-thoughtful-passthrough-merge-2
name: Open LLM Leaderboard
State of the art for size on Open LLM Leaderboard on acc_norm score
SOTA at size level as of acc_norm score on 9/30/2024, viewable at open-llm-leaderboard/Solshine__Llama-3-1-big-thoughtful-passthrough-merge-2-details
acc_norm of 31.5% according to open llm leaderboard test result dataset.
Due to the merged and minimally retrained nature of this model, this score may not reflect in human evaluated general performance in some domains.
merge
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the passthrough merge method.
Models Merged
The following models were included in the merge:
- Solshine/reflection-llama-3.1-8B
- Solshine/Meta-Llama-3.1-8B-Instruct-Python-Coder
- mlabonne/Hermes-3-Llama-3.1-8B-lorablated
Configuration
The following YAML configuration was used to produce this model:
slices:
- sources:
- layer_range: [0, 16]
model: mlabonne/Hermes-3-Llama-3.1-8B-lorablated
- sources:
- layer_range: [4, 20]
model: Solshine/reflection-llama-3.1-8B
- sources:
- layer_range: [8, 24]
model: Solshine/Meta-Llama-3.1-8B-Instruct-Python-Coder
- sources:
- layer_range: [12, 28]
model: Solshine/reflection-llama-3.1-8B
- sources:
- layer_range: [16, 32]
model: mlabonne/Hermes-3-Llama-3.1-8B-lorablated
merge_method: passthrough
dtype: float16