Spaces:

---

open-llm-leaderboard

/

open_llm_leaderboard

like 12.8k

Running on CPU Upgrade

cc @clefourrier

I'd like to test prithivMLmods's models and meet the same bug

https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/prithivMLmods/Sombrero-QwQ-32B-Elite11_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/prithivMLmods/Sombrero-QwQ-32B-Elite10_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/prithivMLmods/Sombrero-QwQ-32B-Elite9_eval_request_False_bfloat16_Original.json

Hi @rootxhacker ,

It looks like there is a shape mismatch when loading the models weights:

ValueError: Trying to set a tensor of shape torch.Size([151665, 5120]) in "weight" (which has shape torch.Size([152064, 5120])), this looks incorrect.

The expected tensor shape doesn't match the shape in the checkpoint, which usually happens due to an incorrect model configuration or a mismatch between the model architecture and the checkpoint

Could you please verify your models settings?

@Cran-May , could you please open a separate discussion? Thus I'll be able to check your models, it can be an another error

Hey @alozowski
I am able to load the model Apollo_v2-32B
Using below

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "rootxhacker/Apollo_v2-32B”

model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "How many r's are in the word "strawberry""
messages = [
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
**model_inputs,
max_new_tokens=32768
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)