allenai

/

OLMo-7B-Instruct-hf

like 2

Follow

Ai2 1,387

Thanks for all of your releases!
Just noting that there's a small typo in the model card. The 5th line of the inference Hello World script should use 'message', not 'messages'.

message = [{"role": "user", "content": "What is 2+2?"}]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")

I also think the olmo.generate() call doesn't need the unpack operator as inputs should just be a tensor, not a dict.

Here's my corrected version which runs:

from transformers import AutoModelForCausalLM, AutoTokenizer
olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-7B-Instruct-hf")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-7B-Instruct-hf")
message = [{"role": "user", "content": "What is 2+2?"}]
inputs = tokenizer.apply_chat_template(message, tokenize=True, add_generation_prompt=True, return_tensors="pt")
# optional verifying cuda
inputs = inputs.cuda()
olmo = olmo.to('cuda')
response = olmo.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])