microsoft

/

Phi-4-mini-instruct

like 320

Follow

Microsoft 10.1k

Description

When running the sample inference code with the Phi-4-mini-instruct checkpoints, I encountered a RuntimeError related to tensor dimension mismatch during generation.

Reproduction Steps

1. Obtain the Phi-4-mini-instruct model checkpoints.
2. Run the following sample code (inference.py)

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
 
torch.random.manual_seed(0)

model_path = "microsoft/Phi-4-mini-instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_path)
 
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
]
 
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)
 
generation_args = {
    "max_new_tokens": 500,
    "return_full_text": False,
    "temperature": 0.0,
    "do_sample": False,
}
 
output = pipe(messages, **generation_args)
print(output[0]['generated_text'])

3. Execute the script in a CUDA-enabled environment (Python 3.10).
4. The error occurs during the generation step.

Error Output (excerpt)

...
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.72it/s]
Device set to use cuda:0
...
Traceback (most recent call last):
  File "/home/<username>/new_work/phi4mini/cookbook/inference.py", line 36, in <module>
    output = pipe(messages, **generation_args)
  File ".../transformers/pipelines/text_generation.py", line 278, in __call__
    return super().__call__(Chat(text_inputs), **kwargs)
  ...
  File ".../transformers/generation/utils.py", line 3309, in _sample
    input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
RuntimeError: Tensors must have same number of dimensions: got 2 and 3

Additional Notes:

The execution environment uses CUDA on Python 3.10.
Personal identifiers in paths have been replaced with for privacy.
Could you please investigate this issue and provide guidance on how to resolve it? Any assistance would be greatly appreciated.