microsoft

/

Phi-4-multimodal-instruct

like 1.28k

Follow

Microsoft 11k

I'm getting the following error on vLLM 0.8.1:

ValueError: lora_extra_vocab_size (0) must be one of (256, 512).

With the following command:

"${HOME}/vllm/bin/python" -m vllm.entrypoints.openai.api_server \
  --port 9099 \
  --model "${HOME}/models/textgen/Phi-4-multimodal-instruct" \
  --dtype auto \
  --gpu-memory-utilization 0.97 \
  --trust-remote-code \
  --max-model-len 43875 \
  --enable-lora \
  --lora-extra-vocab-size 0 \
  --max-lora-rank 320 \
  --limit-mm-per-prompt audio=3,image=3 \
  --max-loras 2 \
  --lora-modules speech="${HOME}/models/textgen/Phi-4-multimodal-instruct/speech-lora" vision="${HOME}/models/textgen/Phi-4-multimodal-instruct/vision-lora"

Removing the argument seems to resolve the error, but inference breaks down into gibberish after around 64 tokens. Is there a specific vLLM version we should be using?

Thanks again.

EDIT: I also experience this on vLLM 0.8.0 and nightly.