vLLM serving is broken
#8
by
bedobedo
- opened
With vllm=v0.9.2, transformers==v4.53.3
Running vllm serve baidu/ERNIE-4.5-300B-A47B-PT results in
File "/usr/local/lib/python3.11/dist-packages/pydantic/_internal/_dataclasses.py", line 123, in __init__
s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
Value error, The checkpoint you are trying to load has model type `ernie4_5_moe` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
You may need to use this transformers release: https://github.com/huggingface/transformers/releases/tag/v4.53.2-Ernie-4.5-preview, the next official support of this model in transformers, I believe is v4.54.0 as stated in this preview release page.
bedobedo
changed discussion status to
closed