mlx-community

/

Qwen3-Embedding-8B-4bit-DWQ

like 3

Follow

MLX Community 5.97k

Currently, the mlx-lm doesn't support embedding models. I used my Swift version of mlx embeddings for the embedding model, which can be found at https://github.com/mzbac/mlx.embeddings. If you are working with Python, you may need to modify the mlx-lm by loading the model with the lm_head removed and do last token pooling. you can have a look of the gist here see https://gist.github.com/mzbac/96d8427542496fd75ca67849b5cb6ce9

By the way, the root cause of 8B not working with mlx-lm is 8B has tie_word_embeddings set to false. As a result, mlx-lm attempts to load the lm_head, which does not exist in the embedding model. I think the embedding model will never need the lm_head layer as it uses the hidestate from encoding layers, so it would be better for you to implement your own embedding model instead of using mlx-lm's default one. An easy workaround might be changing the config.json file to set tie_word_embeddings to true.

Here's a clearer example of how to load Qwen3 embedding using MLX-LM: https://gist.github.com/mzbac/67d92c2cfe8bcf75579ac55144d1174f