mlx-community

/

e5-mistral-7b-instruct-mlx

like 9

Follow

MLX Community 4.37k

To run the example with the current version of mlx-llm I needed to change the name to an existing registered model name mistral_7b_instruct_v0.2 and call embeds, _ = model.embed(x) however the results aren't consistant after each run.

Fix to execute example:

model = create_model(
  "mistral_7b_instruct_v0.2",
  weights="e5-mistral-7b-instruct/weights.npz",
  strict=False
)

# ... rest of code

# compute embed
embeds, _ = model.embed(x)

Custom Registration Attempt

After the previous results were inconsistent, I tried registering a model with v0.1 specifications:

# src/mlx_llm/model/_registry.py

@register_model("mistral_7b_v0.1")
def mistral_7b_v01(
    vocab_size: int = 32000, norm_eps: float = 1e-5, rope_theta: float = 10000.0, rope_traditional: bool = True
) -> Tuple[Transformer, ModelConfig]:
    """Create a Mistral 7B v0.1 model.

    Args:
        vocab_size (int, optional): vocab size. Defaults to 32000.
        norm_eps (float, optional): norm epsilon. Defaults to 1e-5.
        rope_theta (float, optional): rope theta. Defaults to 10000.0.
        rope_traditional (bool, optional): whether to use traditional rope. Defaults to True.

    Returns:
        Tuple[Transformer, ModelConfig]: model, config
    """
    model = Transformer(
        dim=4096,
        hidden_dim=14336,
        vocab_size=vocab_size,
        n_layers=32,
        n_heads=32,
        n_kv_heads=8,
        norm_eps=norm_eps,
        rope_theta=rope_theta,
        rope_traditional=rope_traditional,
    )

    config = ModelConfig(
        hf=HFConfig(repo_id="mistralai/Mistral-7B-v0.1"),  
        converter=mistral_to_mlxllm
    )

    return model, config

And changed the example code to:

model = create_model(
  "mistral_7b_v0.1",
  weights="e5-mistral-7b-instruct/weights.npz",
  strict=False
)

Unfortunately, the results are still inconsistent.