mergekit-community/mergekit-dare_ties-vnjbera

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using alpindale/Llama-3.2-3B as a base.

Models Merged

The following models were included in the merge:

• belyakoff/llama-3.2-3b-instruct-fine-tuned
• unsloth/Llama-3.2-3B-Instruct
• Medragondot/llama-3.2-3b-thinking

Configuration

The following YAML configuration was used to produce this model:

# Define the models to be used in the merge, along with their respective weight and density parameters.
models:
  - model: alpindale/Llama-3.2-3B  # The first model in the merge.
    parameters:
      weight: 0.3  # Weight determines how much influence this model will have in the final output. Higher weight means more influence.
      density: 0.35  # Density specifies how many of the model's parameters are retained during the merging process. Higher density keeps more parameters.

  - model: unsloth/Llama-3.2-3B-Instruct  # The second model in the merge.
    parameters:
      weight: 0.25  # A slightly smaller weight indicates this model will have less impact on the final merged model.
      density: 0.25  # A moderate density ensures this model's parameters are included, but not as heavily as others.

  - model: belyakoff/llama-3.2-3b-instruct-fine-tuned  # The third model in the merge.
    parameters:
      weight: 0.25  # Similar to the second model, this model contributes less to the final model.
      density: 0.25  # Keeps a balanced contribution of parameters for the merge.

  - model: Medragondot/llama-3.2-3b-thinking  # The fourth model in the merge.
    parameters:
      weight: 0.2  # This model will have the least influence on the merged output.
      density: 0.15  # The lowest density means fewer of this model’s parameters will contribute to the final merge.

# Specify the merge method to be used.
merge_method: dare_ties  # The DARE-TIES method is used to merge models by estimating residuals between them. This allows for fine-tuning and adjusting contributions for each model layer.

# Set the base model for the merge. This model serves as the foundation for blending the other models.
base_model: alpindale/Llama-3.2-3B  # The base model is typically the one that will retain the highest influence in the final merged model.

# Define additional parameters to customize the merging behavior.
parameters:
  normalize: true  # Normalization ensures the weights across models are balanced so the merge remains stable and well-scaled.
  int8_mask: true  # Enables int8 masking, which optimizes performance by using 8-bit integers for certain computations, reducing memory usage.
  interpolation_factor: 0.7  # Controls the blending strength between the models. Values closer to 1 will favor the base model, while values closer to 0 distribute more influence evenly among models.
dtype: bfloat16  # Uses bfloat16 (brain floating point 16) format to store weights, offering a good balance between numerical precision and memory efficiency for model merging.