Note Analyzes the optimization landscape of training neural networks using linear interpolation experiments.

Note Mixout is a regularization technique that improves stability and performance of LLMs on downstream tasks by stochastically mixing parameters of two models. Mixout acts as an L2 regularizer and prevents catastrophic forgetting/divergence.

Note A layer-wise fusion algorithm. This allows one-shot knowledge transfer without retraining and outperforms normal averaging. It also enables fusing models of different size, hence facilitating compression and federated learning

Note FedMA is a layer-wise federated learning algorithm for CNNs and LSTMs that averages hidden elements with similar feature extraction signatures.

Note Fisher merging, a weighted averaging method for combining NNs. It shows better performance over standard (unweighted )parameter averaging in model ensembling. It's a cheaper alternative to traditional transfer learning methods.

Note CLAFusion is a method to fuse neural networks doing cross-layer alignment and layer balancing in an efficient way. This works with networks with different depths.

Note Average weights of multiple fine-tuned models for computer vision. This yields better accuracy and no extra inference cost, and improves robustness to distribution shift.

Note Fuse multiple fine-tuned models by averaging their weights. It creates a better base model for future target tasks, showing better results vs using a pretrained model. This was published almost at the same time as model soups, but in this paper the approach is to have a generalizable base model that is then fine-tuned on different target tasks.

Note BTM is an algorithm to independently train expert LMs on different textual domains.

Note Proposes algorithms to align weights of independently trained models by permitting units

Note Lo-fi achieves similar or better accuracy compared to standard distributed training with communication when fine-tuning vision transformers on image classification and language models on text, without requiring any communication between nodes during training.

Note Iterative method to improve models by fusing together fine-tuned models without sharing datasets. First, you pick a base model. Different contributors download the base model and fine-tune on their own dataset. We then fuse/average all the fine-tunes to improve the base model. We repeat this process to generate a stronger base model with better performance and gains in few-shot learning.

Note Fine-tuning strategy that reuses fine-tunes of the same base models as initialization to parallel fine-tunings on the same target task. This leads to strong out-of-distribution generalization by leveraging diversity across auxiliary tasks. This technique is more robust to the choice of auxiliary tasks compared to other reusing strategies.

Note Improves backwards comaptibility of models when re-finetuning on a bigger dataset

Note A new method, TIES-MERGING (TRIM, ELECT SIGN & MERGE), that merges multiple models into a single multitask model. TRIM addresses interference that can be caused by redundant and conflicting parameters across models. Incorrect signs at top parameters cam lead to huge performance drops, so SIGN resolution help with this.

Note This is a technique to align LLMs using human preferences by interpolating weights fine-tuned on different proxy rewards.

Note UnIVAL proposes model merging across different modalities (image, video, audio-text tasks).

Note Why model merging works, when it can fail, and how it can be improved by unifying many existing merging schemes.