This model is part of LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation.
LoRI (LoRA with Reduced Interference) is a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference.
LoRI (LoRA with Reduced Interference) is a novel parameter-efficient fine-tuning (PEFT) method designed to address the overhead and parameter interference often encountered in multi-task scenarios for Large Language Models (LLMs). By freezing projection matrices A as random projections and sparsifying matrices B with task-specific masks, LoRI significantly reduces trainable parameters while preserving strong task performance and minimizing cross-task interference. This approach also supports continual learning by leveraging sparsity to mitigate catastrophic forgetting.
This specific model, LoRI-D_code_llama3_rank_32, is a LoRI adapter based on the meta-llama/Meta-Llama-3-8B model, specifically fine-tuned for code generation tasks.
meta-llama/Meta-Llama-3-8BLoRI adapters are intended for efficient fine-tuning of Large Language Models on various downstream tasks, including:
They can be loaded as PEFT adapters on top of a base LLM to enhance performance with reduced parameter overhead.
LoRI enables effective adapter merging for multi-task applications, allowing a single model to perform well across several distinct tasks without significant performance degradation. It also supports continual learning, facilitating model adaptation to new tasks sequentially while mitigating catastrophic forgetting.
This model is not intended for generating harmful, biased, or unethical content. Users should ensure compliance with the ethical guidelines of the base models and apply appropriate content filtering. It is not suitable for standalone use without a compatible base LLM or for tasks outside its trained domains without further adaptation.
As an adapter for Large Language Models, LoRI inherits potential biases and limitations from its base model (meta-llama/Meta-Llama-3-8B) and the datasets it was fine-tuned on. Users should be aware that results might reflect biases present in the training data. While LoRI aims to reduce cross-task interference, complete elimination of such interference or catastrophic forgetting is not guaranteed. Performance on highly out-of-distribution tasks may be limited.
Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. It is recommended to perform further evaluation on specific target datasets and contexts to understand its performance and potential biases relevant to the application. Implement appropriate safeguards and content filtering when deploying the model in production environments.
Use the code below to get started with the model.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load the base model
base_model_path = "meta-llama/Meta-Llama-3-8B"
base_model = AutoModelForCausalLM.from_pretrained(
base_model_path,
torch_dtype=torch.bfloat16, # Use torch.float16 if bfloat16 is not supported
device_map="auto" # Distributes the model across available devices
)
# Load the LoRI adapter (this model is for code generation)
lori_adapter_path = "tomg-group-umd/LoRI-D_code_llama3_rank_32"
model = PeftModel.from_pretrained(base_model, lori_adapter_path)
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_path)
# Example usage for code generation
prompt = "def factorial(n):
if n == 0:
return 1
else:
return n * "
# Apply chat template if using a chat-tuned base model (Llama-3 is chat-tuned)
# Or directly tokenize if the base model is not primarily chat-tuned
messages = [{"role": "user", "content": prompt}]
input_ids = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
# Generate text
outputs = model.generate(
input_ids,
max_new_tokens=50,
do_sample=True,
temperature=0.7,
top_p=0.9
)
# Decode and print the generated response (excluding the prompt for cleaner output)
response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True)
print(response)
LoRI is implemented using Fully Sharded Data Parallel (FSDP) and can be executed in a multi-GPU environment. Training scripts are provided for various tasks covering Natural Language Understanding (NLU), Code Generation, Mathematical Reasoning, and Safety Alignment.
LoRI adapters were trained on datasets corresponding to various tasks:
More details on these datasets can be found in the LoRI GitHub repository.
LoRI training typically involves two main stages:
A are frozen as random projections, and matrices B are trained. This stage is used to extract sparse masks.B are sparsified using task-specific masks (e.g., 90% sparsity for LoRI-S).lora_alpha): 64r): 32 (for this specific model LoRI-D_code_llama3_rank_32)lora_dropout): 0.05q_proj, o_proj, k_proj, v_proj, down_proj, gate_proj, up_projExtensive experiments were conducted across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks. The paper demonstrates that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than standard LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code generation performance on HumanEval is evaluated using the bigcode-evaluation-harness. For detailed quantitative results, please refer to the paper.
LoRI introduces a novel architecture built upon existing Transformer-based LLMs, adapting the standard LoRA approach. It achieves efficiency and interference reduction by:
A as random projections.B using task-specific masks.
The primary objective is to minimize cross-task interference in multi-task low-rank adaptation, while significantly reducing the trainable parameter count and maintaining strong performance.The training and evaluation were performed on multi-GPU environments, leveraging technologies like FSDP. Experiments were conducted with LLaMA-3-8B and Mistral-7B base models.
The implementation relies on PyTorch, transformers, and peft libraries. It builds on the codebase of dpo-rlaif and incorporates code from lottery-ticket-adaptation. Evaluation for code generation uses the bigcode-evaluation-harness.
If you use LoRI in your work, please cite:
@article{zhang2025lori,
title={LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation},
author={Zhang, Juzheng and You, Jiacheng and Panda, Ashwinee and Goldstein, Tom},
journal={arXiv preprint arXiv:2504.07448},
year={2025}
}
For further details on installation, training from scratch, adapter merging, continual learning, and customizing base models/losses, please refer to the official GitHub repository.
Base model
meta-llama/Meta-Llama-3-8B