MoLA-LM: Mixture of LoRA Adapters LLM

MoLA-LM combines multiple LoRA adapters with an intelligent router to automatically select the best adapter for each input prompt. This approach enables specialized performance across different tasks while maintaining efficiency.

⚠️ This is a test model

Evals are coming...

Model Details

Model Type: Mixture of LoRA Adapters Language Model
Base Model: Qwen/Qwen2.5-3B-Instruct
Total Adapters: 11
Architecture: Custom MoLAForCausalLM with automatic adapter routing

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model (trust_remote_code=True is required for custom architecture)
model = AutoModelForCausalLM.from_pretrained(
    "MoLA-LLM/MoLA-11x3b-v1", 
    trust_remote_code=True, 
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("MoLA-LLM/MoLA-11x3b-v1", trust_remote_code=True)

# Use like any other language model - adapter selection is automatic
prompt = "Write a Python function to calculate fibonacci numbers"
messages = [{"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7, do_sample=True)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)

print(f"Selected LoRA: {model.get_current_lora()}")
print(response)

You can also use load_in_4bit and load_in_8bit directly when loading!

Training Details

This all ya get for now: (Mad respect to the ogs)

Available LoRA Adapters

The model includes specialized adapters for:

reasoning
planning
information_seeking
math
creative_writing
brainstorming
advice_seeking
role_playing
data_analysis
coding
editing

Architecture

The MoLA-LM architecture consists of:

Base Model: Qwen/Qwen2.5-3B-Instruct
Router Network: Sentence transformer + MLP for adapter selection
LoRA Adapters: 11 task-specific fine-tuned adapters
Dynamic Switching: Automatic adapter application based on input

Technical Details

Router Input: 512-token context window for task classification
Adapter Count: 11 specialized LoRA adapters
Selection Method: Argmax over router logits
Memory Efficient: Only one adapter active at a time

Paper coming soon™