---
license: apache-2.0
---

# Weaver Distilled - MMLU Pro (ModernBERT-large)

This is a distilled cross-encoder model based on ModernBERT-large, trained to predict the correctness of answers on MMLU Pro. This specialized verifier was trained on Weaver scores aggregated over 35 different verifiers and reward models.

## Model Details

- **Base Model**: [answerdotai/ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large)
- **Architecture**: Cross-encoder with MLP head (1024 → 512 → 256 → 1)
- **Max Sequence Length**: 4096
- **Training Data**: [MMLU Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) Subset (500 queries) scored by 35 different LM Judges and reward models, aggregated to form sample-level scores with Weaver
- **Training Objective**: Binary classification (correct/incorrect answer prediction)

## Usage

```python
from custom_crossencoder import CustomCrossEncoder, TrainingConfig

# Initialize model
config = TrainingConfig(
    model_name="answerdotai/ModernBERT-large",
    max_length=4096,
    mlp_hidden_dims=[1024, 512, 256]
)
model = CustomCrossEncoder(config)

# Load checkpoint
model.load_state_dict(torch.load("hazyresearch/Weaver_Distilled_ModernBERT_Large_for_MMLU-Pro"))
model.eval()

# Get prediction
instruction = "Your instruction here"
answer = "Your answer here"
encoded = model.tokenizer(
    text=instruction,
    text_pair=answer,
    truncation=True,
    max_length=4096,
    padding="max_length",
    return_tensors="pt"
)
with torch.no_grad():
    prediction = model(encoded["input_ids"], encoded["attention_mask"])
```

## Running Evaluation

TODO: ADD EVALUATION_SIMPLE COMMAND HERE

## License

[Your chosen license]

## Citation

If you use this model in your research, please cite:

```bibtex
TODO
```