File size: 2,554 Bytes
c31216e 90544bd 6726500 c31216e b302259 6726500 b302259 40d10c3 b302259 6726500 b302259 6726500 b302259 6726500 9bbf33c b302259 6726500 9bbf33c 6726500 9bbf33c 6726500 9bbf33c 6726500 b302259 6726500 b302259 9bbf33c 6726500 b302259 6726500 b302259 6726500 b302259 342d7be 6726500 b302259 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
---
license: mit
pipeline_tag: text-classification
library_name: transformers
base_model: answerdotai/ModernBERT-large
tags:
- academic
- reasoning
- verification
- weaver
- cross-encoder
- mmlu
language:
- en
---
# Weaver Distilled for MMLU-Pro
This is a distilled cross-encoder model based on ModernBERT-large, trained to predict the correctness of answers on MMLU Pro. This specialized verifier was trained on Weaver scores aggregated over 35 different verifiers and reward models.
## Model Details
- **Base Model**: [answerdotai/ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) (395M parameters)
- **Architecture**: Cross-encoder with MLP head (1024 → 512 → 256 → 1)
- **Max Sequence Length**: 4096 tokens
- **Training Data**: MMLU-Pro problems with Weaver scores from 35 LM judges and reward models
- **Task**: Binary classification for answer correctness prediction
## Quick Start
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
# Load model and tokenizer
model_name = "hazyresearch/Weaver_Distilled_for_MMLU-Pro"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
# Example usage
instruction = "Which of the following is NOT a fundamental force in physics? A) Electromagnetic force B) Weak nuclear force C) Strong nuclear force D) Centrifugal force"
response = "The answer is D) Centrifugal force. Centrifugal force is not a fundamental force but rather a fictitious force that appears in rotating reference frames..."
# Tokenize input pair
inputs = tokenizer(
instruction,
response,
truncation=True,
max_length=4096,
padding=True,
return_tensors="pt"
)
# Get correctness score
with torch.no_grad():
outputs = model(**inputs)
score = torch.sigmoid(outputs.logits).item()
print(f"Correctness score: {score:.3f}")
print(f"Prediction: {'Correct' if score > 0.5 else 'Incorrect'}")
```
## Citation
```bibtex
@misc{saadfalcon2025shrinkinggenerationverificationgapweak,
title={Shrinking the Generation-Verification Gap with Weak Verifiers},
author={Jon Saad-Falcon and E. Kelly Buchanan and Mayee F. Chen and Tzu-Heng Huang and Brendan McLaughlin and Tanvir Bhathal and Shang Zhu and Ben Athiwaratkun and Frederic Sala and Scott Linderman and Azalia Mirhoseini and Christopher Ré},
year={2025},
eprint={2506.18203},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2506.18203},
}
``` |