--- language: - en license: apache-2.0 tags: - sentence-transformers - cross-encoder - generated_from_trainer - dataset_size:482388 - loss:BinaryCrossEntropyLoss base_model: answerdotai/ModernBERT-base pipeline_tag: text-ranking library_name: sentence-transformers metrics: - map - mrr@10 - ndcg@10 model-index: - name: ModernBERT-base trained on GooAQ results: - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: gooaq dev type: gooaq-dev metrics: - type: map value: 0.7089 name: Map - type: mrr@10 value: 0.7076 name: Mrr@10 - type: ndcg@10 value: 0.755 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoMSMARCO R100 type: NanoMSMARCO_R100 metrics: - type: map value: 0.554 name: Map - type: mrr@10 value: 0.5472 name: Mrr@10 - type: ndcg@10 value: 0.6229 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNFCorpus R100 type: NanoNFCorpus_R100 metrics: - type: map value: 0.3421 name: Map - type: mrr@10 value: 0.5284 name: Mrr@10 - type: ndcg@10 value: 0.3792 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNQ R100 type: NanoNQ_R100 metrics: - type: map value: 0.6312 name: Map - type: mrr@10 value: 0.638 name: Mrr@10 - type: ndcg@10 value: 0.6915 name: Ndcg@10 - task: type: cross-encoder-nano-beir name: Cross Encoder Nano BEIR dataset: name: NanoBEIR R100 mean type: NanoBEIR_R100_mean metrics: - type: map value: 0.5091 name: Map - type: mrr@10 value: 0.5712 name: Mrr@10 - type: ndcg@10 value: 0.5645 name: Ndcg@10 --- # ModernBERT-base trained on GooAQ This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search. ## Model Details ### Model Description - **Model Type:** Cross Encoder - **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) - **Maximum Sequence Length:** 8192 tokens - **Number of Output Labels:** 1 label - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import CrossEncoder # Download from the 🤗 Hub model = CrossEncoder("tomaarsen/reranker-ModernBERT-base-gooaq-bce-soft-negs") # Get scores for pairs of texts pairs = [ ['what is the difference between ground level ozone and the ozone layer?', 'Here, ground-level or "bad" ozone is an air pollutant that is harmful to breathe and it damages crops, trees and other vegetation. ... The stratosphere or "good" ozone layer extends upward from about 6 to 30 miles and protects life on Earth from the sun\'s harmful ultraviolet (UV) rays.'], ['what is the difference between ground level ozone and the ozone layer?', 'In the stratosphere, temperature increases with altitude. The reason is that the direct heat source for the stratosphere is the Sun. A layer of ozone molecules absorbs solar radiation, which heats the stratosphere.'], ['what is the difference between ground level ozone and the ozone layer?', "Atmosphere layers. Earth's atmosphere is divided into five main layers: the exosphere, the thermosphere, the mesosphere, the stratosphere and the troposphere. ... Ozone is abundant here and it heats the atmosphere while also absorbing harmful radiation from the sun."], ['what is the difference between ground level ozone and the ozone layer?', "['Water vapor (H. 2O)', 'Carbon dioxide (CO. ... ', 'Methane (CH. ... ', 'Nitrous oxide (N. 2O)', 'Ozone (O. ... ', 'Chlorofluorocarbons (CFCs)', 'Hydrofluorocarbons (includes HCFCs and HFCs)']"], ['what is the difference between ground level ozone and the ozone layer?', "Gases in the atmosphere, such as carbon dioxide, trap heat just like the glass roof of a greenhouse. These heat-trapping gases are called greenhouse gases. During the day, the Sun shines through the atmosphere. Earth's surface warms up in the sunlight."], ] scores = model.predict(pairs) print(scores.shape) # (5,) # Or rank different texts based on similarity to a single text ranks = model.rank( 'what is the difference between ground level ozone and the ozone layer?', [ 'Here, ground-level or "bad" ozone is an air pollutant that is harmful to breathe and it damages crops, trees and other vegetation. ... The stratosphere or "good" ozone layer extends upward from about 6 to 30 miles and protects life on Earth from the sun\'s harmful ultraviolet (UV) rays.', 'In the stratosphere, temperature increases with altitude. The reason is that the direct heat source for the stratosphere is the Sun. A layer of ozone molecules absorbs solar radiation, which heats the stratosphere.', "Atmosphere layers. Earth's atmosphere is divided into five main layers: the exosphere, the thermosphere, the mesosphere, the stratosphere and the troposphere. ... Ozone is abundant here and it heats the atmosphere while also absorbing harmful radiation from the sun.", "['Water vapor (H. 2O)', 'Carbon dioxide (CO. ... ', 'Methane (CH. ... ', 'Nitrous oxide (N. 2O)', 'Ozone (O. ... ', 'Chlorofluorocarbons (CFCs)', 'Hydrofluorocarbons (includes HCFCs and HFCs)']", "Gases in the atmosphere, such as carbon dioxide, trap heat just like the glass roof of a greenhouse. These heat-trapping gases are called greenhouse gases. During the day, the Sun shines through the atmosphere. Earth's surface warms up in the sunlight.", ] ) # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...] ``` ## Evaluation ### Metrics #### Cross Encoder Reranking * Dataset: `gooaq-dev` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": false } ``` | Metric | Value | |:------------|:---------------------| | map | 0.7089 (+0.1778) | | mrr@10 | 0.7076 (+0.1836) | | **ndcg@10** | **0.7550 (+0.1637)** | #### Cross Encoder Reranking * Datasets: `NanoMSMARCO_R100`, `NanoNFCorpus_R100` and `NanoNQ_R100` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": true } ``` | Metric | NanoMSMARCO_R100 | NanoNFCorpus_R100 | NanoNQ_R100 | |:------------|:---------------------|:---------------------|:---------------------| | map | 0.5540 (+0.0644) | 0.3421 (+0.0811) | 0.6312 (+0.2116) | | mrr@10 | 0.5472 (+0.0697) | 0.5284 (+0.0286) | 0.6380 (+0.2113) | | **ndcg@10** | **0.6229 (+0.0825)** | **0.3792 (+0.0541)** | **0.6915 (+0.1908)** | #### Cross Encoder Nano BEIR * Dataset: `NanoBEIR_R100_mean` * Evaluated with [CrossEncoderNanoBEIREvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "msmarco", "nfcorpus", "nq" ], "rerank_k": 100, "at_k": 10, "always_rerank_positives": true } ``` | Metric | Value | |:------------|:---------------------| | map | 0.5091 (+0.1190) | | mrr@10 | 0.5712 (+0.1032) | | **ndcg@10** | **0.5645 (+0.1092)** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 482,388 training samples * Columns: question, answer, and label * Approximate statistics based on the first 1000 samples: | | question | answer | label | |:--------|:----------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | question | answer | label | |:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | what is the difference between ground level ozone and the ozone layer? | Here, ground-level or "bad" ozone is an air pollutant that is harmful to breathe and it damages crops, trees and other vegetation. ... The stratosphere or "good" ozone layer extends upward from about 6 to 30 miles and protects life on Earth from the sun's harmful ultraviolet (UV) rays. | 1 | | what is the difference between ground level ozone and the ozone layer? | In the stratosphere, temperature increases with altitude. The reason is that the direct heat source for the stratosphere is the Sun. A layer of ozone molecules absorbs solar radiation, which heats the stratosphere. | 0 | | what is the difference between ground level ozone and the ozone layer? | Atmosphere layers. Earth's atmosphere is divided into five main layers: the exosphere, the thermosphere, the mesosphere, the stratosphere and the troposphere. ... Ozone is abundant here and it heats the atmosphere while also absorbing harmful radiation from the sun. | 0 | * Loss: [BinaryCrossEntropyLoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters: ```json { "activation_fct": "torch.nn.modules.linear.Identity", "pos_weight": 5 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 64 - `per_device_eval_batch_size`: 64 - `learning_rate`: 2e-05 - `num_train_epochs`: 1 - `warmup_ratio`: 0.1 - `seed`: 12 - `bf16`: True - `dataloader_num_workers`: 4 - `load_best_model_at_end`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 64 - `per_device_eval_batch_size`: 64 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 12 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 4 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | gooaq-dev_ndcg@10 | NanoMSMARCO_R100_ndcg@10 | NanoNFCorpus_R100_ndcg@10 | NanoNQ_R100_ndcg@10 | NanoBEIR_R100_mean_ndcg@10 | |:----------:|:--------:|:-------------:|:--------------------:|:------------------------:|:-------------------------:|:--------------------:|:--------------------------:| | -1 | -1 | - | 0.1488 (-0.4424) | 0.0573 (-0.4832) | 0.2647 (-0.0604) | 0.0388 (-0.4619) | 0.1202 (-0.3351) | | 0.0001 | 1 | 1.3143 | - | - | - | - | - | | 0.0265 | 200 | 1.2539 | - | - | - | - | - | | 0.0531 | 400 | 0.9497 | - | - | - | - | - | | 0.0796 | 600 | 0.5613 | - | - | - | - | - | | 0.1061 | 800 | 0.4687 | - | - | - | - | - | | 0.1327 | 1000 | 0.4042 | 0.7103 (+0.1191) | 0.5262 (-0.0142) | 0.3298 (+0.0048) | 0.5589 (+0.0583) | 0.4717 (+0.0163) | | 0.1592 | 1200 | 0.3562 | - | - | - | - | - | | 0.1857 | 1400 | 0.3543 | - | - | - | - | - | | 0.2123 | 1600 | 0.3467 | - | - | - | - | - | | 0.2388 | 1800 | 0.3153 | - | - | - | - | - | | 0.2653 | 2000 | 0.3033 | 0.7317 (+0.1405) | 0.5662 (+0.0258) | 0.3859 (+0.0609) | 0.6828 (+0.1822) | 0.5450 (+0.0896) | | 0.2919 | 2200 | 0.2986 | - | - | - | - | - | | 0.3184 | 2400 | 0.3016 | - | - | - | - | - | | 0.3449 | 2600 | 0.2984 | - | - | - | - | - | | 0.3715 | 2800 | 0.2646 | - | - | - | - | - | | 0.3980 | 3000 | 0.3048 | 0.7359 (+0.1447) | 0.5713 (+0.0309) | 0.3987 (+0.0736) | 0.6960 (+0.1953) | 0.5553 (+0.1000) | | 0.4245 | 3200 | 0.2714 | - | - | - | - | - | | 0.4510 | 3400 | 0.2773 | - | - | - | - | - | | 0.4776 | 3600 | 0.2621 | - | - | - | - | - | | 0.5041 | 3800 | 0.2529 | - | - | - | - | - | | 0.5306 | 4000 | 0.2533 | 0.7459 (+0.1546) | 0.5893 (+0.0489) | 0.3887 (+0.0637) | 0.6749 (+0.1743) | 0.5510 (+0.0956) | | 0.5572 | 4200 | 0.2822 | - | - | - | - | - | | 0.5837 | 4400 | 0.2299 | - | - | - | - | - | | 0.6102 | 4600 | 0.2554 | - | - | - | - | - | | 0.6368 | 4800 | 0.2373 | - | - | - | - | - | | 0.6633 | 5000 | 0.2248 | 0.7497 (+0.1584) | 0.6110 (+0.0706) | 0.3782 (+0.0531) | 0.6885 (+0.1878) | 0.5592 (+0.1038) | | 0.6898 | 5200 | 0.2315 | - | - | - | - | - | | 0.7164 | 5400 | 0.2313 | - | - | - | - | - | | 0.7429 | 5600 | 0.2294 | - | - | - | - | - | | 0.7694 | 5800 | 0.2384 | - | - | - | - | - | | 0.7960 | 6000 | 0.2195 | 0.7530 (+0.1617) | 0.6249 (+0.0845) | 0.3873 (+0.0623) | 0.6773 (+0.1766) | 0.5632 (+0.1078) | | 0.8225 | 6200 | 0.2047 | - | - | - | - | - | | 0.8490 | 6400 | 0.2192 | - | - | - | - | - | | 0.8756 | 6600 | 0.1926 | - | - | - | - | - | | 0.9021 | 6800 | 0.2185 | - | - | - | - | - | | **0.9286** | **7000** | **0.2365** | **0.7550 (+0.1637)** | **0.6229 (+0.0825)** | **0.3792 (+0.0541)** | **0.6915 (+0.1908)** | **0.5645 (+0.1092)** | | 0.9552 | 7200 | 0.2173 | - | - | - | - | - | | 0.9817 | 7400 | 0.2249 | - | - | - | - | - | | -1 | -1 | - | 0.7550 (+0.1637) | 0.6229 (+0.0825) | 0.3792 (+0.0541) | 0.6915 (+0.1908) | 0.5645 (+0.1092) | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.11.10 - Sentence Transformers: 3.5.0.dev0 - Transformers: 4.49.0 - PyTorch: 2.5.1+cu124 - Accelerate: 1.5.2 - Datasets: 2.21.0 - Tokenizers: 0.21.0 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```