rahulseetharaman/reranker-msmarco-v1.1-ettin-encoder-17m-listnet

CrossEncoder based on jhu-clsp/ettin-encoder-17m

This is a Cross Encoder model finetuned from jhu-clsp/ettin-encoder-17m on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: jhu-clsp/ettin-encoder-17m
• Maximum Sequence Length: 7999 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("rahulseetharaman/reranker-msmarco-v1.1-ettin-encoder-17m-listnet")
# Get scores for pairs of texts
pairs = [
    ['what are jellyfishes enemies', 'Jellyfish enemies are sea stars and sea turtles. They are favorite meals to them. Other species of jellyfish are among the most common and important jellyfish predators, some of which specialize in jellies. Other predators include tuna, shark, swordfish, sea turtles, and at least one species of Pacific salmon.'],
    ['what are jellyfishes enemies', 'Other species of jellyfish are among the most common and important  jellyfish predators, some of which specialize in jellies. Other  predators include tuna, shark, swordfish … , sea turtles, and at least  one species of Pacific salmon.'],
    ['what are jellyfishes enemies', 'The jellyfish mainly feeds on the zooplankton, snails, small fishes and larvae and eggs of other marine animals. It catches its prey with its tentacles, which has a venom to immobilise them.'],
    ['what are jellyfishes enemies', 'There are many kinds, or species, of jellyfish in all the oceans of the earth. The main predator of jellyfish is other jellyfish, usually of a different species. But jellyfish also have a number of other natural enemies that like to eat them. These predators include tunas, sharks, swordfish and some species of salmon. Sea turtles also like to eat jellyfish.'],
    ['what are jellyfishes enemies', 'Jellyfish Enemies. The jellyfish is a strange creature inhabiting the oceans of the world. It is fascinating to watch a jellyfish swim in the sea and people are trying to breed them in home aquariums and tanks. The jellyfish is a delicate creature and will just collapse if taken out of the water. The different species are known to survive in the ocean at all depths, and in different water temperatures.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'what are jellyfishes enemies',
    [
        'Jellyfish enemies are sea stars and sea turtles. They are favorite meals to them. Other species of jellyfish are among the most common and important jellyfish predators, some of which specialize in jellies. Other predators include tuna, shark, swordfish, sea turtles, and at least one species of Pacific salmon.',
        'Other species of jellyfish are among the most common and important  jellyfish predators, some of which specialize in jellies. Other  predators include tuna, shark, swordfish … , sea turtles, and at least  one species of Pacific salmon.',
        'The jellyfish mainly feeds on the zooplankton, snails, small fishes and larvae and eggs of other marine animals. It catches its prey with its tentacles, which has a venom to immobilise them.',
        'There are many kinds, or species, of jellyfish in all the oceans of the earth. The main predator of jellyfish is other jellyfish, usually of a different species. But jellyfish also have a number of other natural enemies that like to eat them. These predators include tunas, sharks, swordfish and some species of salmon. Sea turtles also like to eat jellyfish.',
        'Jellyfish Enemies. The jellyfish is a strange creature inhabiting the oceans of the world. It is fascinating to watch a jellyfish swim in the sea and people are trying to breed them in home aquariums and tanks. The jellyfish is a delicate creature and will just collapse if taken out of the water. The different species are known to survive in the ocean at all depths, and in different water temperatures.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	NanoMSMARCO_R100	NanoNFCorpus_R100	NanoNQ_R100
map	0.3623 (-0.1273)	0.2789 (+0.0179)	0.2369 (-0.1827)
mrr@10	0.3446 (-0.1329)	0.4065 (-0.0933)	0.2211 (-0.2056)
ndcg@10	0.4090 (-0.1314)	0.2520 (-0.0731)	0.2731 (-0.2276)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Metric	Value
map	0.2927 (-0.0974)
mrr@10	0.3241 (-0.1439)
ndcg@10	0.3113 (-0.1440)

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 39,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 11 characters mean: 33.65 characters max: 96 characters	min: 2 elements mean: 6.00 elements max: 10 elements	min: 2 elements mean: 6.00 elements max: 10 elements

• Samples:

  query	docs	labels
  what is PMA sales	['A history of helping people create great futures. PMA USA, headquartered in Dallas, Texas is a national company that provides insurance benefits solutions and markets voluntary insurance products. PMA USA has been helping hard-working middle Americans protect their families’ financial futures since 1999.', 'PMA is the full-service trade association representing the $113-billion metalforming industry of North America―the industry that creates precision metal products using stamping, fabricating and other value-added processes.', 'PMA USA, headquartered in Dallas, Texas is a national company that provides insurance benefits solutions and markets voluntary insurance products. We offer supplemental health, accident and life insurance, serving individual customers and employer groups across the country.', 'Who we arePMA USA, headquartered in Dallas, Texas is a national company that provides insurance benefits solutions and markets voluntary insurance products.', 'Who we are. PMA USA, head...	[1, 1, 0, 0, 0, ...]
  what is whitebait	['Whitebait is the immature fry of fish, in this case sardines and anchovies fished on the Riviera. Whitebait is a collective term for the immature fry of fish, typically between 25 and 50 millimetres long. Such young fish often travel together in schools along the coast, and move into estuaries and sometimes up rivers where they can be easily caught with fine meshed fishing nets. Whitebait are tender and edible, and can be regarded as a delicacy. The entire fish is eaten including head, fins, bones, and guts. Some species make better eating than others, and the particular species that are marketed as whitebait varies in different parts of the world.', 'Whitebait recipes. Whitebait are tiny, immature, silvery members of the herring group that are typically deep-fried to serve. They are widely thought to be baby herring and are usually sold frozen. Preparation. Whitebait require little preparation. Toss them in well-seasoned flour (for devilled whitebait, small quantities of dried Engli...	[1, 0, 0, 0, 0, ...]
  how to disable automatic sign in hotmail	['If you are referring to the Web Messenger in Hotmail, you can definitely turn it off automatically by following the steps below: 1. Sign in your account. 2. On the left pane of the window, click on Sign out of Messenger.. 3. Sign out of Hotmail. 4. Sign in your account again. On the other hand, the option to turn off the Messenger in Outlook.com is not available. Thank you.', "1) Click on the messenger tab at the top. I see my contacts, the option to sign out of messenger (don't get excited it will log you in again the next time you open hotmail), contacts, profile, add friends and invitations. Why isn't the auto sign in option here. Why isn't everything to do with messenger under the messenger button. 1. Sign in your account. 2. On the left pane of the window, click on Sign out of Messenger.. 3. Sign out of Hotmail. 4. Sign in your account again. On the other hand, the option to turn off the Messenger in Outlook.com is not available.", "Disabling automatic sign-in. If you're being...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 40,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 9 characters mean: 34.15 characters max: 144 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  what are jellyfishes enemies	['Jellyfish enemies are sea stars and sea turtles. They are favorite meals to them. Other species of jellyfish are among the most common and important jellyfish predators, some of which specialize in jellies. Other predators include tuna, shark, swordfish, sea turtles, and at least one species of Pacific salmon.', 'Other species of jellyfish are among the most common and important jellyfish predators, some of which specialize in jellies. Other predators include tuna, shark, swordfish … , sea turtles, and at least one species of Pacific salmon.', 'The jellyfish mainly feeds on the zooplankton, snails, small fishes and larvae and eggs of other marine animals. It catches its prey with its tentacles, which has a venom to immobilise them.', 'There are many kinds, or species, of jellyfish in all the oceans of the earth. The main predator of jellyfish is other jellyfish, usually of a different species. But jellyfish also have a number of other natural enemies that like to eat them. These predators include tunas, sharks, swordfish and some species of salmon. Sea turtles also like to eat jellyfish.', 'Jellyfish Enemies. The jellyfish is a strange creature inhabiting the oceans of the world. It is fascinating to watch a jellyfish swim in the sea and people are trying to breed them in home aquariums and tanks. The jellyfish is a delicate creature and will just collapse if taken out of the water. The different species are known to survive in the ocean at all depths, and in different water temperatures.']	[1, 0, 0, 0, 0]
  how much does a medical secretary earn	['$32,670. With an average salary of $33,140 in 2013, medical secretaries earned more than medical assistants ($30,780) and pharmacy technicians ($30,840) but slightly less than emergency medical technicians and paramedics ($34,870). Best Paying Cities for Medical Secretaries. The highest paid in the medical secretary profession work in the metropolitan areas of Oakland, California, San Francisco, and San Jose, California. The New York City area also pays well, as does the city of Norwich, Connecticut.', 'Contributing Factors. A hospital unit secretary may earn more in certain industries. In 2012, medical secretaries, who perform similar duties, earned some of the highest salaries of $40,790 working for state-owned hospitals, according to the BLS -- versus an industry average of $32,676. Hospital unit secretaries may also earn more in state-owned hospitals. Related Reading: Medical Unit Secretary Training. A hospital unit secretary may earn more in certain industries. In 2012, medical ...	[1, 0, 0, 0, 0, ...]
  average apartment costs for college students	['The survey revealed the following: With a 5% student housing fee increase effective 8/1/12, the monthly fee for a RV Phase I two bedroom apartment would be $934 and a similar RV Phase II apartment would be $1013. Average of $973.50/month. ', "1 of 6. Students can't control the price of college tuition, but they have options when it comes to everything else. According to the College Board, students attending a four-year, in-state public institution spend an average of $12,368 per year on housing, books, transportation and other fees. That's more than $5,300 above the average cost of tuition. ", 'The cost of room and board depends on the campus housing and food plans you choose. The College Board reports that the average cost of room and board in 2014–2015 ranged from $9,804 at four-year public schools to $11,188 at private schools. Colleges also provide room and board estimates for living off campus based on typical student costs. The College Board reports the average cost for books a...	[1, 0, 0, 0, 0, ...]

• Loss: ListNetLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	NanoMSMARCO_R100_ndcg@10	NanoNFCorpus_R100_ndcg@10	NanoNQ_R100_ndcg@10	NanoBEIR_R100_mean_ndcg@10
-1	-1	-	0.0063 (-0.5341)	0.1891 (-0.1359)	0.0144 (-0.4863)	0.0699 (-0.3854)
0.0004	1	2.565	-	-	-	-
-1	-1	-	0.4090 (-0.1314)	0.2520 (-0.0731)	0.2731 (-0.2276)	0.3113 (-0.1440)

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.0.0
• Transformers: 4.56.0.dev0
• PyTorch: 2.7.1+cu126
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ListNetLoss

@inproceedings{cao2007learning,
    title={Learning to Rank: From Pairwise Approach to Listwise Approach},
    author={Cao, Zhe and Qin, Tao and Liu, Tie-Yan and Tsai, Ming-Feng and Li, Hang},
    booktitle={Proceedings of the 24th international conference on Machine learning},
    pages={129--136},
    year={2007}
}