rasyosef/splade-small

SPLADE-BERT-Small-Distil

This is a SPLADE sparse retrieval model based on BERT-Small (29M) that was trained by distilling a Cross-Encoder on the MSMARCO dataset. The cross-encoder used was ms-marco-MiniLM-L6-v2.

This SPLADE model is 2x smaller than Naver's official splade-v3-distilbert while having 91% of it's performance on the MSMARCO benchmark. This model is small enough to be used without a GPU on a dataset of a few thousand documents.

• Collection: https://huggingface.co/collections/rasyosef/splade-tiny-msmarco-687c548c0691d95babf65b70
• Distillation Dataset: https://huggingface.co/datasets/yosefw/msmarco-train-distil-v2
• Code: https://github.com/rasyosef/splade-tiny-msmarco

Performance

The splade models were evaluated on 55 thousand queries and 8.84 million documents from the MSMARCO dataset.

	Size (# Params)	MRR@10 (MS MARCO dev)
BM25	-	18.0
rasyosef/splade-tiny	4.4M	30.9
rasyosef/splade-mini	11.2M	34.1
rasyosef/splade-small	28.8M	35.4
naver/splade-v3-distilbert	67.0M	38.7

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("rasyosef/splade-small")
# Run inference
queries = [
    "how many tablespoons of garlic powder are in an ounce",
]
documents = [
    '1 Fluid Ounce (fl oz) = 2 tablespoons   16 Tablespoons = 1 cup   16 Fluid Ounce (fl oz) = 2 cup. two ! 9.7 grams of garlic powder will be present in a tablespoon. 1 dry ounce is between 2 and 2.38 tablespoons, 16 tablespoons is incorrect. --------------------- 16 tablespoons per dry ounce. It is approximately 1/2 ounce. Usually 1/4 to 1/2 tsp.',
    'Spices, garlic powder weigh(s) 164 gram per (metric cup) or 5.47 ounce per (US cup)',
    'How many teaspoons of garlic powder equal a clove of garlic? Weigh the garlic clove and then weigh the garlic powder to make sure it is the same weight. That is how much powder equals a clove of garlic.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[26.3104, 20.4381, 15.5539]])

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: prajjwal1/bert-small
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

More

Click to expand

Evaluation

Metrics

Sparse Information Retrieval

• Evaluated with SparseInformationRetrievalEvaluator

Metric	Value
dot_accuracy@1	0.4547
dot_accuracy@3	0.7685
dot_accuracy@5	0.8786
dot_accuracy@10	0.9484
dot_precision@1	0.4547
dot_precision@3	0.2634
dot_precision@5	0.1828
dot_precision@10	0.0998
dot_recall@1	0.44
dot_recall@3	0.7543
dot_recall@5	0.8678
dot_recall@10	0.9424
dot_ndcg@10	0.7031
dot_mrr@10	0.6288
dot_map@100	0.6253
query_active_dims	24.9142
query_sparsity_ratio	0.9992
corpus_active_dims	171.8592
corpus_sparsity_ratio	0.9944

Training Details

Training Dataset

Unnamed Dataset

• Size: 800,000 training samples
• Columns: query, positive, negative_1, negative_2, negative_3, negative_4, and label
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative_1	negative_2	negative_3	negative_4	label
  type	string	string	string	string	string	string	list
  details	min: 4 tokens mean: 8.96 tokens max: 37 tokens	min: 19 tokens mean: 79.51 tokens max: 230 tokens	min: 22 tokens mean: 78.09 tokens max: 203 tokens	min: 14 tokens mean: 77.84 tokens max: 215 tokens	min: 15 tokens mean: 76.65 tokens max: 249 tokens	min: 19 tokens mean: 74.67 tokens max: 227 tokens	size: 4 elements

• Samples:

  query	positive	negative_1	negative_2	negative_3	negative_4	label
  who was president during detente	Détente ended after the Soviet intervention in Afghanistan, which led to the United States boycott of the 1980 Olympics in Moscow. Ronald Reagan's election as president in 1980, based in large part on an anti-détente campaign, marked the close of détente and a return to Cold War tensions.	Soviet Premier Alexei Kosygin (front) next to U.S. President Lyndon B. Johnson (behind) during the Glassboro Summit Conference The most obvious manifestation of détente was the series of summits held between the leaders of the two superpowers and the treaties that resulted from these meetings.	The activities of President Ronald Reagan returned tensions to a fever pitch. Soviet relations with the People's Republic of China Détente could probably not have taken place, and certainly wouldn't have assumed the form that it did, without the rift that developed between the world's two primary communist regimes, the Soviet Union and the People's Republic of China (PRC).	Détente is the easing of strained relations, especially in a political situation. The term originates in the time of the Triple Entente and Entente cordiale in reference to an easing of tensions between England and France who, subsequent to being commingled polities under Norman rule, were warring rivals for the better part of a millennium but pursuant to a policy of détente became enduring allies. In the context of the Cold War, the lessening of tensions between the East and West, along ...	Détente (French pronunciation: ​[detɑ̃t], meaning relaxation)[1] is the easing of strained relations, especially in political situation.	[1.0, 2.1879749298095703, 8.371654510498047, 10.16702938079834]
  what is an ftp file	File Transfer Protocol (FTP) is a standard Internet protocol for transmitting files between computers on the Internet over TCP/IP connections. FTP is a client-server protocol that relies on two communications channels between client and server: a command channel for controlling the conversation and a data channel for transmitting file content. Clients initiate conversations with servers by requesting to download a file.	The FTP (File Transfer Protocol) utility program is commonly used for copying files to and from other computers. These computers may be at the same site or at different sites thousands of miles apart. FTP is a general protocol that works on UNIX systems as well as a variety of other (non-UNIX) systems.	To transfer files via File Transfer Protocol (FTP), you need to establish an FTP connection. To make an FTP connection you can use a standard Web browser (Internet Explorer, Mozilla Firefox, etc.) or an FTP Client. To transfer a file with FTP you need to have an FTP accounts for the web space you are going to transfer the file to. FTP hosting account where you plan to upload your files.	The Difference Between FTP Servers and File Servers. When two terms are similar and even describe a similar concept, people have a tendency to start using them interchangeably. This is definitely true in the case of FTP servers and file servers, which sound like they accomplish the same goal but in reality are two very different animals altogether.	The command-line secure file transfer program (sftp) and graphical SFTP clients, such as WinSCP and Fetch, use SSH2 encryption to authenticate and establish secure channels between networked hosts.	[1.0, 5.810799598693848, 7.961757183074951, 18.629709243774414]
  what causes a t wave abnormality	T –wave abnormalities may not necessarily indicate the presence of a severe heart condition. There are non-specific wave changes that result from common, non-specific causes of T-wave abnormality which includes the following: 1 No obvious causes, which are usually associated with women. Fever.	5 Causes Of T-Wave Abnormality. T wave is basically the diagrammatically representation of ventricular polarization called electrocardiography. The structure of a T wave is like slight inverted upward stroke that follow a peak generated by R and S waves. One heart beat is represented in form of Q, R, S and T wave. The abnormality in T waves may be indicated by longer, flatter or higher peaks in the diagram. The measurement of heart beat in such a way makes it possible to diagnose heart related problems easily. If you are suffering from any heart related disease, the electrocardiogram will show an abnormality in the T wave. In this write up we will discuss various causes of abnormality in T wave measurement.	Specific states or conditions that cause T-wave abnormality. Complete inversions can signify the presence of cardiovascular diseases and other serious complications, which include the following: Ischemia is a condition in which oxygenated blood becomes constrained in a certain body part.	T Wave Abnormalities. christine m. smith. I just received a copy of an ECG I had done in Sept 1998. The preliminary report was borderline ECG and the final interpretation was Abnormal ECG. There was normal sinus rhythm and Nonspecific Anterior T abnormalities. When compared to an ECG taken in 1986 there was minimal T wave change. I have been told by many that abnormalities like this are usually no problem.	Prolonged Q-T interval. Long QT syndrome is a heart rhythm disorder that can cause serious irregular heart rhythms (arrhythmias). In a normal heart, your heart circulates blood throughout your body during each heartbeat. Your heart's chambers contract and relax to pump blood.	[1.0, 1.0, 5.3564229011535645, 11.585516929626465]

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMarginMSELoss",
      "document_regularizer_weight": 0.12,
      "query_regularizer_weight": 0.2
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 48
• per_device_eval_batch_size: 48
• learning_rate: 4e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.025
• fp16: True
• load_best_model_at_end: True
• push_to_hub: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 48
• per_device_eval_batch_size: 48
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 4e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.025
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dot_ndcg@10
1.0	16667	8.363	0.6961
2.0	33334	6.5021	0.7031
3.0	50001	5.2209	0.7031

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.0
• Transformers: 4.55.3
• PyTorch: 2.8.0+cu126
• Accelerate: 1.10.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}