tomaarsen/splade-distilbert-base-uncased-gooaq

splade-distilbert-base-uncased trained on GooAQ

This is a SPLADE Sparse Encoder model finetuned from distilbert/distilbert-base-uncased on the gooaq dataset using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Training Dataset:
  • gooaq
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("tomaarsen/splade-distilbert-base-uncased-gooaq")
# Run inference
sentences = [
    'what is the difference between 18 and 20 inch tires?',
    'The only real difference is a 20" rim would be more likely to be damaged, as you pointed out. Beyond looks, there is zero benefit for the 20" rim. Also, just the availability of tires will likely be much more limited for the larger rim. ... Tire selection is better for 18" wheels than 20" wheels.',
    'So extracurricular activities are just activities that you do outside of class. The Common App says that extracurricular activities "include arts, athletics, clubs, employment, personal commitments, and other pursuits."',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# (3, 30522)

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Sparse Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with SparseInformationRetrievalEvaluator

Metric	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoFiQA2018	NanoHotpotQA	NanoMSMARCO	NanoNFCorpus	NanoNQ	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
dot_accuracy@1	0.22	0.52	0.68	0.34	0.74	0.26	0.4	0.38	0.82	0.38	0.1	0.54	0.6531
dot_accuracy@3	0.38	0.68	0.9	0.6	0.9	0.42	0.56	0.6	0.96	0.56	0.48	0.66	0.898
dot_accuracy@5	0.48	0.72	0.92	0.6	0.92	0.56	0.62	0.7	0.96	0.62	0.58	0.72	0.898
dot_accuracy@10	0.54	0.86	0.94	0.68	0.96	0.78	0.66	0.82	1.0	0.76	0.72	0.78	0.9796
dot_precision@1	0.22	0.52	0.68	0.34	0.74	0.26	0.4	0.38	0.82	0.38	0.1	0.54	0.6531
dot_precision@3	0.1333	0.4267	0.3067	0.2733	0.4467	0.14	0.34	0.2133	0.3733	0.24	0.16	0.2333	0.6327
dot_precision@5	0.108	0.396	0.188	0.192	0.3	0.112	0.316	0.148	0.236	0.188	0.116	0.156	0.5347
dot_precision@10	0.066	0.36	0.1	0.12	0.158	0.078	0.25	0.086	0.132	0.148	0.072	0.088	0.4612
dot_recall@1	0.1167	0.0417	0.6567	0.1459	0.37	0.26	0.0237	0.36	0.734	0.0807	0.1	0.505	0.0452
dot_recall@3	0.1707	0.1169	0.8567	0.3665	0.67	0.42	0.0767	0.57	0.9113	0.1497	0.48	0.635	0.1317
dot_recall@5	0.2157	0.1515	0.8767	0.3961	0.75	0.56	0.0965	0.66	0.922	0.1947	0.58	0.69	0.1831
dot_recall@10	0.254	0.2449	0.9167	0.5118	0.79	0.78	0.1197	0.77	0.9833	0.3037	0.72	0.77	0.2997
dot_ndcg@10	0.2262	0.4348	0.8076	0.403	0.7355	0.4857	0.3007	0.5619	0.9062	0.2913	0.4188	0.6443	0.5207
dot_mrr@10	0.3172	0.6113	0.7872	0.4699	0.8237	0.3958	0.4807	0.5104	0.8922	0.4865	0.3219	0.616	0.7655
dot_map@100	0.1814	0.3308	0.7674	0.3255	0.6711	0.4043	0.13	0.4968	0.8722	0.2168	0.3358	0.6035	0.3842

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ]
  }
  

Metric	Value
dot_accuracy@1	0.4641
dot_accuracy@3	0.6614
dot_accuracy@5	0.7152
dot_accuracy@10	0.8061
dot_precision@1	0.4641
dot_precision@3	0.3015
dot_precision@5	0.2301
dot_precision@10	0.163
dot_recall@1	0.2646
dot_recall@3	0.4273
dot_recall@5	0.4828
dot_recall@10	0.5741
dot_ndcg@10	0.5182
dot_mrr@10	0.5753
dot_map@100	0.44

Training Details

Training Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 3,011,496 training samples
• Columns: question and answer
• Approximate statistics based on the first 1000 samples:

  	question	answer
  type	string	string
  details	min: 18 characters mean: 43.42 characters max: 96 characters	min: 54 characters mean: 252.96 characters max: 426 characters

• Samples:

  question	answer
  what is the difference between clay and mud mask?	The main difference between the two is that mud is a skin-healing agent, while clay is a cosmetic, drying agent. Clay masks are most useful for someone who has oily skin and is prone to breakouts of acne and blemishes.
  myki how much on card?	A full fare myki card costs $6 and a concession, seniors or child myki costs $3. For more information about how to use your myki, visit ptv.vic.gov.au or call 1800 800 007.
  how to find out if someone blocked your phone number on iphone?	If you get a notification like "Message Not Delivered" or you get no notification at all, that's a sign of a potential block. Next, you could try calling the person. If the call goes right to voicemail or rings once (or a half ring) then goes to voicemail, that's further evidence you may have been blocked.

• Loss: SpladeLoss with these parameters:

  {'loss': SparseMultipleNegativesRankingLoss(
    (model): SparseEncoder(
      (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
      (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': None})
    )
    (cross_entropy_loss): CrossEntropyLoss()
  ), 'lambda_corpus': 3e-05, 'lambda_query': 5e-05, 'corpus_regularizer': FlopsLoss(
    (model): SparseEncoder(
      (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
      (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': None})
    )
  ), 'query_regularizer': FlopsLoss(
    (model): SparseEncoder(
      (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
      (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': None})
    )
  )}
  

Evaluation Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 1,000 evaluation samples
• Columns: question and answer
• Approximate statistics based on the first 1000 samples:

  	question	answer
  type	string	string
  details	min: 18 characters mean: 43.17 characters max: 98 characters	min: 51 characters mean: 254.12 characters max: 360 characters

• Samples:

  question	answer
  how do i program my directv remote with my tv?	['Press MENU on your remote.', 'Select Settings & Help > Settings > Remote Control > Program Remote.', 'Choose the device (TV, audio, DVD) you wish to program. ... ', 'Follow the on-screen prompts to complete programming.']
  are rodrigues fruit bats nocturnal?	Before its numbers were threatened by habitat destruction, storms, and hunting, some of those groups could number 500 or more members. Sunrise, sunset. Rodrigues fruit bats are most active at dawn, at dusk, and at night.
  why does your heart rate increase during exercise bbc bitesize?	During exercise there is an increase in physical activity and muscle cells respire more than they do when the body is at rest. The heart rate increases during exercise. The rate and depth of breathing increases - this makes sure that more oxygen is absorbed into the blood, and more carbon dioxide is removed from it.

• Loss: SpladeLoss with these parameters:

  {'loss': SparseMultipleNegativesRankingLoss(
    (model): SparseEncoder(
      (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
      (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': None})
    )
    (cross_entropy_loss): CrossEntropyLoss()
  ), 'lambda_corpus': 3e-05, 'lambda_query': 5e-05, 'corpus_regularizer': FlopsLoss(
    (model): SparseEncoder(
      (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
      (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': None})
    )
  ), 'query_regularizer': FlopsLoss(
    (model): SparseEncoder(
      (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
      (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': None})
    )
  )}
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 1
• bf16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoClimateFEVER_dot_ndcg@10	NanoDBPedia_dot_ndcg@10	NanoFEVER_dot_ndcg@10	NanoFiQA2018_dot_ndcg@10	NanoHotpotQA_dot_ndcg@10	NanoMSMARCO_dot_ndcg@10	NanoNFCorpus_dot_ndcg@10	NanoNQ_dot_ndcg@10	NanoQuoraRetrieval_dot_ndcg@10	NanoSCIDOCS_dot_ndcg@10	NanoArguAna_dot_ndcg@10	NanoSciFact_dot_ndcg@10	NanoTouche2020_dot_ndcg@10	NanoBEIR_mean_dot_ndcg@10
0.0213	2000	0.75	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0425	4000	0.071	0.0924	0.1931	0.2903	0.5966	0.3079	0.6182	0.3378	0.1867	0.3781	0.3784	0.1966	0.2325	0.4148	0.5139	0.3573
0.0638	6000	0.0578	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0850	8000	0.0511	0.0589	0.1826	0.2911	0.5719	0.3820	0.6818	0.2417	0.2032	0.2925	0.4541	0.2090	0.2306	0.5240	0.5183	0.3679
0.1063	10000	0.0464	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1275	12000	0.0458	0.0795	0.1978	0.2958	0.6206	0.3664	0.6673	0.2691	0.1872	0.2327	0.6770	0.2008	0.3288	0.5384	0.5017	0.3911
0.1488	14000	0.0427	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1700	16000	0.0392	0.0581	0.2785	0.4104	0.8125	0.3832	0.7265	0.5093	0.2688	0.6075	0.7879	0.2760	0.3342	0.5722	0.5301	0.4998
0.1913	18000	0.039	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2125	20000	0.0366	0.0472	0.2319	0.3466	0.7349	0.3774	0.7174	0.4061	0.2189	0.4166	0.7486	0.2364	0.3560	0.5907	0.5211	0.4541
0.2338	22000	0.0312	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2550	24000	0.0322	0.0543	0.2169	0.4469	0.7618	0.4014	0.6831	0.4412	0.2707	0.5253	0.8104	0.2621	0.3581	0.6006	0.5037	0.4832
0.2763	26000	0.0292	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2975	28000	0.03	0.0529	0.2257	0.5070	0.7976	0.4014	0.7442	0.5165	0.3216	0.5799	0.8483	0.3318	0.3206	0.5665	0.5149	0.5135
0.3188	30000	0.0294	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3400	32000	0.0291	0.0512	0.1754	0.4539	0.8196	0.3903	0.7372	0.4689	0.2948	0.5548	0.8643	0.2791	0.4040	0.5229	0.5055	0.4977
0.3613	34000	0.0284	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3825	36000	0.0268	0.0404	0.2566	0.4462	0.8142	0.3737	0.7281	0.4418	0.2568	0.5135	0.8305	0.2749	0.3775	0.5485	0.5228	0.4912
0.4038	38000	0.0262	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4250	40000	0.0238	0.0416	0.2464	0.5235	0.8004	0.4016	0.7418	0.4483	0.2915	0.5771	0.8538	0.2523	0.3536	0.6227	0.4967	0.5084
0.4463	42000	0.0253	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4675	44000	0.0224	0.0360	0.2080	0.5100	0.8317	0.3775	0.7223	0.4447	0.2789	0.5586	0.8324	0.3151	0.4005	0.6089	0.5119	0.5077
0.4888	46000	0.0225	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5100	48000	0.0209	0.0232	0.2386	0.5045	0.8204	0.3746	0.7390	0.4662	0.2963	0.5380	0.8580	0.3292	0.4010	0.6336	0.5214	0.5170
0.5313	50000	0.0225	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5525	52000	0.0205	0.0380	0.2237	0.5114	0.7952	0.3583	0.6979	0.4310	0.2816	0.5364	0.8747	0.2703	0.4009	0.5947	0.5038	0.4984
0.5738	54000	0.0204	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5950	56000	0.0204	0.0365	0.2316	0.4676	0.8134	0.3754	0.7280	0.4536	0.2927	0.5205	0.8662	0.2859	0.3589	0.6281	0.5069	0.5022
0.6163	58000	0.0199	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6376	60000	0.0196	0.0365	0.2233	0.4897	0.8149	0.3385	0.7395	0.4778	0.2725	0.5365	0.8610	0.2836	0.4031	0.5380	0.5146	0.4995
0.6588	62000	0.0187	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6801	64000	0.0184	0.0453	0.2333	0.4792	0.7881	0.3653	0.7402	0.5062	0.3008	0.5607	0.8922	0.2857	0.4039	0.5972	0.5217	0.5134
0.7013	66000	0.0182	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7226	68000	0.0162	0.0323	0.2341	0.4678	0.8283	0.3855	0.7567	0.5229	0.3297	0.5445	0.8909	0.2787	0.3917	0.5904	0.5115	0.5179
0.7438	70000	0.0195	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7651	72000	0.0181	0.0243	0.2082	0.4374	0.7487	0.4010	0.7245	0.4712	0.3179	0.5168	0.8721	0.2794	0.4312	0.5801	0.5129	0.5001
0.7863	74000	0.0171	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8076	76000	0.0164	0.0284	0.2153	0.4654	0.7985	0.4027	0.7528	0.4871	0.3267	0.5385	0.9092	0.2997	0.3852	0.5979	0.5001	0.5138
0.8288	78000	0.0169	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8501	80000	0.0168	0.0244	0.2032	0.4466	0.7855	0.4042	0.7396	0.4971	0.2946	0.5485	0.9071	0.2983	0.3919	0.5862	0.5149	0.5091
0.8713	82000	0.0144	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8926	84000	0.0155	0.0229	0.2262	0.4348	0.8076	0.403	0.7355	0.4857	0.3007	0.5619	0.9062	0.2913	0.4188	0.6443	0.5207	0.5182
0.9138	86000	0.0144	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9351	88000	0.0149	0.0211	0.2259	0.4212	0.8098	0.3938	0.7309	0.4665	0.3051	0.5301	0.9061	0.2881	0.4086	0.6390	0.5199	0.5111
0.9563	90000	0.013	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9776	92000	0.0143	0.0231	0.2288	0.4224	0.8176	0.4130	0.7332	0.4807	0.3033	0.5424	0.9007	0.2772	0.4215	0.6354	0.5170	0.5149
0.9988	94000	0.0129	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.2262	0.4348	0.8076	0.4030	0.7355	0.4857	0.3007	0.5619	0.9062	0.2913	0.4188	0.6443	0.5207	0.5182

• The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Energy Consumed: 1.338 kWh
• Carbon Emitted: 0.520 kg of CO2
• Hours Used: 3.894 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3090
• CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM Size: 31.78 GB

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 4.2.0.dev0
• Transformers: 4.49.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.1
• Datasets: 2.21.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}