--- language: - en license: apache-2.0 tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:20792 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss base_model: kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k widget: - source_sentence: "caregiver, or family member after receiving training in the appropriate\ \ technique.\nVestibular training\nVestibular functions are specific sensory functions\ \ of the inner ear related to position, balance and movement. Vestibular therapy\ \ includes exercises and techniques \nto address symptoms of vestibular dysfunction,\ \ such as dizziness, visual or gaze \ndisturbances and balance disorders. The\ \ exercises and techniques are practised" sentences: - STZ-induced diabetes model for DCM research - vestibular training for family caregivers - duration and frequency of acupuncture sessions for back pain - source_sentence: 'Favours Control FIGURE 8 Forest plot of the effects of exercise training versus control on visceral fat. Data are reported as SMD (95% confidence limits). SMD, standardized mean difference.' sentences: - chronic symptoms and quality of life after traumatic brain injury - supported education intervention in school and university settings - exercise vs control on visceral fat SMD - source_sentence: 'tions that do not appear to have a relationship with exercise. Consider the following case: a 30-year-old female at ABC fit- ness center would like guidance on setting up an aerobic exercise program of moderate intensity. She has a heart rate monitor and is interested in using heart rate to gauge her intensity. Hav- ing completed the health screening questionnaire and other nec- essary documents, she meets with an exercise professional to de- termine what heart rate range would be appropriate. Based on the ACSM Guidelines, 64% to 76% of heart rate max is suggested for moderate intensity (1). An estimated maximal heart rate is calculated based on her age (for simplicity in this example, 220 −age is used; for more information on estimation of maxi- mal heart rate see “Estimating Maximal Heart Rate” (23)). This re- sults in an estimated maximal heart rate of 190 (calculated as 220–30 = 190) and a suggested range of 122–144 beats per minute TABLE: Select Physiologic Responses and Examples of Medication Class Effects (12) Areas of Potential Impact Medication Class' sentences: - IBS and mental health disorders research - ACSM Guidelines for moderate intensity aerobic exercise heart rate range - ginger extract and omega-3 fatty acids supplementation for diabetic cardiomyopathy - source_sentence: "● Overall assessment and training in \nabilities of self-care,\ \ managing the \nchanging environment, returning to \nroles in the family and\ \ community.\n© Handicap International" sentences: - difference between exogenous and endogenous ketones - Bisphosphonates for heterotopic ossification - self-care and community reintegration training - source_sentence: "Kasperczyk A, Kasperczyk S, Vendemiale G. An open-label, single-center\ \ \npilot study to test the effects of an amino acid mixture in older patients\ \ admitted to internal medicine wards. Nutrition. 2020;69:110588.\n\t38.\t Paddon-Jones\ \ D, Rasmussen BB. Dietary protein recommendations and the prevention of sarcopenia.\ \ Curr Opin Clin Nutr Metab Care. \n2009;12(1):86–90. 39.\t MacKenzie-Shalders\ \ KL, King NA, Byrne NM, Slater GJ. Increasing Protein \nDistribution Has No Effect\ \ on Changes in Lean Mass During a Rugby Preseason. Int J Sport Nutr Exerc Metab.\ \ 2016;26(1):1–7. 40.\t Zanini B, Simonetto A, Zubani M, Castellano M, Gilioli\ \ G: The Effects of \nCow-Milk Protein Supplementation in Elderly Population:\ \ Systematic Review and Narrative Synthesis. Nutrients. 2020, 12(9)." sentences: - effect of sitting time on obesity and diabetes - dietary protein recommendations for sarcopenia prevention - alternative treatments for chronic lower back pain in older adults pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: ModernBERT Embed base fitness health Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.5577672003461704 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.5633924707918649 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.5768065772392903 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.6572912159238425 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.5577672003461704 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.5587768642723208 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.5546516659454782 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.488749459108611 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.07671206122845729 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.22954893017730485 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.37230530075054374 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.607515754908188 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.5889008778621518 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.5732042405884898 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6480888290704816 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.5508437905668542 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.5573344872349633 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.5703158805711813 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.6512332323669408 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.5508437905668542 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.552141929900476 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.548334054521852 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.4848117697966249 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.07565244485079678 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.22624747899614794 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.3669823430843505 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.6015669568225397 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.5828389278226814 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.5666920799763724 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6430351850654122 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.5499783643444396 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.5543054954565123 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.5707485936823886 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.6538295110341843 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.5499783643444396 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.5505553151593826 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.5471224578104718 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.4856771960190394 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.07515575798786531 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.22481099241159822 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.36529494112791383 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.6017858069285081 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.5820878299131311 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.5658907021628786 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6428043114206009 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.5196884465599307 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.5222847252271744 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.5348334054521852 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.6205106014712246 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.5196884465599307 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.5196884465599307 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.5151882302033751 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.45867589787970575 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.07059233617000817 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.21080992904878668 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.34166460612804184 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.564437701406452 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.5472355538402842 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.5342628079646684 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.6136096665025225 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.45261791432280396 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.4552141929900476 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.475118996105582 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.5551709216789269 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.45261791432280396 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.4527621520265397 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.45114668974469935 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.4054954565123324 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.06187471238315029 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.18494725021120523 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.3014507909034088 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.5026563777104669 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.4836333347992592 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.46809753903003604 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.5520511526692713 name: Cosine Map@100 --- # ModernBERT Embed base fitness health Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k](https://huggingface.co/kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k) on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k](https://huggingface.co/kokojake/modernbert-embed-base-fitness-health-matryoshka-8-epochs-25k) - **Maximum Sequence Length:** 8192 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - json - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("kokojake/modernbert-embed-base-fitness-health-matryoshka-epoch-15") # Run inference sentences = [ 'Kasperczyk A, Kasperczyk S, Vendemiale G. An open-label, single-center \npilot study to test the effects of an amino acid mixture in older patients admitted to internal medicine wards. Nutrition. 2020;69:110588.\n\t38.\t Paddon-Jones D, Rasmussen BB. Dietary protein recommendations and the prevention of sarcopenia. Curr Opin Clin Nutr Metab Care. \n2009;12(1):86–90. 39.\t MacKenzie-Shalders KL, King NA, Byrne NM, Slater GJ. Increasing Protein \nDistribution Has No Effect on Changes in Lean Mass During a Rugby Preseason. Int J Sport Nutr Exerc Metab. 2016;26(1):1–7. 40.\t Zanini B, Simonetto A, Zubani M, Castellano M, Gilioli G: The Effects of \nCow-Milk Protein Supplementation in Elderly Population: Systematic Review and Narrative Synthesis. Nutrients. 2020, 12(9).', 'dietary protein recommendations for sarcopenia prevention', 'effect of sitting time on obesity and diabetes', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_768` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 768 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.5578 | | cosine_accuracy@3 | 0.5634 | | cosine_accuracy@5 | 0.5768 | | cosine_accuracy@10 | 0.6573 | | cosine_precision@1 | 0.5578 | | cosine_precision@3 | 0.5588 | | cosine_precision@5 | 0.5547 | | cosine_precision@10 | 0.4887 | | cosine_recall@1 | 0.0767 | | cosine_recall@3 | 0.2295 | | cosine_recall@5 | 0.3723 | | cosine_recall@10 | 0.6075 | | **cosine_ndcg@10** | **0.5889** | | cosine_mrr@10 | 0.5732 | | cosine_map@100 | 0.6481 | #### Information Retrieval * Dataset: `dim_512` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 512 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.5508 | | cosine_accuracy@3 | 0.5573 | | cosine_accuracy@5 | 0.5703 | | cosine_accuracy@10 | 0.6512 | | cosine_precision@1 | 0.5508 | | cosine_precision@3 | 0.5521 | | cosine_precision@5 | 0.5483 | | cosine_precision@10 | 0.4848 | | cosine_recall@1 | 0.0757 | | cosine_recall@3 | 0.2262 | | cosine_recall@5 | 0.367 | | cosine_recall@10 | 0.6016 | | **cosine_ndcg@10** | **0.5828** | | cosine_mrr@10 | 0.5667 | | cosine_map@100 | 0.643 | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 256 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.55 | | cosine_accuracy@3 | 0.5543 | | cosine_accuracy@5 | 0.5707 | | cosine_accuracy@10 | 0.6538 | | cosine_precision@1 | 0.55 | | cosine_precision@3 | 0.5506 | | cosine_precision@5 | 0.5471 | | cosine_precision@10 | 0.4857 | | cosine_recall@1 | 0.0752 | | cosine_recall@3 | 0.2248 | | cosine_recall@5 | 0.3653 | | cosine_recall@10 | 0.6018 | | **cosine_ndcg@10** | **0.5821** | | cosine_mrr@10 | 0.5659 | | cosine_map@100 | 0.6428 | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 128 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.5197 | | cosine_accuracy@3 | 0.5223 | | cosine_accuracy@5 | 0.5348 | | cosine_accuracy@10 | 0.6205 | | cosine_precision@1 | 0.5197 | | cosine_precision@3 | 0.5197 | | cosine_precision@5 | 0.5152 | | cosine_precision@10 | 0.4587 | | cosine_recall@1 | 0.0706 | | cosine_recall@3 | 0.2108 | | cosine_recall@5 | 0.3417 | | cosine_recall@10 | 0.5644 | | **cosine_ndcg@10** | **0.5472** | | cosine_mrr@10 | 0.5343 | | cosine_map@100 | 0.6136 | #### Information Retrieval * Dataset: `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 64 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.4526 | | cosine_accuracy@3 | 0.4552 | | cosine_accuracy@5 | 0.4751 | | cosine_accuracy@10 | 0.5552 | | cosine_precision@1 | 0.4526 | | cosine_precision@3 | 0.4528 | | cosine_precision@5 | 0.4511 | | cosine_precision@10 | 0.4055 | | cosine_recall@1 | 0.0619 | | cosine_recall@3 | 0.1849 | | cosine_recall@5 | 0.3015 | | cosine_recall@10 | 0.5027 | | **cosine_ndcg@10** | **0.4836** | | cosine_mrr@10 | 0.4681 | | cosine_map@100 | 0.5521 | ## Training Details ### Training Dataset #### json * Dataset: json * Size: 20,792 training samples * Columns: positive and anchor * Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | positive | anchor | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| | interpretations, if a common framework
like the ICF is used”23, the unit recommends
using the ICF for communications outside
the association, particularly in research
contexts.
Health conditions (disorder or disease)
Activities
© WHO, International Classification of Functioning, Disability and Health, 2001
Participation
Body Functions
and Structures
Environmental
Factors
Personal
Factors | ICF usage in research communications for health disorders | | Physiol. Regul. Integr. Comp. Physiol. 2015, 309, R767–R779. [CrossRef]
39. Laurentino, G.C.; Ugrinowitsch, C.; Roschel, H.; Aoki, M.S.; Soares, A.G.; Neves, M.; Aihara, A.Y.; Fernandes | Laurentino et al. research on integrative physiology | | Telling your client to “push through your heels” when performing a squat or “explode
through your hips or push through your feet” when performing jumping and sprinting
movements are examples of internal cues. You also may utilize external cues to enhance motor learning and performance in all
populations. External cues—or external focus of attention—direct a client’s attention
toward the effect the movement will have on the surrounding environment and the movement outcome, as it relates to the exercise being performed (Winkelman et al., 2017; | effect of external focus of attention on motor learning | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `tp_size`: 0 - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 | |:----------:|:-------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:| | 0.2462 | 10 | 7.2777 | - | - | - | - | - | | 0.4923 | 20 | 7.6341 | - | - | - | - | - | | 0.7385 | 30 | 7.1497 | - | - | - | - | - | | 0.9846 | 40 | 6.8322 | 0.5820 | 0.5741 | 0.5679 | 0.5308 | 0.4724 | | 1.2462 | 50 | 6.779 | - | - | - | - | - | | 1.4923 | 60 | 5.5133 | - | - | - | - | - | | 1.7385 | 70 | 6.1867 | - | - | - | - | - | | 1.9846 | 80 | 6.0276 | 0.5829 | 0.5798 | 0.5769 | 0.5409 | 0.4897 | | 2.2462 | 90 | 4.971 | - | - | - | - | - | | 2.4923 | 100 | 5.0184 | - | - | - | - | - | | 2.7385 | 110 | 5.1473 | - | - | - | - | - | | 2.9846 | 120 | 5.6456 | 0.5880 | 0.5830 | 0.5780 | 0.5472 | 0.4872 | | 3.2462 | 130 | 5.0487 | - | - | - | - | - | | 3.4923 | 140 | 4.7154 | - | - | - | - | - | | 3.7385 | 150 | 5.1362 | - | - | - | - | - | | **3.9846** | **160** | **4.931** | **0.5889** | **0.5828** | **0.5821** | **0.5472** | **0.4836** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.11.12 - Sentence Transformers: 4.0.2 - Transformers: 4.51.2 - PyTorch: 2.6.0+cu124 - Accelerate: 1.5.2 - Datasets: 3.5.0 - Tokenizers: 0.21.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```