--- language: - en license: apache-2.0 tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:6300 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss base_model: BAAI/bge-base-en-v1.5 widget: - source_sentence: What are the components of Comcast's domestic distribution revenue? sentences: - Cash used in investing activities was $2.3 billion for fiscal 2023, compared to $2.1 billion for fiscal 2022. - Domestic distribution revenue primarily includes revenue generated from the distribution of our television networks operating predominantly in the United States to traditional and virtual multichannel video providers, and from NBC-affiliated and Telemundo-affiliated local broadcast television stations. Our revenue from distribution agreements is generally based on the number of subscribers receiving the programming on our television networks and a per subscriber fee. Distribution revenue also includes Peacock subscription fees. - In January 2023, Alphabet Inc. announced a reduction of its workforce, consequently recording employee severance and related charges of $2.1 billion for the year. - source_sentence: What was the noncash pre-tax impairment charge recorded due to the disposal of Vrio's operations in 2021, and what are the main components contributing to this amount? sentences: - The cash equities rate per contract (per 100 shares) for NYSE increased by 6%, from $0.045 in 2022 to $0.048 in 2023. - In the second quarter of 2021, we classified the Vrio disposal group as held-for-sale and reported the disposal group at fair value less cost to sell, which resulted in a noncash, pre-tax impairment charge of $4,555, including approximately $2,100 related to accumulated foreign currency translation adjustments and $2,500 related to property, plant and equipment and intangible assets. - 'SECRET LAIR - our internet-based storefront where MAGIC: THE GATHERING fans can purchase exclusive and limited versions of cards.' - source_sentence: What does the Corporate and Other segment include in its composition? sentences: - The segment consists of unallocated corporate expenses and administrative costs and activities not considered when evaluating segment performance as well as certain assets benefiting more than one segment. In addition, intersegment transactions are eliminated within the Corporate and Other segment. - Net cash provided by (used in) operating activities was recorded at $20,930 million for the reported year. - Forward-Looking Statements Certain statements in this report, other than purely historical information, including estimates, projections, statements relating to our business plans, objectives and expected operating results, and the assumptions upon which those statements are based, are “forward-looking statements” within the meaning of the Private Securities Litigation Reform Act of 1995, Section 27A of the Securities Act of 1933 and Section 21E of the Securities Exchange Act of 1934. - source_sentence: What was the purchase price for the repurchase of Mobility preferred interests by AT&T in 2023? sentences: - Net revenue increased $1.5 billion, or 19%, to $9.6 billion in 2023 from $8.1 billion in 2022. On a constant dollar basis, net revenue increased 20%. Comparable sales increased 13%, or 14% on a constant dollar basis. The increase in net revenue was primarily due to increased Americas net revenue. China Mainland and Rest of World net revenue also increased. - Google Services includes products and services such as ads, Android, Chrome, devices, Google Maps, Google Play, Search, and YouTube. Google Services generates revenues primarily from advertising; fees received for consumer subscription-based products such. as YouTube TV, YouTube Music and Premium, and NFL Sunday Ticket; and the sale of apps and in-app purchases and devices. - In April 2023, we also accepted the December 2022 put option notice from the AT&T pension trust and repurchased the remaining 213 million Mobility preferred interests for a purchase price, including accrued and unpaid distributions, of $5,414. - source_sentence: What is the maximum leverage ratio allowed before default under the company's credit facility? sentences: - If the company's leverage ratio exceeds 3.50 to 1, it would be in default of its revolving credit facility, impairing its ability to borrow under the facility. - Research and Development Because the industries in which the Company competes are characterized by rapid technological advances, the Company’s ability to compete successfully depends heavily upon its ability to ensure a continual and timely flow of competitive products, services and technologies to the marketplace. - Visa is focused on extending, enhancing and investing in VisaNet, their proprietary advanced transaction processing network, to offer a single connection point for facilitating payment transactions to multiple endpoints through various form factors. datasets: - philschmid/finanical-rag-embedding-dataset pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: BGE base Financial Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.6771428571428572 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8371428571428572 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8685714285714285 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9185714285714286 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6771428571428572 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.27904761904761904 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17371428571428568 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09185714285714283 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6771428571428572 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8371428571428572 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8685714285714285 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9185714285714286 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.800782444183487 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.762721088435374 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7655884035994069 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.6828571428571428 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8371428571428572 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8757142857142857 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.92 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6828571428571428 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.27904761904761904 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17514285714285713 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09199999999999998 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6828571428571428 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8371428571428572 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8757142857142857 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.92 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.80444342170685 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7670583900226756 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7699510134898729 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.6757142857142857 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8228571428571428 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8642857142857143 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9185714285714286 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6757142857142857 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2742857142857143 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17285714285714285 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09185714285714283 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6757142857142857 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8228571428571428 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8642857142857143 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9185714285714286 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7984105242762846 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7599024943310656 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7625291382895937 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.6714285714285714 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8114285714285714 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8485714285714285 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9014285714285715 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6714285714285714 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2704761904761904 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.16971428571428568 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09014285714285714 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6714285714285714 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8114285714285714 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8485714285714285 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9014285714285715 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7872870842648211 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7507193877551018 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7542921487122674 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.6242857142857143 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.7842857142857143 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.82 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8828571428571429 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6242857142857143 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.26142857142857145 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.16399999999999998 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08828571428571429 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6242857142857143 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.7842857142857143 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.82 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8828571428571429 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7546358861091382 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7135277777777775 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7174129354945035 name: Cosine Map@100 --- # BGE base Financial Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) on the [finanical-rag-embedding-dataset](https://huggingface.co/datasets/philschmid/finanical-rag-embedding-dataset) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - [finanical-rag-embedding-dataset](https://huggingface.co/datasets/philschmid/finanical-rag-embedding-dataset) - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("bnkc123/bge-base-financial-matryoshka") # Run inference sentences = [ "What is the maximum leverage ratio allowed before default under the company's credit facility?", "If the company's leverage ratio exceeds 3.50 to 1, it would be in default of its revolving credit facility, impairing its ability to borrow under the facility.", 'Research and Development Because the industries in which the Company competes are characterized by rapid technological advances, the Company’s ability to compete successfully depends heavily upon its ability to ensure a continual and timely flow of competitive products, services and technologies to the marketplace.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `dim_768` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 768 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.6771 | | cosine_accuracy@3 | 0.8371 | | cosine_accuracy@5 | 0.8686 | | cosine_accuracy@10 | 0.9186 | | cosine_precision@1 | 0.6771 | | cosine_precision@3 | 0.279 | | cosine_precision@5 | 0.1737 | | cosine_precision@10 | 0.0919 | | cosine_recall@1 | 0.6771 | | cosine_recall@3 | 0.8371 | | cosine_recall@5 | 0.8686 | | cosine_recall@10 | 0.9186 | | **cosine_ndcg@10** | **0.8008** | | cosine_mrr@10 | 0.7627 | | cosine_map@100 | 0.7656 | #### Information Retrieval * Dataset: `dim_512` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 512 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.6829 | | cosine_accuracy@3 | 0.8371 | | cosine_accuracy@5 | 0.8757 | | cosine_accuracy@10 | 0.92 | | cosine_precision@1 | 0.6829 | | cosine_precision@3 | 0.279 | | cosine_precision@5 | 0.1751 | | cosine_precision@10 | 0.092 | | cosine_recall@1 | 0.6829 | | cosine_recall@3 | 0.8371 | | cosine_recall@5 | 0.8757 | | cosine_recall@10 | 0.92 | | **cosine_ndcg@10** | **0.8044** | | cosine_mrr@10 | 0.7671 | | cosine_map@100 | 0.77 | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 256 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.6757 | | cosine_accuracy@3 | 0.8229 | | cosine_accuracy@5 | 0.8643 | | cosine_accuracy@10 | 0.9186 | | cosine_precision@1 | 0.6757 | | cosine_precision@3 | 0.2743 | | cosine_precision@5 | 0.1729 | | cosine_precision@10 | 0.0919 | | cosine_recall@1 | 0.6757 | | cosine_recall@3 | 0.8229 | | cosine_recall@5 | 0.8643 | | cosine_recall@10 | 0.9186 | | **cosine_ndcg@10** | **0.7984** | | cosine_mrr@10 | 0.7599 | | cosine_map@100 | 0.7625 | #### Information Retrieval * Dataset: `dim_128` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 128 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.6714 | | cosine_accuracy@3 | 0.8114 | | cosine_accuracy@5 | 0.8486 | | cosine_accuracy@10 | 0.9014 | | cosine_precision@1 | 0.6714 | | cosine_precision@3 | 0.2705 | | cosine_precision@5 | 0.1697 | | cosine_precision@10 | 0.0901 | | cosine_recall@1 | 0.6714 | | cosine_recall@3 | 0.8114 | | cosine_recall@5 | 0.8486 | | cosine_recall@10 | 0.9014 | | **cosine_ndcg@10** | **0.7873** | | cosine_mrr@10 | 0.7507 | | cosine_map@100 | 0.7543 | #### Information Retrieval * Dataset: `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters: ```json { "truncate_dim": 64 } ``` | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.6243 | | cosine_accuracy@3 | 0.7843 | | cosine_accuracy@5 | 0.82 | | cosine_accuracy@10 | 0.8829 | | cosine_precision@1 | 0.6243 | | cosine_precision@3 | 0.2614 | | cosine_precision@5 | 0.164 | | cosine_precision@10 | 0.0883 | | cosine_recall@1 | 0.6243 | | cosine_recall@3 | 0.7843 | | cosine_recall@5 | 0.82 | | cosine_recall@10 | 0.8829 | | **cosine_ndcg@10** | **0.7546** | | cosine_mrr@10 | 0.7135 | | cosine_map@100 | 0.7174 | ## Training Details ### Training Dataset #### finanical-rag-embedding-dataset * Dataset: [finanical-rag-embedding-dataset](https://huggingface.co/datasets/philschmid/finanical-rag-embedding-dataset) at [e0b1781](https://huggingface.co/datasets/philschmid/finanical-rag-embedding-dataset/tree/e0b17819cf52d444066c99f4a176f5717e066300) * Size: 6,300 training samples * Columns: anchor and positive * Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:---------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | anchor | positive | |:----------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | What was the amount of premiums written by Berkshire Hathaway's Insurance Underwriting in 2023, and how did it compare to the previous year? | Premiums written increased $3.5 billion (24.1%) in 2023 compared to 2022. The increase was primarily due to RSUI and CapSpecialty ($2.1 billion), as well as comparative increases from BHSI and BH Direct, and to a lesser extent the other businesses. Premiums written | $ | 18,142 | | | | $ | 14,619 | | | What types of transportation equipment does XTRA Corporation manage in its fleet? | XTRA manages a diverse fleet of approximately 90,000 units located at 47 facilities throughout the U.S. The fleet includes over-the-road and storage trailers, chassis, temperature-controlled vans and flatbed trailers. | | What seasonal trends affect the company's sales volumes? | Sales volumes for the company are highest in the second fiscal quarter due to seasonal influences, particularly during the spring season in the regions it serves. | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `push_to_hub`: True - `hub_model_id`: bnkc123/bge-base-financial-matryoshka - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `tp_size`: 0 - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: True - `resume_from_checkpoint`: None - `hub_model_id`: bnkc123/bge-base-financial-matryoshka - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 | |:---------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:| | 0.8122 | 10 | 25.483 | - | - | - | - | - | | 1.0 | 13 | - | 0.7890 | 0.7887 | 0.7815 | 0.7647 | 0.7280 | | 1.5685 | 20 | 9.1323 | - | - | - | - | - | | 2.0 | 26 | - | 0.7952 | 0.7982 | 0.7933 | 0.7801 | 0.7477 | | 2.3249 | 30 | 6.7535 | - | - | - | - | - | | 3.0 | 39 | - | 0.8019 | 0.8048 | 0.7989 | 0.7865 | 0.7547 | | 3.0812 | 40 | 6.5646 | - | - | - | - | - | | **3.731** | **48** | **-** | **0.8008** | **0.8044** | **0.7984** | **0.7873** | **0.7546** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.12.6 - Sentence Transformers: 4.1.0 - Transformers: 4.51.3 - PyTorch: 2.7.0+cu126 - Accelerate: 1.6.0 - Datasets: 3.5.1 - Tokenizers: 0.21.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```