SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m on the mortgage-qa-dataset dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: google/embeddinggemma-300m
  • Maximum Sequence Length: 2048 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity
  • Training Dataset:
  • Language: en

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the ๐Ÿค— Hub
model = SentenceTransformer("sugiv/embeddinggemma-300m-mortgage")
# Run inference
queries = [
    "When is a borrower eligible for a streamline refinance?",
]
documents = [
    'A borrower is eligible for a streamline refinance if they have made at least six consecutive on-time payments on their current mortgage.',
    'fha_handbook_4000_1',
    'fha_handbook_4000_1_chunk_007',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.8276, -0.0791, -0.0792]])

Evaluation

Metrics

Information Retrieval

Metric mortgage-eval mortgage-test
cosine_accuracy@1 0.3442 0.2781
cosine_accuracy@3 0.6469 0.5621
cosine_accuracy@5 0.7864 0.7101
cosine_accuracy@10 0.9318 0.8728
cosine_precision@1 0.3442 0.2781
cosine_precision@3 0.2156 0.1874
cosine_precision@5 0.1573 0.142
cosine_precision@10 0.0932 0.0873
cosine_recall@1 0.3442 0.2781
cosine_recall@3 0.6469 0.5621
cosine_recall@5 0.7864 0.7101
cosine_recall@10 0.9318 0.8728
cosine_ndcg@10 0.6209 0.5531
cosine_mrr@10 0.5233 0.453
cosine_map@100 0.5285 0.4622

Training Details

Training Dataset

mortgage-qa-dataset

  • Dataset: mortgage-qa-dataset at de29792
  • Size: 2,699 training samples
  • Columns: question, answer, source_document, and source_chunk
  • Approximate statistics based on the first 1000 samples:
    question answer source_document source_chunk
    type string string string string
    details
    • min: 9 tokens
    • mean: 16.15 tokens
    • max: 27 tokens
    • min: 14 tokens
    • mean: 31.67 tokens
    • max: 62 tokens
    • min: 8 tokens
    • mean: 9.99 tokens
    • max: 14 tokens
    • min: 14 tokens
    • mean: 15.99 tokens
    • max: 20 tokens
  • Samples:
    question answer source_document source_chunk
    When is a full appraisal required for a mortgage application? A full appraisal is required for most transactions, but it can be waived for certain streamlined refinance programs if the Loan-to-Value LTV ratio is 90 or less. fha_handbook_4000_1 fha_handbook_4000_1_chunk_005
    When getting a mortgage, who orders the title insurance for the lender? While often coordinated by the settlement agent, the lender typically requires and is the ultimate recipient of the lenders title insurance policy to protect their financial interest. va_chapter4_underwriting va_chapter4_underwriting_chunk_012
    What components of a loan application does an underwriter assess? Underwriters analyze the four Cs of credit: Capacity income and DTI, Capital assets and reserves, Collateral property value, and Credit credit history and score. va_chapter5_processing va_chapter5_processing_chunk_005
  • Loss: CachedMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "mini_batch_size": 32,
        "gather_across_devices": false
    }
    

Evaluation Dataset

mortgage-qa-dataset

  • Dataset: mortgage-qa-dataset at de29792
  • Size: 337 evaluation samples
  • Columns: question, answer, source_document, and source_chunk
  • Approximate statistics based on the first 337 samples:
    question answer source_document source_chunk
    type string string string string
    details
    • min: 9 tokens
    • mean: 16.44 tokens
    • max: 28 tokens
    • min: 17 tokens
    • mean: 32.28 tokens
    • max: 62 tokens
    • min: 8 tokens
    • mean: 10.14 tokens
    • max: 14 tokens
    • min: 14 tokens
    • mean: 16.14 tokens
    • max: 20 tokens
  • Samples:
    question answer source_document source_chunk
    What financial metrics are crucial for an AUS to issue an approval? Key AUS factors include credit score, loan-to-value ratio, debt-to-income ratio, and the overall strength and stability of the borrowers financial profile. va_chapter4_underwriting va_chapter4_underwriting_chunk_017
    Can you explain how an LTV ratio is figured out? The LTV ratio is calculated by dividing the mortgage loan amount by the appraised value or purchase price of the property, whichever is lower. fannie_mae_servicing_guide fannie_mae_servicing_guide_chunk_002
    How do lenders verify a borrowers employment history? Lenders verify employment by contacting employers directly and typically require a two-year history, which may be confirmed via recent pay stubs and W-2 forms. freddie_mac_guide freddie_mac_guide_chunk_002
  • Loss: CachedMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "mini_batch_size": 32,
        "gather_across_devices": false
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • learning_rate: 3e-06
  • num_train_epochs: 4
  • warmup_steps: 100
  • fp16: True
  • load_best_model_at_end: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 3e-06
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 100
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • hub_revision: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • liger_kernel_config: None
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss Validation Loss mortgage-eval_cosine_ndcg@10 mortgage-test_cosine_ndcg@10
-1 -1 - - 0.5803 -
0.1479 25 0.1574 - - -
0.2959 50 0.1053 0.0722 0.5993 -
0.4438 75 0.0969 - - -
0.5917 100 0.0765 0.0773 0.6085 -
0.7396 125 0.079 - - -
0.8876 150 0.0802 0.0858 0.6056 -
1.0355 175 0.021 - - -
1.1834 200 0.0728 0.0549 0.6093 -
1.3314 225 0.0857 - - -
1.4793 250 0.071 0.0659 0.6145 -
1.6272 275 0.0633 - - -
1.7751 300 0.1844 0.0687 0.6209 -
1.9231 325 0.0545 - - -
2.0710 350 0.0474 0.0646 0.6025 -
2.2189 375 0.0702 - - -
2.3669 400 0.0831 0.0699 0.6026 -
2.5148 425 0.0635 - - -
2.6627 450 0.103 0.0674 0.6031 -
2.8107 475 0.097 - - -
2.9586 500 0.077 0.0686 0.6032 -
3.1065 525 0.0713 - - -
3.2544 550 0.1617 0.0668 0.6087 -
3.4024 575 0.1084 - - -
3.5503 600 0.0791 0.0658 0.6038 -
3.6982 625 0.0477 - - -
3.8462 650 0.0956 0.0659 0.6073 -
3.9941 675 0.0587 - - -
-1 -1 - - 0.6209 0.5531
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.11.11
  • Sentence Transformers: 5.1.0
  • Transformers: 4.57.0.dev0
  • PyTorch: 2.8.0.dev20250319+cu128
  • Accelerate: 1.10.1
  • Datasets: 4.0.0
  • Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
Downloads last month
10
Safetensors
Model size
303M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for sugiv/embeddinggemma-300m-mortgage

Finetuned
(32)
this model

Evaluation results