yyzheng00/snomed_triplet_500k

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2 on the parquet dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • parquet

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("yyzheng00/snomed_triplet_500k")
# Run inference
sentences = [
    '|Structure of right side of trunk| + |Structure of soft tissue of back of thoracic segment of trunk| : |Laterality| = |Right|, ',
    'Structure of soft tissue of right half of back of thoracic segment of trunk',
    'Structure of vein of right limb',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: snomed_triplet_500k_3_4_3-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9986

Triplet

• Dataset: snomed_triplet_500k_3_4_3-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9987

Training Details

Training Dataset

parquet

• Dataset: parquet
• Size: 500,000 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 18 tokens mean: 61.91 tokens max: 247 tokens	min: 4 tokens mean: 11.04 tokens max: 39 tokens	min: 4 tokens mean: 11.29 tokens max: 50 tokens

• Samples:

  anchor	positive	negative
  	Supernumerary deciduous mandibular tooth	: {
  	Hemorrhage into subdural space of neuraxis	+
  	Deep mammalian bite wound	+

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.2
  }
  

Evaluation Dataset

parquet

• Dataset: parquet
• Size: 500,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 18 tokens mean: 62.56 tokens max: 256 tokens	min: 3 tokens mean: 10.81 tokens max: 33 tokens	min: 3 tokens mean: 11.32 tokens max: 60 tokens

• Samples:

  anchor	positive	negative
  	Anastomosis of duodenum to colon	+
  	Benign neoplasm of colon	+
  	Closed traumatic dislocation of hip	+

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.2
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	snomed_triplet_500k_3_4_3-dev_cosine_accuracy
0.0053	100	0.0143	0.0084	0.9899
0.0107	200	0.009	0.0066	0.9918
0.016	300	0.0082	0.0057	0.9925
0.0213	400	0.0066	0.0053	0.993
0.0267	500	0.0054	0.0049	0.9936
0.032	600	0.0059	0.0047	0.9939
0.0373	700	0.0053	0.0044	0.9946
0.0427	800	0.0047	0.0042	0.9945
0.048	900	0.0048	0.0040	0.9947
0.0533	1000	0.0046	0.0040	0.9951
0.0587	1100	0.0048	0.0039	0.9954
0.064	1200	0.0057	0.0038	0.9948
0.0693	1300	0.0049	0.0038	0.9952
0.0747	1400	0.0038	0.0042	0.9944
0.08	1500	0.0035	0.0038	0.9953
0.0853	1600	0.0034	0.0034	0.9962
0.0907	1700	0.0044	0.0036	0.9957
0.096	1800	0.0041	0.0038	0.9950
0.1013	1900	0.0042	0.0040	0.9948
0.1067	2000	0.0052	0.0037	0.9952
0.112	2100	0.0045	0.0038	0.9953
0.1173	2200	0.0054	0.0034	0.9961
0.1227	2300	0.0041	0.0034	0.9956
0.128	2400	0.0053	0.0035	0.9954
0.1333	2500	0.0043	0.0035	0.9957
0.1387	2600	0.0046	0.0034	0.9955
0.144	2700	0.0039	0.0033	0.9959
0.1493	2800	0.0039	0.0034	0.9958
0.1547	2900	0.0041	0.0033	0.9959
0.16	3000	0.0036	0.0033	0.9960
0.1653	3100	0.0046	0.0033	0.9960
0.1707	3200	0.0041	0.0033	0.9958
0.176	3300	0.0043	0.0033	0.9959
0.1813	3400	0.0041	0.0032	0.9958
0.1867	3500	0.0042	0.0031	0.9961
0.192	3600	0.0049	0.0029	0.9963
0.1973	3700	0.0035	0.0030	0.9960
0.2027	3800	0.0042	0.0029	0.9963
0.208	3900	0.0034	0.0028	0.9966
0.2133	4000	0.0035	0.0030	0.9959
0.2187	4100	0.0045	0.0028	0.9966
0.224	4200	0.0041	0.0028	0.9965
0.2293	4300	0.0027	0.0027	0.9965
0.2347	4400	0.0031	0.0027	0.9967
0.24	4500	0.0039	0.0027	0.9967
0.2453	4600	0.0051	0.0026	0.9972
0.2507	4700	0.0027	0.0029	0.9963
0.256	4800	0.0034	0.0027	0.9968
0.2613	4900	0.003	0.0025	0.9969
0.2667	5000	0.003	0.0025	0.9972
0.272	5100	0.0023	0.0026	0.9971
0.2773	5200	0.0038	0.0026	0.9967
0.2827	5300	0.0027	0.0026	0.9968
0.288	5400	0.0039	0.0029	0.9973
0.2933	5500	0.0031	0.0027	0.9969
0.2987	5600	0.0022	0.0025	0.9970
0.304	5700	0.0024	0.0025	0.9972
0.3093	5800	0.0032	0.0025	0.9968
0.3147	5900	0.0035	0.0025	0.9968
0.32	6000	0.002	0.0025	0.9972
0.3253	6100	0.0025	0.0026	0.9970
0.3307	6200	0.003	0.0026	0.9972
0.336	6300	0.0032	0.0027	0.9970
0.3413	6400	0.0033	0.0026	0.997
0.3467	6500	0.0026	0.0024	0.9973
0.352	6600	0.0029	0.0024	0.9972
0.3573	6700	0.002	0.0024	0.9973
0.3627	6800	0.0027	0.0024	0.9971
0.368	6900	0.0025	0.0025	0.9971
0.3733	7000	0.0034	0.0023	0.9976
0.3787	7100	0.0027	0.0023	0.9975
0.384	7200	0.0029	0.0025	0.9974
0.3893	7300	0.003	0.0024	0.9976
0.3947	7400	0.0022	0.0022	0.9975
0.4	7500	0.0023	0.0022	0.9974
0.4053	7600	0.0028	0.0022	0.9973
0.4107	7700	0.0027	0.0022	0.9974
0.416	7800	0.004	0.0024	0.9972
0.4213	7900	0.0027	0.0024	0.9977
0.4267	8000	0.0043	0.0023	0.9977
0.432	8100	0.0031	0.0021	0.9977
0.4373	8200	0.0027	0.0022	0.9974
0.4427	8300	0.0031	0.0022	0.9973
0.448	8400	0.0023	0.0022	0.9975
0.4533	8500	0.0028	0.0022	0.9973
0.4587	8600	0.0027	0.0021	0.9976
0.464	8700	0.0027	0.0021	0.9977
0.4693	8800	0.0024	0.0021	0.9977
0.4747	8900	0.0027	0.0021	0.9979
0.48	9000	0.0025	0.0022	0.9974
0.4853	9100	0.0027	0.0022	0.9974
0.4907	9200	0.0035	0.0019	0.9978
0.496	9300	0.0027	0.0019	0.9978
0.5013	9400	0.0028	0.0019	0.9979
0.5067	9500	0.0023	0.0019	0.9979
0.512	9600	0.0015	0.0019	0.9979
0.5173	9700	0.0023	0.0019	0.9979
0.5227	9800	0.0021	0.0018	0.9979
0.528	9900	0.0019	0.0018	0.9979
0.5333	10000	0.0029	0.0018	0.9981
0.5387	10100	0.003	0.0019	0.9978
0.544	10200	0.0021	0.0019	0.9978
0.5493	10300	0.0022	0.0019	0.9978
0.5547	10400	0.0018	0.0018	0.9978
0.56	10500	0.0017	0.0019	0.9977
0.5653	10600	0.0023	0.0018	0.9978
0.5707	10700	0.0017	0.0017	0.9979
0.576	10800	0.0023	0.0018	0.998
0.5813	10900	0.0025	0.0018	0.9978
0.5867	11000	0.0024	0.0017	0.9979
0.592	11100	0.0022	0.0018	0.9979
0.5973	11200	0.0028	0.0017	0.9980
0.6027	11300	0.0016	0.0017	0.9978
0.608	11400	0.0021	0.0017	0.9980
0.6133	11500	0.0013	0.0017	0.9981
0.6187	11600	0.0022	0.0017	0.9980
0.624	11700	0.0022	0.0019	0.9976
0.6293	11800	0.0017	0.0018	0.9977
0.6347	11900	0.0024	0.0017	0.9980
0.64	12000	0.0023	0.0017	0.9980
0.6453	12100	0.0019	0.0016	0.9981
0.6507	12200	0.0016	0.0016	0.9982
0.656	12300	0.0025	0.0016	0.9982
0.6613	12400	0.002	0.0016	0.9981
0.6667	12500	0.0019	0.0016	0.9981
0.672	12600	0.0021	0.0016	0.998
0.6773	12700	0.0023	0.0016	0.9980
0.6827	12800	0.0021	0.0016	0.9982
0.688	12900	0.0017	0.0016	0.9981
0.6933	13000	0.0026	0.0016	0.9981
0.6987	13100	0.0021	0.0016	0.9982
0.704	13200	0.0025	0.0016	0.9981
0.7093	13300	0.0019	0.0016	0.9980
0.7147	13400	0.0019	0.0016	0.9981
0.72	13500	0.0015	0.0016	0.9982
0.7253	13600	0.0023	0.0016	0.9980
0.7307	13700	0.0028	0.0016	0.9981
0.736	13800	0.0018	0.0015	0.9982
0.7413	13900	0.002	0.0016	0.9980
0.7467	14000	0.0023	0.0015	0.9983
0.752	14100	0.0017	0.0015	0.9981
0.7573	14200	0.0018	0.0015	0.9983
0.7627	14300	0.0023	0.0014	0.9983
0.768	14400	0.0014	0.0014	0.9982
0.7733	14500	0.0014	0.0014	0.9982
0.7787	14600	0.0012	0.0014	0.9982
0.784	14700	0.0022	0.0014	0.9982
0.7893	14800	0.0015	0.0014	0.9983
0.7947	14900	0.0018	0.0014	0.9983
0.8	15000	0.0019	0.0014	0.9984
0.8053	15100	0.0014	0.0014	0.9982
0.8107	15200	0.0017	0.0014	0.9982
0.816	15300	0.0014	0.0014	0.9983
0.8213	15400	0.0017	0.0014	0.9982
0.8267	15500	0.001	0.0014	0.9982
0.832	15600	0.0021	0.0014	0.9982
0.8373	15700	0.0013	0.0014	0.9982
0.8427	15800	0.002	0.0013	0.9983
0.848	15900	0.0014	0.0013	0.9983
0.8533	16000	0.0015	0.0013	0.9983
0.8587	16100	0.001	0.0013	0.9983
0.864	16200	0.0014	0.0013	0.9983
0.8693	16300	0.0013	0.0013	0.9983
0.8747	16400	0.0019	0.0013	0.9983
0.88	16500	0.0017	0.0013	0.9983
0.8853	16600	0.0014	0.0013	0.9984
0.8907	16700	0.0018	0.0013	0.9984
0.896	16800	0.001	0.0013	0.9984
0.9013	16900	0.0019	0.0013	0.9984
0.9067	17000	0.0022	0.0013	0.9985
0.912	17100	0.0017	0.0013	0.9985
0.9173	17200	0.002	0.0013	0.9986
0.9227	17300	0.0012	0.0013	0.9986
0.928	17400	0.0015	0.0013	0.9985
0.9333	17500	0.0012	0.0013	0.9985
0.9387	17600	0.0014	0.0013	0.9985
0.944	17700	0.0019	0.0013	0.9985
0.9493	17800	0.0018	0.0013	0.9985
0.9547	17900	0.001	0.0013	0.9985
0.96	18000	0.0016	0.0013	0.9986
0.9653	18100	0.0011	0.0013	0.9986
0.9707	18200	0.0016	0.0013	0.9986
0.976	18300	0.0024	0.0013	0.9986
0.9813	18400	0.0026	0.0013	0.9986
0.9867	18500	0.001	0.0013	0.9986
0.992	18600	0.0015	0.0013	0.9986
0.9973	18700	0.0017	0.0013	0.9986
1.0	18750	-	-	0.9987

Framework Versions

• Python: 3.11.1
• Sentence Transformers: 3.3.1
• Transformers: 4.47.0
• PyTorch: 2.1.1+cu121
• Accelerate: 1.2.0
• Datasets: 2.18.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}