pj-mathematician/JobSkillBGE-large-en-v1.5

Job-Skill matching fintuned BAAI/bge-large-en-v1.5

Top performing model on TalentCLEF 2025 Task B. Use it for job title <-> skill set matching

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-large-en-v1.5
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pj-mathematician/JobSkillBGE-large-en-v1.5")
# Run inference
sentences = [
    'For roles such as import/export manager, graduate export manager, senior export manager, and other related positions in meat and meat products, the key skills include a strong understanding of international trade regulations, meat product knowledge, customs compliance, and excellent negotiation and communication skills to manage global supply chains effectively. Additionally, proficiency in relevant trade software and languages can be highly beneficial.',
    'Job roles such as Performance Analyst, Quality Assurance Engineer, and Test Manager require skills in conducting performance measurement and organizing or managing conversion testing to ensure software and systems meet performance standards and function correctly in real-world scenarios.',
    'Job roles that require skills such as managing staff, coordinating employees, and performing HR activities include Human Resources Managers, Team Leaders, Supervisors, and Department Heads, all of whom are responsible for overseeing personnel, implementing HR policies, and ensuring efficient team operations.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: full_en
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7303
cosine_accuracy@20	0.9934
cosine_accuracy@50	0.9967
cosine_accuracy@100	1.0
cosine_accuracy@150	1.0
cosine_accuracy@200	1.0
cosine_precision@1	0.7303
cosine_precision@20	0.4998
cosine_precision@50	0.3918
cosine_precision@100	0.3112
cosine_precision@150	0.2652
cosine_precision@200	0.2322
cosine_recall@1	0.0102
cosine_recall@20	0.1337
cosine_recall@50	0.2541
cosine_recall@100	0.3948
cosine_recall@150	0.4963
cosine_recall@200	0.5721
cosine_ndcg@1	0.7303
cosine_ndcg@20	0.5385
cosine_ndcg@50	0.4499
cosine_ndcg@100	0.4428
cosine_ndcg@150	0.4895
cosine_ndcg@200	0.5346
cosine_mrr@1	0.7303
cosine_mrr@20	0.8342
cosine_mrr@50	0.8343
cosine_mrr@100	0.8344
cosine_mrr@150	0.8344
cosine_mrr@200	0.8344
cosine_map@1	0.7303
cosine_map@20	0.3435
cosine_map@50	0.2378
cosine_map@100	0.2116
cosine_map@150	0.229
cosine_map@200	0.2478
cosine_map@500	0.2982

Training Details

Training Dataset

Unnamed Dataset

• Size: 114,699 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 43 tokens mean: 65.45 tokens max: 116 tokens	min: 34 tokens mean: 55.34 tokens max: 162 tokens

• Samples:

  anchor	positive
  A technical director or any of its synonyms requires a strong blend of technical expertise and leadership skills, including the ability to oversee technical operations, manage teams, and ensure the successful execution of technical projects while maintaining operational efficiency and innovation.	Job roles that require promoting health and safety include occupational health and safety specialists, safety managers, and public health educators, all of whom work to ensure safe and healthy environments in workplaces and communities.
  A technical director or any of its synonyms requires a strong blend of technical expertise and leadership skills, including the ability to oversee technical operations, manage teams, and ensure the successful execution of technical projects while maintaining operational efficiency and innovation.	Job roles that require organizing rehearsals include directors, choreographers, and conductors in theater, dance, and music ensembles, who must efficiently plan and schedule practice sessions to prepare performers for a successful final performance.
  A technical director or any of its synonyms requires a strong blend of technical expertise and leadership skills, including the ability to oversee technical operations, manage teams, and ensure the successful execution of technical projects while maintaining operational efficiency and innovation.	Job roles such as Health and Safety Managers, Environmental Health Officers, and Risk Management Specialists often require the skill of negotiating health and safety issues with third parties to ensure compliance and protection standards are met across different organizations and sites.

• Loss: CachedGISTEmbedLoss with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.01, 'mini_batch_size': 32, 'margin_strategy': 'absolute', 'margin': 0.0}
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 128
• gradient_accumulation_steps: 2
• num_train_epochs: 5
• warmup_ratio: 0.05
• log_on_each_node: False
• fp16: True
• dataloader_num_workers: 4
• ddp_find_unused_parameters: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: False
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: True
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	full_en_cosine_ndcg@200
-1	-1	-	0.4784
0.0011	1	9.119	-
0.1116	100	4.1469	-
0.2232	200	2.5294	0.5362
0.3348	300	2.3611	-
0.4464	400	2.192	0.5318
0.5580	500	2.0338	-
0.6696	600	1.9009	0.5383
0.7812	700	1.8404	-
0.8929	800	1.7692	0.5352
1.0045	900	1.6921	-
1.1161	1000	1.3861	0.5368
1.2277	1100	1.3863	-
1.3393	1200	1.3546	0.5259
1.4509	1300	1.373	-
1.5625	1400	1.3364	0.5303
1.6741	1500	1.2876	-
1.7857	1600	1.3094	0.5323
1.8973	1700	1.2784	-
2.0089	1800	1.2204	0.5330
2.1205	1900	0.9617	-
2.2321	2000	1.0004	0.5277
2.3438	2100	0.9694	-
2.4554	2200	0.9843	0.5356
2.5670	2300	0.9743	-
2.6786	2400	0.9252	0.5320
2.7902	2500	0.9272	-
2.9018	2600	0.9279	0.5333
3.0134	2700	0.857	-
3.125	2800	0.7313	0.5300
3.2366	2900	0.7103	-
3.3482	3000	0.7187	0.5319
3.4598	3100	0.7067	-
3.5714	3200	0.7157	0.5369
3.6830	3300	0.7113	-
3.7946	3400	0.7013	0.5341
3.9062	3500	0.6903	-
4.0179	3600	0.6462	0.5335
4.1295	3700	0.5162	-
4.2411	3800	0.524	0.5352
4.3527	3900	0.5303	-
4.4643	4000	0.5269	0.5341
4.5759	4100	0.4824	-
4.6875	4200	0.5222	0.5342
4.7991	4300	0.5104	-
4.9107	4400	0.5002	0.5346

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 4.1.0
• Transformers: 4.51.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}