Fine-tuned all-mpnet-base-v2 for SCION RAG retrieval

README.md

@@ -1,258 +1,40 @@
 ---
+language: en
+license: apache-2.0
 tags:
 - sentence-transformers
-- sentence-similarity
-- feature-extraction
-- generated_from_trainer
-- dataset_size:22349
-- loss:MultipleNegativesRankingLoss
-base_model: sentence-transformers/all-MiniLM-L6-v2
-
-pipeline_tag: sentence-similarity
-library_name: sentence-transformers
+- text-embedding
+- rag
+- scion-architecture
+datasets:
+- custom
 metrics:
-- cosine_accuracy@1
-- cosine_accuracy@3
-- cosine_accuracy@5
-- cosine_accuracy@10
-- cosine_precision@1
-- cosine_precision@3
-- cosine_precision@5
-- cosine_precision@10
-- cosine_recall@1
-- cosine_recall@3
-- cosine_recall@5
-- cosine_recall@10
-- cosine_ndcg@10
-- cosine_mrr@10
-- cosine_map@100
-model-index:
-- name: SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
-  results:
-  - task:
-      type: information-retrieval
-      name: Information Retrieval
-    dataset:
-      name: val ir eval
-      type: val-ir-eval
-    metrics:
-    - type: cosine_accuracy@1
-      value: 0.6766146993318486
-      name: Cosine Accuracy@1
-    - type: cosine_accuracy@3
-      value: 0.8712694877505568
-      name: Cosine Accuracy@3
-    - type: cosine_accuracy@5
-      value: 0.9131403118040089
-      name: Cosine Accuracy@5
-    - type: cosine_accuracy@10
-      value: 0.9532293986636972
-      name: Cosine Accuracy@10
-    - type: cosine_precision@1
-      value: 0.6766146993318486
-      name: Cosine Precision@1
-    - type: cosine_precision@3
-      value: 0.29042316258351886
-      name: Cosine Precision@3
-    - type: cosine_precision@5
-      value: 0.18271714922048995
-      name: Cosine Precision@5
-    - type: cosine_precision@10
-      value: 0.09536748329621379
-      name: Cosine Precision@10
-    - type: cosine_recall@1
-      value: 0.6766146993318486
-      name: Cosine Recall@1
-    - type: cosine_recall@3
-      value: 0.8710467706013363
-      name: Cosine Recall@3
-    - type: cosine_recall@5
-      value: 0.9131403118040089
-      name: Cosine Recall@5
-    - type: cosine_recall@10
-      value: 0.9532293986636972
-      name: Cosine Recall@10
-    - type: cosine_ndcg@10
-      value: 0.8214443655056687
-      name: Cosine Ndcg@10
-    - type: cosine_mrr@10
-      value: 0.778392194294197
-      name: Cosine Mrr@10
-    - type: cosine_map@100
-      value: 0.7806624678050911
-      name: Cosine Map@100
+- retrieval
 ---
 
-# SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
-
-This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) on a dataset of questions about [SCION](https://scion-architecture.net/) Internet Architecture paired with relevant passages from related research papers and documentation. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
-
-## Model Details
-
-### Model Description
-- **Model Type:** Sentence Transformer
-- **Base model:** [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) <!-- at revision fa97f6e7cb1a59073dff9e6b13e2715cf7475ac9 -->
-- **Maximum Sequence Length:** 256 tokens
-- **Output Dimensionality:** 384 dimensions
-- **Similarity Function:** Cosine Similarity
-<!-- - **Training Dataset:** Unknown -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
-
-### Model Sources
-
-- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
-- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
-- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
-
-### Full Model Architecture
-
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
-  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
-  (2): Normalize()
-)
-```
-
-## Usage
-
-### Direct Usage (Sentence Transformers)
-
-First install the Sentence Transformers library:
-
-```bash
-pip install -U sentence-transformers
-```
-
-Then you can load this model and run inference.
-```python
-from sentence_transformers import SentenceTransformer
-
-# Download from the 🤗 Hub
-model = SentenceTransformer("tjohn327/scion-minilm-l6-v2")
-# Run inference
-sentences = [
-    ""
-]
-embeddings = model.encode(sentences)
-print(embeddings.shape)
-# [3, 384]
-
-# Get the similarity scores for the embeddings
-similarities = model.similarity(embeddings, embeddings)
-print(similarities.shape)
-# [3, 3]
-```
-
-<!--
-### Direct Usage (Transformers)
-
-<details><summary>Click to see the direct usage in Transformers</summary>
-
-</details>
--->
-
-<!--
-### Downstream Usage (Sentence Transformers)
-
-You can finetune this model on your own dataset.
-
-<details><summary>Click to expand</summary>
-
-</details>
--->
-
-<!--
-### Out-of-Scope Use
-
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
-
-## Evaluation
-
-### Metrics
-
-#### Information Retrieval
-
-* Dataset: `val-ir-eval`
-* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
-
-| Metric              | Value      |
-|:--------------------|:-----------|
-| cosine_accuracy@1   | 0.6766     |
-| cosine_accuracy@3   | 0.8713     |
-| cosine_accuracy@5   | 0.9131     |
-| cosine_accuracy@10  | 0.9532     |
-| cosine_precision@1  | 0.6766     |
-| cosine_precision@3  | 0.2904     |
-| cosine_precision@5  | 0.1827     |
-| cosine_precision@10 | 0.0954     |
-| cosine_recall@1     | 0.6766     |
-| cosine_recall@3     | 0.871      |
-| cosine_recall@5     | 0.9131     |
-| cosine_recall@10    | 0.9532     |
-| **cosine_ndcg@10**  | **0.8214** |
-| cosine_mrr@10       | 0.7784     |
-| cosine_map@100      | 0.7807     |
-
-<!--
-## Bias, Risks and Limitations
-
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
-
-<!--
-### Recommendations
-
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
-
-
+# SCION Questions Embedding Model
 
-## Citation
+This model is fine-tuned from [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) on a dataset of questions about SCION internet architecture paired with relevant document passages.
 
-### BibTeX
+## Model description
 
-#### Sentence Transformers
-```bibtex
-@inproceedings{reimers-2019-sentence-bert,
-    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
-    author = "Reimers, Nils and Gurevych, Iryna",
-    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
-    month = "11",
-    year = "2019",
-    publisher = "Association for Computational Linguistics",
-    url = "https://arxiv.org/abs/1908.10084",
-}
-```
+The model was fine-tuned to optimize for retrieval performance in RAG applications related to SCION internet architecture.
 
-#### MultipleNegativesRankingLoss
-```bibtex
-@misc{henderson2017efficient,
-    title={Efficient Natural Language Response Suggestion for Smart Reply},
-    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
-    year={2017},
-    eprint={1705.00652},
-    archivePrefix={arXiv},
-    primaryClass={cs.CL}
-}
-```
+## Intended uses & limitations
 
-<!--
-## Glossary
+This model is specifically trained for retrieving relevant passages from a corpus of SCION Internet Architecture related documentation, specifications and research papers.
 
-*Clearly define terms in order to be accessible across audiences.*
--->
+## Training procedure
 
-<!--
-## Model Card Authors
+The model was trained using sentence-transformers with MultipleNegativesRankingLoss on query-document pairs.
 
-*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
--->
 
-<!--
-## Model Card Contact
+## Performance
 
-*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--->
+| Metric | Base Model | Fine-tuned | Improvement |
+|--------|------------|------------|-------------|
+| ndcg@10 | 0.6009 | 0.7928 | +31.92% |
+| mrr | 0.5476 | 0.7475 | +36.52% |
+| hits@1 | 0.4395 | 0.6457 | +46.94% |
+| hits@3 | 0.6211 | 0.8327 | +34.08% |
+| hits@10 | 0.7686 | 0.9323 | +21.30% |

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "./scion-minilm-l6-v2",
+  "_name_or_path": "./scion-minilm",
   "architectures": [
     "BertModel"
   ],

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:386d3d2094d30f14f2341e38fb8b54d57a869dc2bd0c1ad9e6fe0baeb19b365a
+oid sha256:fae86901ed3c73648f0d768f6a3c6587ec960c43533d2a89296d03cbb96e657b
 size 90864192