Created the golden dataset.

TODO.md

@@ -0,0 +1,11 @@
+# TODO
+
+1. Package the RAG (non-agentic) model generator into a factory function 
+2. Ensure one can swap embeddings
+3. Vibe check text embeddings vs snowflake-arctic-embed-l
+4. Generate dataset
+5. Finetune snowflake
+6. Upload to HF
+7. Use in the agentic
+
+https://www.notion.so/Session-11-Certification-Challenge-1c8cd547af3d8103a6b6c5f3408d5854

dataset_card.md

@@ -0,0 +1,115 @@
+---
+language:
+- en
+license: mit
+pretty_name: PsTuts-RAG Q&A Dataset
+size_categories:
+- 10K<n<100K
+tags:
+- rag
+- question-answering
+- photoshop
+- ragas
+---
+
+# 📊 PsTuts-RAG Q&A Dataset
+
+This dataset contains question-answer pairs generated using [RAGAS](https://github.com/explodinggradients/ragas) from Photoshop tutorial video transcripts. It's designed for training and evaluating RAG (Retrieval-Augmented Generation) systems focused on Photoshop tutorials.
+
+## 📝 Dataset Description
+
+### Dataset Summary
+
+The dataset contains 100 question-answer pairs related to Photoshop usage, generated from video transcripts using RAGAS's knowledge graph and testset generation capabilities. The questions are formulated from the perspective of different user personas (Beginner Photoshop User and Photoshop Trainer).
+
+### Dataset Creation
+
+The dataset was created through the following process:
+1. Loading transcripts from Photoshop tutorial videos
+2. Building a knowledge graph using RAGAS with the following transformations:
+   - Headlines extraction
+   - Headline splitting
+   - Summary extraction
+   - Embedding extraction
+   - Theme extraction
+   - NER extraction
+   - Similarity calculations
+3. Generating synthetic question-answer pairs using different query synthesizers:
+   - SingleHopSpecificQuerySynthesizer (80%)
+   - MultiHopAbstractQuerySynthesizer (10%)
+   - MultiHopSpecificQuerySynthesizer (10%)
+
+### Languages
+
+The dataset is in English.
+
+## 📊 Dataset Structure
+
+### Data Instances
+
+Each instance in the dataset contains:
+- `user_input`: A question about Photoshop
+- `reference`: The reference answer
+- Additional metadata from RAGAS generation
+
+Example:
+```
+{
+  "user_input": "How can I use the Move tool to move many layers at once in Photoshop?",
+  "reference": "If you have the Move tool selected in Photoshop, you can move multiple layers at once by selecting those layers in the Layers panel first, then dragging any of the selected layers with the Move tool."
+}
+```
+
+### Data Fields
+
+- `user_input`: String containing the question
+- `reference`: String containing the reference answer
+- Additional RAGAS metadata fields
+
+### Data Splits
+
+The dataset was generated from test and dev splits of the original transcripts.
+
+## 🚀 Usage
+
+This dataset can be used for:
+- Fine-tuning RAG systems for Photoshop-related queries
+- Evaluating RAG system performance on domain-specific (Photoshop) knowledge
+- Benchmarking question-answering models in the design/creative software domain
+
+### Loading the Dataset
+
+```python
+from datasets import load_dataset
+
+dataset = load_dataset("mbudisic/pstuts_rag_qa")
+```
+
+## 📚 Additional Information
+
+### Source Data
+
+The source data consists of transcripts from Photoshop tutorial videos, processed and transformed into a knowledge graph using RAGAS.
+
+### Personas Used in Generation
+
+1. **Beginner Photoshop User**: Learning to complete simple tasks, use tools in Photoshop, and navigate the UI
+2. **Photoshop Trainer**: Experienced trainer looking to develop step-by-step guides for Photoshop beginners
+
+### Citation
+
+If you use this dataset in your research, please cite:
+
+```
+@misc{pstuts_rag_qa,
+  author = {Budisic, Marko},
+  title = {PsTuts-RAG Q&A Dataset},
+  year = {2023},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/datasets/mbudisic/pstuts_rag_qa}}
+}
+```
+
+### Contributions
+
+Thanks to RAGAS for providing the framework to generate this dataset. 

notebooks/create_golden_dataset.ipynb

@@ -717,6 +717,71 @@
     "kg"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 165,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "264714962ac74dec896548157e691803",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(HTML(value='<center> <img\\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "560ae4c3b1b1446f9ba6bd9687e11c57",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Uploading the dataset shards:   0%|          | 0/1 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "c89b9b036ee44a93907628e1ad9ba2af",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Creating parquet from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "CommitInfo(commit_url='https://huggingface.co/datasets/mbudisic/pstuts_rag_qa/commit/6b0d6bb3d96c1d26e5678fd1b781401047ea5d92', commit_message='Upload dataset', commit_description='', oid='6b0d6bb3d96c1d26e5678fd1b781401047ea5d92', pr_url=None, repo_url=RepoUrl('https://huggingface.co/datasets/mbudisic/pstuts_rag_qa', endpoint='https://huggingface.co', repo_type='dataset', repo_id='mbudisic/pstuts_rag_qa'), pr_revision=None, pr_num=None)"
+      ]
+     },
+     "execution_count": 165,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from huggingface_hub import login\n",
+    "login()\n",
+    "ragas_dataset = testset.to_hf_dataset()\n",
+    "# ragas_dataset.push_to_hub(\"mbudisic/pstuts_rag_qa\")"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

uv.lock

@@ -998,6 +998,15 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/78/ce/5e897ee51b7d26ab4e47e5105e7368d40ce6cfae2367acdf3165396d50be/ipython-9.2.0-py3-none-any.whl", hash = "sha256:fef5e33c4a1ae0759e0bba5917c9db4eb8c53fee917b6a526bd973e1ca5159f6", size = 604277 },
 ]
 
+[[package]]
+name = "ipython-genutils"
+version = "0.2.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e8/69/fbeffffc05236398ebfcfb512b6d2511c622871dca1746361006da310399/ipython_genutils-0.2.0.tar.gz", hash = "sha256:eb2e116e75ecef9d4d228fdc66af54269afa26ab4463042e33785b887c628ba8", size = 22208 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl", hash = "sha256:72dd37233799e619666c9f639a9da83c34013a73e8bbc79a7a6348d93c61fab8", size = 26343 },
+]
+
 [[package]]
 name = "ipython-pygments-lexers"
 version = "1.1.1"
@@ -1251,6 +1260,37 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/ca/77/71d78d58f15c22db16328a476426f7ac4a60d3a5a7ba3b9627ee2f7903d4/jupyter_console-6.6.3-py3-none-any.whl", hash = "sha256:309d33409fcc92ffdad25f0bcdf9a4a9daa61b6f341177570fdac03de5352485", size = 24510 },
 ]
 
+[[package]]
+name = "jupyter-contrib-core"
+version = "0.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "jupyter-core" },
+    { name = "notebook" },
+    { name = "setuptools" },
+    { name = "tornado" },
+    { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/50/94/0d37e5b49ea1c8bf204c46f9b0257c1f3319a4ab88acbd401da2cab25e55/jupyter_contrib_core-0.4.2.tar.gz", hash = "sha256:1887212f3ca9d4487d624c0705c20dfdf03d5a0b9ea2557d3aaeeb4c38bdcabb", size = 17490 }
+
+[[package]]
+name = "jupyter-contrib-nbextensions"
+version = "0.7.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "ipython-genutils" },
+    { name = "jupyter-contrib-core" },
+    { name = "jupyter-core" },
+    { name = "jupyter-highlight-selected-word" },
+    { name = "jupyter-nbextensions-configurator" },
+    { name = "lxml" },
+    { name = "nbconvert" },
+    { name = "notebook" },
+    { name = "tornado" },
+    { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/50/91/78cc4362611dbde2b0cd068204aaf1b8899d0459c50d8ff9daca8c069791/jupyter_contrib_nbextensions-0.7.0.tar.gz", hash = "sha256:06e33f005885eb92f89cbe82711e921278201298d08ab0d886d1ba09e8c3e9ca", size = 23462252 }
+
 [[package]]
 name = "jupyter-core"
 version = "5.7.2"
@@ -1284,6 +1324,15 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/e2/48/577993f1f99c552f18a0428731a755e06171f9902fa118c379eb7c04ea22/jupyter_events-0.12.0-py3-none-any.whl", hash = "sha256:6464b2fa5ad10451c3d35fabc75eab39556ae1e2853ad0c0cc31b656731a97fb", size = 19430 },
 ]
 
+[[package]]
+name = "jupyter-highlight-selected-word"
+version = "0.2.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cd/a5/3dfeb7c8643ef502e82969fdebb201b63b33ded15a7761b27299bacebc3a/jupyter_highlight_selected_word-0.2.0.tar.gz", hash = "sha256:9fa740424859a807950ca08d2bfd28a35154cd32dd6d50ac4e0950022adc0e7b", size = 12592 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/50/d7/19ab7cfd60bf268d2abbacc52d4295a40f52d74dfc0d938e4761ee5e598b/jupyter_highlight_selected_word-0.2.0-py2.py3-none-any.whl", hash = "sha256:9545dfa9cb057eebe3a5795604dcd3a5294ea18637e553f61a0b67c1b5903c58", size = 11699 },
+]
+
 [[package]]
 name = "jupyter-lsp"
 version = "2.2.5"
@@ -1296,6 +1345,23 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/07/e0/7bd7cff65594fd9936e2f9385701e44574fc7d721331ff676ce440b14100/jupyter_lsp-2.2.5-py3-none-any.whl", hash = "sha256:45fbddbd505f3fbfb0b6cb2f1bc5e15e83ab7c79cd6e89416b248cb3c00c11da", size = 69146 },
 ]
 
+[[package]]
+name = "jupyter-nbextensions-configurator"
+version = "0.6.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "jupyter-contrib-core" },
+    { name = "jupyter-core" },
+    { name = "jupyter-server" },
+    { name = "notebook" },
+    { name = "pyyaml" },
+    { name = "tornado" },
+    { name = "traitlets" },
+]
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/05/fe/cffb14a4fbb43cf276aa3047e42c3f9ecfda851ba3c466295401f6b1e085/jupyter_nbextensions_configurator-0.6.4-py2.py3-none-any.whl", hash = "sha256:fe7a7b0805b5926449692fb077e0e659bab8b27563bc68cba26854532fdf99c7", size = 466890 },
+]
+
 [[package]]
 name = "jupyter-server"
 version = "2.16.0"
@@ -2351,6 +2417,7 @@ dependencies = [
     { name = "isort" },
     { name = "jq" },
     { name = "jupyter" },
+    { name = "jupyter-contrib-nbextensions" },
     { name = "langchain" },
     { name = "langchain-community" },
     { name = "langchain-core" },
@@ -2403,6 +2470,7 @@ requires-dist = [
     { name = "isort", specifier = ">=6.0.1" },
     { name = "jq", specifier = ">=1.8.0" },
     { name = "jupyter", specifier = ">=1.1.1" },
+    { name = "jupyter-contrib-nbextensions", specifier = ">=0.7.0" },
     { name = "langchain", specifier = ">=0.3.25" },
     { name = "langchain-community", specifier = ">=0.3.23" },
     { name = "langchain-core", specifier = ">=0.3.59" },