{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# PSTUTS_RAG RAG evaluation\n", "\n", "We are going to be comparing the RAG with (a) base, (b) fine-tuned embedding model.\n" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "base_model_HF_id = \"Snowflake/snowflake-arctic-embed-s\"\n", "ft_model_HF_id = \"mbudisic/snowflake-arctic-embed-s-ft-pstuts\"" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "import logging\n", "\n", "import requests\n", "from dotenv import load_dotenv\n", "from langchain_openai import ChatOpenAI\n", "\n", "from qdrant_client import QdrantClient\n", "\n", "from pstuts_rag.rag import RAGChainInstance\n", "import nest_asyncio\n", "\n", "\n", "from dataclasses import dataclass\n", "from datasets import load_dataset\n", "from langsmith import EvaluationResult\n", "from ragas import EvaluationDataset\n", "from pstuts_rag.evaluator_utils import apply_rag_chain_inplace, summary_stats\n", "from pandas import DataFrame\n", "from langchain_core.runnables import Runnable\n", "\n", "load_dotenv()\n", "\n", "def set_api_key_if_not_present(key_name, prompt_message=\"\"):\n", " if len(prompt_message) == 0:\n", " prompt_message=key_name\n", " if key_name not in os.environ or not os.environ[key_name]:\n", " os.environ[key_name] = getpass.getpass(prompt_message)\n", "\n", "\n", "set_api_key_if_not_present(\"OPENAI_API_KEY\")\n", "\n", "logging.getLogger(\"httpx\").setLevel(logging.WARNING)\n", "logging.getLogger(\"langchain\").setLevel(logging.WARNING)\n", "nest_asyncio.apply()\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Raw data is now stored on huggingface, so we can download it directly." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import pstuts_rag.loader\n", "\n", "url = \"https://huggingface.co/datasets/mbudisic/PsTuts-VQA/raw/main/train.json\"\n", "resp = requests.get(url)\n", "resp.raise_for_status()\n", "group = url.split('/')[-1].split('.')[0]\n", "docs_json = pstuts_rag.loader.load_json_string(resp.content.decode('utf-8'), group)\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's create the base chain." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from langchain_openai import OpenAIEmbeddings\n", "\n", "from langchain_huggingface import HuggingFaceEmbeddings\n", "\n", "\n", "qdrant_client = QdrantClient(\":memory:\")\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from dataclasses import dataclass\n", "from dataclasses import field\n", "\n", "\n", "@dataclass\n", "class DataGroup:\n", " rag:RAGChainInstance= field(init=False) \n", " dataset:EvaluationDataset= field(init=False) \n", " result:EvaluationResult= field(init=False) \n", " statistics:DataFrame= field(init=False) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Base model" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "\n", "base = DataGroup()\n", "base.rag = RAGChainInstance(name=\"base\",\n", " qdrant_client=qdrant_client,\n", " llm=ChatOpenAI(model=\"gpt-4.1-nano\"),\n", " embeddings=HuggingFaceEmbeddings(model_name=base_model_HF_id))\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's populate the datastore of the first chain and create the chain\n", "and test it out.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "==================================\u001b[1m Ai Message \u001b[0m==================================\n", "\n", "A layer is like a separate sheet in your Photoshop document that you can work on independently. You can add new layers, rename them, and change their order. These layers can hold different parts of your image, like colors or drawings, and you can manipulate them without affecting the rest of the image. (Timestamp: 00:02:21)\n", "**REFERENCES**\n", "[\n", " {\n", " \"title\": \"Learn layer basics\",\n", " \"source\": \"https://images-tv.adobe.com/avp/vr/b758b4c4-2a74-41f4-8e67-e2f2eab83c6a/01a575ae-f8b7-486c-987b-bcb4f2f4e57d/3868e305-c73c-4931-82a0-5e46f5eb41e5_20170727011800.1280x720at2400_h264.mp4\",\n", " \"start\": 141.29,\n", " \"stop\": 156.87\n", " },\n", " {\n", " \"title\": \"Unlock the Background layer\",\n", " \"source\": \"https://images-tv.adobe.com/avp/vr/b758b4c4-2a74-41f4-8e67-e2f2eab83c6a/696245e0-aaad-42df-b48f-8b44b1f5211a/22729011-a533-48a4-a7a2-0b5f86d4eedd_20170727011751.1280x720at2400_h264.mp4\",\n", " \"start\": 113.65,\n", " \"stop\": 227.99\n", " }\n", "]\n" ] } ], "source": [ "_ = await base.rag.build_chain(docs_json)\n", "response = base.rag.rag_chain.invoke({\"question\":\"What is a layer?\"})\n", "response.pretty_print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Formal evaluation goes through the \"golden\" dataset also stored \n", "on HF.\n", "\n", "We're going to evaluate only on a portion of it.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "\n", "golden_small_hf = load_dataset(\"mbudisic/pstuts_rag_qa\",split=\"train[:10]\")\n", "\n", "base.dataset = EvaluationDataset.from_hf_dataset(golden_small_hf)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_inputretrieved_contextsreference_contextsresponsereference
0how i use adobe photoshop creative cloud for d...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Here's how to use Perspective Warp in Adobe Ph...in adobe photoshop creative cloud, to use pers...
1wut is Adobee Photoshoop Cretive Cloud?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is a version of...Adobe Photoshop Creative Cloud is a version of...
2As a beginner Photoshop user, can you explain ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...The Perspective Warp feature in Adobe Photosho...Adobe Photoshop Creative Cloud's Perspective W...
3Who is PhotoSpin in relation to the image used...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...PhotoSpin is the company that took the photogr...PhotoSpin is the company that took the photogr...
4How you use Perspective Warp in Photoshop, wha...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop allows you to ch...Perspective Warp in Photoshop let you change t...
5What does the Perspective Warp feature in Phot...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...The Perspective Warp feature in Photoshop allo...Perspective Warp in Photoshop allows you to ch...
6As a Photoshop trainer developing step-by-step...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Based on the transcript, here is how the Persp...The new Perspective Warp feature in Adobe Phot...
7wut is adobee fotoshop cretive clowd?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is a service th...Adobe Photoshop Creative Cloud is a version of...
8Wut duz Perspectiv Warp do in Photoshop?[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop allows you to ch...Perspective Warp in Photoshop lets yu change t...
9How can I, as a Photoshop trainer, explain to ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...In the Perspective Warp tutorial, the trainer ...In the Perspective Warp tutorial, the image us...
\n", "
" ], "text/plain": [ " user_input \\\n", "0 how i use adobe photoshop creative cloud for d... \n", "1 wut is Adobee Photoshoop Cretive Cloud? \n", "2 As a beginner Photoshop user, can you explain ... \n", "3 Who is PhotoSpin in relation to the image used... \n", "4 How you use Perspective Warp in Photoshop, wha... \n", "5 What does the Perspective Warp feature in Phot... \n", "6 As a Photoshop trainer developing step-by-step... \n", "7 wut is adobee fotoshop cretive clowd? \n", "8 Wut duz Perspectiv Warp do in Photoshop? \n", "9 How can I, as a Photoshop trainer, explain to ... \n", "\n", " retrieved_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [If I turn it on and off, you can see the befo... \n", "5 [If I turn it on and off, you can see the befo... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [If I turn it on and off, you can see the befo... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " reference_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [>> What I want to show you in this video is s... \n", "5 [>> What I want to show you in this video is s... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [>> What I want to show you in this video is s... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " response \\\n", "0 Here's how to use Perspective Warp in Adobe Ph... \n", "1 Adobe Photoshop Creative Cloud is a version of... \n", "2 The Perspective Warp feature in Adobe Photosho... \n", "3 PhotoSpin is the company that took the photogr... \n", "4 Perspective Warp in Photoshop allows you to ch... \n", "5 The Perspective Warp feature in Photoshop allo... \n", "6 Based on the transcript, here is how the Persp... \n", "7 Adobe Photoshop Creative Cloud is a service th... \n", "8 Perspective Warp in Photoshop allows you to ch... \n", "9 In the Perspective Warp tutorial, the trainer ... \n", "\n", " reference \n", "0 in adobe photoshop creative cloud, to use pers... \n", "1 Adobe Photoshop Creative Cloud is a version of... \n", "2 Adobe Photoshop Creative Cloud's Perspective W... \n", "3 PhotoSpin is the company that took the photogr... \n", "4 Perspective Warp in Photoshop let you change t... \n", "5 Perspective Warp in Photoshop allows you to ch... \n", "6 The new Perspective Warp feature in Adobe Phot... \n", "7 Adobe Photoshop Creative Cloud is a version of... \n", "8 Perspective Warp in Photoshop lets yu change t... \n", "9 In the Perspective Warp tutorial, the image us... " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "_ = await apply_rag_chain_inplace(base.rag.rag_chain, base.dataset )\n", "base.dataset.to_pandas()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we now have the dataset, let's run it through evalutors." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from ragas.llms import LangchainLLMWrapper\n", "evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model=\"gpt-4.1-mini\"))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "81f92664f2464dc0b3113bb2724e803d", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Evaluating: 0%| | 0/60 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_inputretrieved_contextsreference_contextsresponsereferencecontext_recallfaithfulnessfactual_correctness(mode=f1)answer_relevancycontext_entity_recallnoise_sensitivity(mode=relevant)
0how i use adobe photoshop creative cloud for d...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Here's how to use Perspective Warp in Adobe Ph...in adobe photoshop creative cloud, to use pers...0.3000000.2500000.650.9075310.4000000.062500
1wut is Adobee Photoshoop Cretive Cloud?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is a version of...Adobe Photoshop Creative Cloud is a version of...1.0000001.0000000.670.8796311.0000000.200000
2As a beginner Photoshop user, can you explain ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...The Perspective Warp feature in Adobe Photosho...Adobe Photoshop Creative Cloud's Perspective W...0.4000001.0000000.560.9333970.2222220.250000
3Who is PhotoSpin in relation to the image used...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...PhotoSpin is the company that took the photogr...PhotoSpin is the company that took the photogr...1.0000001.0000001.000.9144220.5000000.000000
4How you use Perspective Warp in Photoshop, wha...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop allows you to ch...Perspective Warp in Photoshop let you change t...0.2500000.6666670.330.9595030.5000000.333333
5What does the Perspective Warp feature in Phot...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...The Perspective Warp feature in Photoshop allo...Perspective Warp in Photoshop allows you to ch...0.5000000.6666670.800.9822261.0000000.333333
6As a Photoshop trainer developing step-by-step...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Based on the transcript, here is how the Persp...The new Perspective Warp feature in Adobe Phot...0.2727270.9166670.500.9456180.4000000.250000
7wut is adobee fotoshop cretive clowd?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is a service th...Adobe Photoshop Creative Cloud is a version of...1.0000000.6000000.600.8469811.0000000.000000
8Wut duz Perspectiv Warp do in Photoshop?[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop allows you to ch...Perspective Warp in Photoshop lets yu change t...1.0000000.6666670.670.9261591.0000000.333333
9How can I, as a Photoshop trainer, explain to ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...In the Perspective Warp tutorial, the trainer ...In the Perspective Warp tutorial, the image us...1.0000000.4444440.760.8490110.3333330.222222
\n", "" ], "text/plain": [ " user_input \\\n", "0 how i use adobe photoshop creative cloud for d... \n", "1 wut is Adobee Photoshoop Cretive Cloud? \n", "2 As a beginner Photoshop user, can you explain ... \n", "3 Who is PhotoSpin in relation to the image used... \n", "4 How you use Perspective Warp in Photoshop, wha... \n", "5 What does the Perspective Warp feature in Phot... \n", "6 As a Photoshop trainer developing step-by-step... \n", "7 wut is adobee fotoshop cretive clowd? \n", "8 Wut duz Perspectiv Warp do in Photoshop? \n", "9 How can I, as a Photoshop trainer, explain to ... \n", "\n", " retrieved_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [If I turn it on and off, you can see the befo... \n", "5 [If I turn it on and off, you can see the befo... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [If I turn it on and off, you can see the befo... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " reference_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [>> What I want to show you in this video is s... \n", "5 [>> What I want to show you in this video is s... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [>> What I want to show you in this video is s... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " response \\\n", "0 Here's how to use Perspective Warp in Adobe Ph... \n", "1 Adobe Photoshop Creative Cloud is a version of... \n", "2 The Perspective Warp feature in Adobe Photosho... \n", "3 PhotoSpin is the company that took the photogr... \n", "4 Perspective Warp in Photoshop allows you to ch... \n", "5 The Perspective Warp feature in Photoshop allo... \n", "6 Based on the transcript, here is how the Persp... \n", "7 Adobe Photoshop Creative Cloud is a service th... \n", "8 Perspective Warp in Photoshop allows you to ch... \n", "9 In the Perspective Warp tutorial, the trainer ... \n", "\n", " reference context_recall \\\n", "0 in adobe photoshop creative cloud, to use pers... 0.300000 \n", "1 Adobe Photoshop Creative Cloud is a version of... 1.000000 \n", "2 Adobe Photoshop Creative Cloud's Perspective W... 0.400000 \n", "3 PhotoSpin is the company that took the photogr... 1.000000 \n", "4 Perspective Warp in Photoshop let you change t... 0.250000 \n", "5 Perspective Warp in Photoshop allows you to ch... 0.500000 \n", "6 The new Perspective Warp feature in Adobe Phot... 0.272727 \n", "7 Adobe Photoshop Creative Cloud is a version of... 1.000000 \n", "8 Perspective Warp in Photoshop lets yu change t... 1.000000 \n", "9 In the Perspective Warp tutorial, the image us... 1.000000 \n", "\n", " faithfulness factual_correctness(mode=f1) answer_relevancy \\\n", "0 0.250000 0.65 0.907531 \n", "1 1.000000 0.67 0.879631 \n", "2 1.000000 0.56 0.933397 \n", "3 1.000000 1.00 0.914422 \n", "4 0.666667 0.33 0.959503 \n", "5 0.666667 0.80 0.982226 \n", "6 0.916667 0.50 0.945618 \n", "7 0.600000 0.60 0.846981 \n", "8 0.666667 0.67 0.926159 \n", "9 0.444444 0.76 0.849011 \n", "\n", " context_entity_recall noise_sensitivity(mode=relevant) \n", "0 0.400000 0.062500 \n", "1 1.000000 0.200000 \n", "2 0.222222 0.250000 \n", "3 0.500000 0.000000 \n", "4 0.500000 0.333333 \n", "5 1.000000 0.333333 \n", "6 0.400000 0.250000 \n", "7 1.000000 0.000000 \n", "8 1.000000 0.333333 \n", "9 0.333333 0.222222 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "base.result.to_pandas()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "context_recall 0.672273\n", "faithfulness 0.721111\n", "factual_correctness(mode=f1) 0.654000\n", "answer_relevancy 0.914448\n", "context_entity_recall 0.635556\n", "noise_sensitivity(mode=relevant) 0.198472\n", "Name: Mean, dtype: float64\n", "context_recall 0.111424\n", "faithfulness 0.081215\n", "factual_correctness(mode=f1) 0.057081\n", "answer_relevancy 0.014215\n", "context_entity_recall 0.102262\n", "noise_sensitivity(mode=relevant) 0.041861\n", "Name: StdDev, dtype: float64\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/mbudisic/Documents/PsTuts-RAG/pstuts_rag/pstuts_rag/evaluator_utils.py:193: FutureWarning: errors='ignore' is deprecated and will raise in a future version. Use to_numeric without passing `errors` and catch exceptions explicitly instead\n", " retval = retval.apply(partial(pd.to_numeric, **{\"errors\": \"ignore\"}))\n" ] } ], "source": [ "base.statistics = summary_stats(base.result.to_pandas())\n", "print( base.statistics.select_dtypes(include=\"number\") \\\n", " .loc[\"Mean\"] )\n", "print( base.statistics.select_dtypes(include=\"number\") \\\n", ".loc[\"StdDev\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fine-tuned model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of BertModel were not initialized from the model checkpoint at mbudisic/snowflake-arctic-embed-s-ft-pstuts and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] } ], "source": [ "ft = DataGroup()\n", "ft.rag = RAGChainInstance(name=\"ft\",\n", " qdrant_client=qdrant_client,\n", " llm=ChatOpenAI(model=\"gpt-4.1-nano\"),\n", " embeddings=HuggingFaceEmbeddings(model_name=ft_model_HF_id))\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "==================================\u001b[1m Ai Message \u001b[0m==================================\n", "\n", "A layer is something you can work with in Photoshop, like a part of your image that you can edit separately. When you make a new layer, it’s like adding a new sheet of paper that you can move or fill with color without affecting the other parts of your image. (Timestamp: 1:41 - 1:56)\n", "**REFERENCES**\n", "[\n", " {\n", " \"title\": \"Learn layer basics\",\n", " \"source\": \"https://images-tv.adobe.com/avp/vr/b758b4c4-2a74-41f4-8e67-e2f2eab83c6a/01a575ae-f8b7-486c-987b-bcb4f2f4e57d/3868e305-c73c-4931-82a0-5e46f5eb41e5_20170727011800.1280x720at2400_h264.mp4\",\n", " \"start\": 141.29,\n", " \"stop\": 156.87\n", " },\n", " {\n", " \"title\": \"Unlock the Background layer\",\n", " \"source\": \"https://images-tv.adobe.com/avp/vr/b758b4c4-2a74-41f4-8e67-e2f2eab83c6a/696245e0-aaad-42df-b48f-8b44b1f5211a/22729011-a533-48a4-a7a2-0b5f86d4eedd_20170727011751.1280x720at2400_h264.mp4\",\n", " \"start\": 113.65,\n", " \"stop\": 227.99\n", " }\n", "]\n" ] } ], "source": [ "_ = await ft.rag.build_chain(docs_json)\n", "response = ft.rag.rag_chain.invoke({\"question\":\"What is a layer?\"})\n", "response.pretty_print()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "ft.dataset = EvaluationDataset.from_hf_dataset(golden_small_hf)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_inputretrieved_contextsreference_contextsresponsereference
0how i use adobe photoshop creative cloud for d...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Here's how to use Perspective Warp in Photosho...in adobe photoshop creative cloud, to use pers...
1wut is Adobee Photoshoop Cretive Cloud?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is mentioned as...Adobe Photoshop Creative Cloud is a version of...
2As a beginner Photoshop user, can you explain ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud's Perspective W...Adobe Photoshop Creative Cloud's Perspective W...
3Who is PhotoSpin in relation to the image used...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...PhotoSpin is the company that took the photogr...PhotoSpin is the company that took the photogr...
4How you use Perspective Warp in Photoshop, wha...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop is a tool that a...Perspective Warp in Photoshop let you change t...
5What does the Perspective Warp feature in Phot...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...The Perspective Warp feature in Photoshop allo...Perspective Warp in Photoshop allows you to ch...
6As a Photoshop trainer developing step-by-step...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Based on the transcript, here's how the Perspe...The new Perspective Warp feature in Adobe Phot...
7wut is adobee fotoshop cretive clowd?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is a version of...Adobe Photoshop Creative Cloud is a version of...
8Wut duz Perspectiv Warp do in Photoshop?[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop allows you to ch...Perspective Warp in Photoshop lets yu change t...
9How can I, as a Photoshop trainer, explain to ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...In the Perspective Warp tutorial, the PhotoSpi...In the Perspective Warp tutorial, the image us...
\n", "
" ], "text/plain": [ " user_input \\\n", "0 how i use adobe photoshop creative cloud for d... \n", "1 wut is Adobee Photoshoop Cretive Cloud? \n", "2 As a beginner Photoshop user, can you explain ... \n", "3 Who is PhotoSpin in relation to the image used... \n", "4 How you use Perspective Warp in Photoshop, wha... \n", "5 What does the Perspective Warp feature in Phot... \n", "6 As a Photoshop trainer developing step-by-step... \n", "7 wut is adobee fotoshop cretive clowd? \n", "8 Wut duz Perspectiv Warp do in Photoshop? \n", "9 How can I, as a Photoshop trainer, explain to ... \n", "\n", " retrieved_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [If I turn it on and off, you can see the befo... \n", "5 [If I turn it on and off, you can see the befo... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [If I turn it on and off, you can see the befo... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " reference_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [>> What I want to show you in this video is s... \n", "5 [>> What I want to show you in this video is s... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [>> What I want to show you in this video is s... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " response \\\n", "0 Here's how to use Perspective Warp in Photosho... \n", "1 Adobe Photoshop Creative Cloud is mentioned as... \n", "2 Adobe Photoshop Creative Cloud's Perspective W... \n", "3 PhotoSpin is the company that took the photogr... \n", "4 Perspective Warp in Photoshop is a tool that a... \n", "5 The Perspective Warp feature in Photoshop allo... \n", "6 Based on the transcript, here's how the Perspe... \n", "7 Adobe Photoshop Creative Cloud is a version of... \n", "8 Perspective Warp in Photoshop allows you to ch... \n", "9 In the Perspective Warp tutorial, the PhotoSpi... \n", "\n", " reference \n", "0 in adobe photoshop creative cloud, to use pers... \n", "1 Adobe Photoshop Creative Cloud is a version of... \n", "2 Adobe Photoshop Creative Cloud's Perspective W... \n", "3 PhotoSpin is the company that took the photogr... \n", "4 Perspective Warp in Photoshop let you change t... \n", "5 Perspective Warp in Photoshop allows you to ch... \n", "6 The new Perspective Warp feature in Adobe Phot... \n", "7 Adobe Photoshop Creative Cloud is a version of... \n", "8 Perspective Warp in Photoshop lets yu change t... \n", "9 In the Perspective Warp tutorial, the image us... " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "_ = await apply_rag_chain_inplace(ft.rag.rag_chain, ft.dataset )\n", "ft.dataset.to_pandas()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1ac9f0608c57477988b371aa2f211842", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Evaluating: 0%| | 0/60 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_inputretrieved_contextsreference_contextsresponsereferencecontext_recallfaithfulnessfactual_correctness(mode=f1)answer_relevancycontext_entity_recallnoise_sensitivity(mode=relevant)
0how i use adobe photoshop creative cloud for d...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Here's how to use Perspective Warp in Photosho...in adobe photoshop creative cloud, to use pers...0.3000000.2500000.470.8969760.4000000.062500
1wut is Adobee Photoshoop Cretive Cloud?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is mentioned as...Adobe Photoshop Creative Cloud is a version of...1.0000000.6000000.500.8764451.0000000.600000
2As a beginner Photoshop user, can you explain ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud's Perspective W...Adobe Photoshop Creative Cloud's Perspective W...0.4000001.0000000.500.9344800.2222220.000000
3Who is PhotoSpin in relation to the image used...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...PhotoSpin is the company that took the photogr...PhotoSpin is the company that took the photogr...1.0000000.5000000.670.9144220.5000000.000000
4How you use Perspective Warp in Photoshop, wha...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop is a tool that a...Perspective Warp in Photoshop let you change t...0.2500000.8000000.430.9602460.5000000.200000
5What does the Perspective Warp feature in Phot...[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...The Perspective Warp feature in Photoshop allo...Perspective Warp in Photoshop allows you to ch...0.5000000.6666670.800.9822261.0000000.333333
6As a Photoshop trainer developing step-by-step...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Based on the transcript, here's how the Perspe...The new Perspective Warp feature in Adobe Phot...0.2727271.0000000.470.0000000.4000000.300000
7wut is adobee fotoshop cretive clowd?[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...Adobe Photoshop Creative Cloud is a version of...Adobe Photoshop Creative Cloud is a version of...1.0000001.0000000.800.8203731.0000000.000000
8Wut duz Perspectiv Warp do in Photoshop?[If I turn it on and off, you can see the befo...[>> What I want to show you in this video is s...Perspective Warp in Photoshop allows you to ch...Perspective Warp in Photoshop lets yu change t...1.0000000.6666670.670.9261591.0000000.333333
9How can I, as a Photoshop trainer, explain to ...[>> What I want to show you in this video is s...[>> What I want to show you in this video is s...In the Perspective Warp tutorial, the PhotoSpi...In the Perspective Warp tutorial, the image us...1.0000001.0000000.670.8793230.3333330.400000
\n", "" ], "text/plain": [ " user_input \\\n", "0 how i use adobe photoshop creative cloud for d... \n", "1 wut is Adobee Photoshoop Cretive Cloud? \n", "2 As a beginner Photoshop user, can you explain ... \n", "3 Who is PhotoSpin in relation to the image used... \n", "4 How you use Perspective Warp in Photoshop, wha... \n", "5 What does the Perspective Warp feature in Phot... \n", "6 As a Photoshop trainer developing step-by-step... \n", "7 wut is adobee fotoshop cretive clowd? \n", "8 Wut duz Perspectiv Warp do in Photoshop? \n", "9 How can I, as a Photoshop trainer, explain to ... \n", "\n", " retrieved_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [If I turn it on and off, you can see the befo... \n", "5 [If I turn it on and off, you can see the befo... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [If I turn it on and off, you can see the befo... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " reference_contexts \\\n", "0 [>> What I want to show you in this video is s... \n", "1 [>> What I want to show you in this video is s... \n", "2 [>> What I want to show you in this video is s... \n", "3 [>> What I want to show you in this video is s... \n", "4 [>> What I want to show you in this video is s... \n", "5 [>> What I want to show you in this video is s... \n", "6 [>> What I want to show you in this video is s... \n", "7 [>> What I want to show you in this video is s... \n", "8 [>> What I want to show you in this video is s... \n", "9 [>> What I want to show you in this video is s... \n", "\n", " response \\\n", "0 Here's how to use Perspective Warp in Photosho... \n", "1 Adobe Photoshop Creative Cloud is mentioned as... \n", "2 Adobe Photoshop Creative Cloud's Perspective W... \n", "3 PhotoSpin is the company that took the photogr... \n", "4 Perspective Warp in Photoshop is a tool that a... \n", "5 The Perspective Warp feature in Photoshop allo... \n", "6 Based on the transcript, here's how the Perspe... \n", "7 Adobe Photoshop Creative Cloud is a version of... \n", "8 Perspective Warp in Photoshop allows you to ch... \n", "9 In the Perspective Warp tutorial, the PhotoSpi... \n", "\n", " reference context_recall \\\n", "0 in adobe photoshop creative cloud, to use pers... 0.300000 \n", "1 Adobe Photoshop Creative Cloud is a version of... 1.000000 \n", "2 Adobe Photoshop Creative Cloud's Perspective W... 0.400000 \n", "3 PhotoSpin is the company that took the photogr... 1.000000 \n", "4 Perspective Warp in Photoshop let you change t... 0.250000 \n", "5 Perspective Warp in Photoshop allows you to ch... 0.500000 \n", "6 The new Perspective Warp feature in Adobe Phot... 0.272727 \n", "7 Adobe Photoshop Creative Cloud is a version of... 1.000000 \n", "8 Perspective Warp in Photoshop lets yu change t... 1.000000 \n", "9 In the Perspective Warp tutorial, the image us... 1.000000 \n", "\n", " faithfulness factual_correctness(mode=f1) answer_relevancy \\\n", "0 0.250000 0.47 0.896976 \n", "1 0.600000 0.50 0.876445 \n", "2 1.000000 0.50 0.934480 \n", "3 0.500000 0.67 0.914422 \n", "4 0.800000 0.43 0.960246 \n", "5 0.666667 0.80 0.982226 \n", "6 1.000000 0.47 0.000000 \n", "7 1.000000 0.80 0.820373 \n", "8 0.666667 0.67 0.926159 \n", "9 1.000000 0.67 0.879323 \n", "\n", " context_entity_recall noise_sensitivity(mode=relevant) \n", "0 0.400000 0.062500 \n", "1 1.000000 0.600000 \n", "2 0.222222 0.000000 \n", "3 0.500000 0.000000 \n", "4 0.500000 0.200000 \n", "5 1.000000 0.333333 \n", "6 0.400000 0.300000 \n", "7 1.000000 0.000000 \n", "8 1.000000 0.333333 \n", "9 0.333333 0.400000 " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ft.result = evaluate(\n", " dataset=ft.dataset,\n", " metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness(), ResponseRelevancy(), ContextEntityRecall(), NoiseSensitivity()],\n", " llm=evaluator_llm,\n", " run_config=custom_run_config\n", ")\n", "ft.result.to_pandas()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Base\n", "[\">> What I want to show you in this video is something that is absolutely amazing. It's a brand new feature in Adobe Photoshop Creative Cloud called Perspective Warp. Now I have a photograph open. I didn't take this photo. It was taken by a company called PhotoSpin. And don't forget if you want to follow along, you can download the assets for this video. What I want to do first though is make a copy of it. I'm going to drag it down- this is one way to do it-make a copy. That is not necessary, but this way we get to see kind of a before and an after. Now it will work with just about any image, but your first test is to go up to the word Edit on the pull-down menu and go down, and you better see Perspective Warp. If you don't, no big deal. Just go out to the cloud, and download the latest version of Photoshop. Now what does it do? What does Perspective Warp do? It literally allows me to re-enter a three-dimensional world to change the perspective of the image as if, as the photographer, I change my position. Now we're going to do two things.\", \"If I turn it on and off, you can see the before and the after. Perspective Warp is an amazing tool, brand new to Photoshop CC, literally allowing you to get back into the third dimension and change the position of where the photograph was taken. Well, that's about it.\"]\n", "['>> What I want to show you in this video is something that is absolutely amazing. It\\'s a brand new feature in Adobe Photoshop Creative Cloud called Perspective Warp. Now I have a photograph open. I didn\\'t take this photo. It was taken by a company called PhotoSpin. And don\\'t forget if you want to follow along, you can download the assets for this video. What I want to do first though is make a copy of it. I\\'m going to drag it down- this is one way to do it-make a copy. That is not necessary, but this way we get to see kind of a before and an after. Now it will work with just about any image, but your first test is to go up to the word Edit on the pull-down menu and go down, and you better see Perspective Warp. If you don\\'t, no big deal. Just go out to the cloud, and download the latest version of Photoshop. Now what does it do? What does Perspective Warp do? It literally allows me to re-enter a three-dimensional world to change the perspective of the image as if, as the photographer, I change my position. Now we\\'re going to do two things. One is very easy. I\\'m going to straighten the buildings out. The other part\\'s more fun. Change the perspective. Now, we\\'ve got the tool selected. You do this by building grids, perspective grids. So if I click somewhere, you can see your first grid. We can get in the middle. Let\\'s talk about how this works. Get in the middle. Move it around as a whole. Go to a pin, move the pins just like that. If I go to the side, top, or bottom and drag, I can move it. But the trick is if you hold down the shift key at the same time and move it, it changes the perspective too. It can help you out. The other thing I do like is if you select a pin, you can use your arrow keys to gently nudge it into place or get on it, click it, delete it. Now we need two. We\\'re working with a three-dimensional item that\\'s been compressed two dimensionally. We need two planes on this, and we\\'re going to use this right here as our dividing line. Click and drag this time. Save you some time in moving it around. I\\'m going to leave it right there. I\\'m going to come up here and click this one and move it over. What I want to do is grab and select that line. Now down here, I think I\\'ll use my arrow keys. Do you have to do this? Not really. But if you do, if you spend the time to get these lines down right based on perspective of the image, it\\'s going to save you time later on. So I\\'m going to come over here and grab this one. I\\'m going to drag it down. Now I can use these lines to match up with these, and that\\'s what I\\'m doing. Let\\'s do this one down here too. So what I\\'m trying to do here is just match those up. Let\\'s come down just a little bit more, just about like that. That doesn\\'t look too bad. Now we need the other side of our box, if you will. So if I click and drag, I can make another one of these. Now here\\'s the fun part. Grab it right here, and drag it over. Don\\'t hold the shift key. Just drag it over until you see a blue line up here. Let go. Pin to pin, it will match it. We can now do the other side. So what I\\'m going to do here is bring that over and again, I can use these lines here against those lines there. And I\\'m going to try to line that up as best I can, come down to this one obviously and bring that one up just about like that. Again, don\\'t forget you can use your arrow keys if you want it precise. That\\'s not too bad. What I want to do is extend it out. So hold the shift key before you drag it just like that. All right. Now we\\'ve got the box built around the buildings or whatever the object is. Come over here and click Warp. Now when we move the pins, it will change it. Now since I took the time to get the angles and the first thing I want to do maybe is straighten it out. That\\'s going to be much easier. Click this button right here and watch what happens. Well it begins to straighten it out, but the pins are still independent. In other words, I can come over to this one and still move it if I wanted to. if I don\\'t like what it did. I find most of the time that if I spend the time to get the perspective, it does help things out. Now you have another one that does your horizontals if you want to do that. We don\\'t. This one does both. This one says, \"you know what? I so messed this up. I wish I could start all over again.\" And if you click it, it will take you right back to where you were. You say, \"I want to redesign the whole thing,\" you can click here. Go right back to Layout. Let\\'s go back here, and let\\'s click that button again because that did work. Now that\\'s easy. In fact you might not even need Perspective Warp to straighten a building. What I love about it is changing the perspective. Now if I come down to this point right here and move it, these are still independent, and here\\'s the trick. Come over to this line, hold the shift key down until it turns yellow, and then click on it. Now that does two things. If it']\n", "FT\n", "[\">> What I want to show you in this video is something that is absolutely amazing. It's a brand new feature in Adobe Photoshop Creative Cloud called Perspective Warp. Now I have a photograph open. I didn't take this photo. It was taken by a company called PhotoSpin. And don't forget if you want to follow along, you can download the assets for this video. What I want to do first though is make a copy of it. I'm going to drag it down- this is one way to do it-make a copy. That is not necessary, but this way we get to see kind of a before and an after. Now it will work with just about any image, but your first test is to go up to the word Edit on the pull-down menu and go down, and you better see Perspective Warp. If you don't, no big deal. Just go out to the cloud, and download the latest version of Photoshop. Now what does it do? What does Perspective Warp do? It literally allows me to re-enter a three-dimensional world to change the perspective of the image as if, as the photographer, I change my position. Now we're going to do two things.\", \"If I turn it on and off, you can see the before and the after. Perspective Warp is an amazing tool, brand new to Photoshop CC, literally allowing you to get back into the third dimension and change the position of where the photograph was taken. Well, that's about it.\"]\n", "['>> What I want to show you in this video is something that is absolutely amazing. It\\'s a brand new feature in Adobe Photoshop Creative Cloud called Perspective Warp. Now I have a photograph open. I didn\\'t take this photo. It was taken by a company called PhotoSpin. And don\\'t forget if you want to follow along, you can download the assets for this video. What I want to do first though is make a copy of it. I\\'m going to drag it down- this is one way to do it-make a copy. That is not necessary, but this way we get to see kind of a before and an after. Now it will work with just about any image, but your first test is to go up to the word Edit on the pull-down menu and go down, and you better see Perspective Warp. If you don\\'t, no big deal. Just go out to the cloud, and download the latest version of Photoshop. Now what does it do? What does Perspective Warp do? It literally allows me to re-enter a three-dimensional world to change the perspective of the image as if, as the photographer, I change my position. Now we\\'re going to do two things. One is very easy. I\\'m going to straighten the buildings out. The other part\\'s more fun. Change the perspective. Now, we\\'ve got the tool selected. You do this by building grids, perspective grids. So if I click somewhere, you can see your first grid. We can get in the middle. Let\\'s talk about how this works. Get in the middle. Move it around as a whole. Go to a pin, move the pins just like that. If I go to the side, top, or bottom and drag, I can move it. But the trick is if you hold down the shift key at the same time and move it, it changes the perspective too. It can help you out. The other thing I do like is if you select a pin, you can use your arrow keys to gently nudge it into place or get on it, click it, delete it. Now we need two. We\\'re working with a three-dimensional item that\\'s been compressed two dimensionally. We need two planes on this, and we\\'re going to use this right here as our dividing line. Click and drag this time. Save you some time in moving it around. I\\'m going to leave it right there. I\\'m going to come up here and click this one and move it over. What I want to do is grab and select that line. Now down here, I think I\\'ll use my arrow keys. Do you have to do this? Not really. But if you do, if you spend the time to get these lines down right based on perspective of the image, it\\'s going to save you time later on. So I\\'m going to come over here and grab this one. I\\'m going to drag it down. Now I can use these lines to match up with these, and that\\'s what I\\'m doing. Let\\'s do this one down here too. So what I\\'m trying to do here is just match those up. Let\\'s come down just a little bit more, just about like that. That doesn\\'t look too bad. Now we need the other side of our box, if you will. So if I click and drag, I can make another one of these. Now here\\'s the fun part. Grab it right here, and drag it over. Don\\'t hold the shift key. Just drag it over until you see a blue line up here. Let go. Pin to pin, it will match it. We can now do the other side. So what I\\'m going to do here is bring that over and again, I can use these lines here against those lines there. And I\\'m going to try to line that up as best I can, come down to this one obviously and bring that one up just about like that. Again, don\\'t forget you can use your arrow keys if you want it precise. That\\'s not too bad. What I want to do is extend it out. So hold the shift key before you drag it just like that. All right. Now we\\'ve got the box built around the buildings or whatever the object is. Come over here and click Warp. Now when we move the pins, it will change it. Now since I took the time to get the angles and the first thing I want to do maybe is straighten it out. That\\'s going to be much easier. Click this button right here and watch what happens. Well it begins to straighten it out, but the pins are still independent. In other words, I can come over to this one and still move it if I wanted to. if I don\\'t like what it did. I find most of the time that if I spend the time to get the perspective, it does help things out. Now you have another one that does your horizontals if you want to do that. We don\\'t. This one does both. This one says, \"you know what? I so messed this up. I wish I could start all over again.\" And if you click it, it will take you right back to where you were. You say, \"I want to redesign the whole thing,\" you can click here. Go right back to Layout. Let\\'s go back here, and let\\'s click that button again because that did work. Now that\\'s easy. In fact you might not even need Perspective Warp to straighten a building. What I love about it is changing the perspective. Now if I come down to this point right here and move it, these are still independent, and here\\'s the trick. Come over to this line, hold the shift key down until it turns yellow, and then click on it. Now that does two things. If it']\n" ] } ], "source": [ "print (\"Base\")\n", "print( base.dataset[0].retrieved_contexts )\n", "print( base.dataset[0].reference_contexts )\n", "print (\"FT\")\n", "print( ft.dataset[0].retrieved_contexts )\n", "print( ft.dataset[0].reference_contexts )\n" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "context_recall 0.672273\n", "faithfulness 0.748333\n", "factual_correctness(mode=f1) 0.598000\n", "answer_relevancy 0.819065\n", "context_entity_recall 0.635556\n", "noise_sensitivity(mode=relevant) 0.222917\n", "Name: Mean, dtype: float64\n", "context_recall 0.111424\n", "faithfulness 0.081742\n", "factual_correctness(mode=f1) 0.044392\n", "answer_relevancy 0.092153\n", "context_entity_recall 0.102262\n", "noise_sensitivity(mode=relevant) 0.064911\n", "Name: StdDev, dtype: float64\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/mbudisic/Documents/PsTuts-RAG/pstuts_rag/pstuts_rag/evaluator_utils.py:193: FutureWarning: errors='ignore' is deprecated and will raise in a future version. Use to_numeric without passing `errors` and catch exceptions explicitly instead\n", " retval = retval.apply(partial(pd.to_numeric, **{\"errors\": \"ignore\"}))\n" ] } ], "source": [ "ft.statistics = summary_stats(ft.result.to_pandas())\n", "print( ft.statistics.select_dtypes(include=\"number\") \\\n", " .loc[\"Mean\"] )\n", "print( ft.statistics.select_dtypes(include=\"number\") \\\n", ".loc[\"StdDev\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Statistics\n", "\n", "Let's now pool the results from base and ft models, and see if there is any\n", "significance to the result." ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "from typing import Tuple\n", "import pandas as pd\n", "from pstuts_rag.evaluator_utils import combine_stats, z_test, summary_stats\n", "# Create a new DataFrame with renamed index\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "means, stds = ( combine_stats( \n", " (base.statistics, ft.statistics), field, [\"Base\",\"FT\"] ) \n", " for field in [\"Mean\",\"StdDev\"] \n", " )\n", "summary_base_ft = pd.concat(\n", " [means, stds],\n", " axis=1,\n", " keys=[\"Mean\",\"StdDev\"]\n", ").swaplevel(axis=1).sort_index(axis=1)\n", "\n", "summary_base_ft" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we'll do a bit of statistics.\n", "\n", "Compute the z-test to determine if the mean has shifted significantly or not\n", "between base and FT.\n", "\n", "Small value (e.g. $< 0.05$) would indicate a statistically significant move. But\n", "we're simply looking for `p` value that stands out as smaller than the rest." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
z_scorep_value
answer_relevancy-1.0229500.306332
factual_correctness(mode=f1)-0.7744320.438675
noise_sensitivity(mode=relevant)0.3164800.751638
faithfulness0.2362450.813243
context_recall0.0000001.000000
context_entity_recall0.0000001.000000
\n", "
" ], "text/plain": [ " z_score p_value\n", "answer_relevancy -1.022950 0.306332\n", "factual_correctness(mode=f1) -0.774432 0.438675\n", "noise_sensitivity(mode=relevant) 0.316480 0.751638\n", "faithfulness 0.236245 0.813243\n", "context_recall 0.000000 1.000000\n", "context_entity_recall 0.000000 1.000000" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "\n", "significance = pd.DataFrame(columns=['z_score', 'p_value'])\n", "# The error occurs because df_combined has a MultiIndex with levels swapped\n", "# We need to access the data differently - the structure is (metric, stat) not (stat, metric)\n", "for c in summary_base_ft.columns.get_level_values(0).unique():\n", " z, p = z_test(summary_base_ft.loc['Base', (c, 'Mean')], \n", " summary_base_ft.loc['FT', (c, 'Mean')], \n", " summary_base_ft.loc['Base', (c, 'StdDev')], \n", " summary_base_ft.loc['FT', (c, 'StdDev')])\n", " significance.loc[c] = [z, p]\n", "\n", "significance = significance.sort_values(by='p_value')\n", "significance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What we see is that there is no difference in context recall.\n", "\n", "My guess is that this result has to do with the specific application.\n", "These were audio transcripts of fairly short videos. Most transcripts therefore\n", "fit completely into a single, or a few, chunks (nodes) of the knowledge graph\n", "used to generate the golden data set.\n", "\n", "At the same time, due to video diversity, transcripts were quite distinct from \n", "each other. Therefore, even a base embedding model likely did as good of a job\n", "as it could. Since the embedding models to train here were chosen so that they\n", "could be fine-tuned on a laptop, it's hard to get better.\n", "\n", "Let's see if we can test that by using a SOTA model (`text-embedding-3-small`) and see how well it does." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "23ccffcd788b4a939609fc7664e78627", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Evaluating: 0%| | 0/60 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
answer_relevancycontext_entity_recallcontext_recallfactual_correctness(mode=f1)faithfulnessnoise_sensitivity(mode=relevant)
MeanStdDevMeanStdDevMeanStdDevMeanStdDevMeanStdDevMeanStdDev
Base0.9144480.0142150.6355560.1022620.6722730.1114240.6540.0570810.7211110.0812150.1984720.041861
SOTA0.9133010.0137160.6733330.0917320.6722730.1114240.5330.0546510.6109070.0852140.1994870.047707
FT0.8190650.0921530.6355560.1022620.6722730.1114240.5980.0443920.7483330.0817420.2229170.064911
\n", "" ], "text/plain": [ " answer_relevancy context_entity_recall \\\n", " Mean StdDev Mean StdDev \n", "Base 0.914448 0.014215 0.635556 0.102262 \n", "SOTA 0.913301 0.013716 0.673333 0.091732 \n", "FT 0.819065 0.092153 0.635556 0.102262 \n", "\n", " context_recall factual_correctness(mode=f1) \\\n", " Mean StdDev Mean StdDev \n", "Base 0.672273 0.111424 0.654 0.057081 \n", "SOTA 0.672273 0.111424 0.533 0.054651 \n", "FT 0.672273 0.111424 0.598 0.044392 \n", "\n", " faithfulness noise_sensitivity(mode=relevant) \n", " Mean StdDev Mean StdDev \n", "Base 0.721111 0.081215 0.198472 0.041861 \n", "SOTA 0.610907 0.085214 0.199487 0.047707 \n", "FT 0.748333 0.081742 0.222917 0.064911 " ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fields = [\"Mean\",\"StdDev\"]\n", "summary = pd.concat(\n", " [combine_stats( (base.statistics, sota.statistics, ft.statistics), field, (\"Base\",\"SOTA\", \"FT\") ) \n", " for field in fields ],\n", " axis=1,\n", " keys=fields\n", ").swaplevel(axis=1).sort_index(axis=1)\n", "\n", "summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Is FT any better or worse than the SOTA model?" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
z_scorep_value
faithfulness-1.1638320.244492
answer_relevancy1.0114580.311797
factual_correctness(mode=f1)-0.9231760.355916
noise_sensitivity(mode=relevant)-0.2908440.771170
context_entity_recall0.2749940.783321
context_recall0.0000001.000000
\n", "
" ], "text/plain": [ " z_score p_value\n", "faithfulness -1.163832 0.244492\n", "answer_relevancy 1.011458 0.311797\n", "factual_correctness(mode=f1) -0.923176 0.355916\n", "noise_sensitivity(mode=relevant) -0.290844 0.771170\n", "context_entity_recall 0.274994 0.783321\n", "context_recall 0.000000 1.000000" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "significance = pd.DataFrame(columns=['z_score', 'p_value'])\n", "# The error occurs because df_combined has a MultiIndex with levels swapped\n", "# We need to access the data differently - the structure is (metric, stat) not (stat, metric)\n", "for c in summary.columns.get_level_values(0).unique():\n", " z, p = z_test(summary.loc['FT', (c, 'Mean')], \n", " summary.loc['SOTA', (c, 'Mean')], \n", " summary.loc['FT', (c, 'StdDev')], \n", " summary.loc['SOTA', (c, 'StdDev')])\n", " significance.loc[c] = [z, p]\n", "\n", "significance = significance.sort_values(by='p_value')\n", "significance\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks doubtful. If anything, swapping SOTA for FT makes the metrics go down.\n", "\n", "Notice that the context recall is still exactly the same.\n", "\n", "\n", "So, in the end, the conclusion is that the embedding model is not the\n", "right spot to optimize this RAG chain." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.11" } }, "nbformat": 4, "nbformat_minor": 2 }