add notebook for bonus unit 2 - agent evaluation (#62)

- add notebook for bonus unit 2 - agent evaluation (d29ce2619de5c91c4b5e580df7f5be7bc5961019)


Co-authored-by: Jannik <[email protected]>

bonus-unit2/monitoring-and-evaluating-agents.ipynb

@@ -0,0 +1,647 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Bonus Unit 1: Observability and Evaluation of Agents\n",
+        "\n",
+        "In this tutorial, we will learn how to **monitor the internal steps (traces) of our AI agent** and **evaluate its performance** using open-source observability tools.\n",
+        "\n",
+        "The ability to observe and evaluate an agent’s behavior is essential for:\n",
+        "- Debugging issues when tasks fail or produce suboptimal results\n",
+        "- Monitoring costs and performance in real-time\n",
+        "- Improving reliability and safety through continuous feedback\n",
+        "\n",
+        "This notebook is part of the [Hugging Face Agents Course](https://www.hf.co/learn/agents-course/unit1/introduction)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Exercise Prerequisites 🏗️\n",
+        "\n",
+        "Before running this notebook, please be sure you have:\n",
+        "\n",
+        "🔲 📚  **Studied** [Introduction to Agents](https://huggingface.co/learn/agents-course/unit1/introduction)\n",
+        "\n",
+        "🔲 📚  **Studied** [The smolagents framework](https://huggingface.co/learn/agents-course/unit2/smolagents/introduction)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 0: Install the Required Libraries\n",
+        "\n",
+        "We will need a few libraries that allow us to run, monitor, and evaluate our agents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%pip install 'smolagents[telemetry]'\n",
+        "%pip install opentelemetry-sdk opentelemetry-exporter-otlp openinference-instrumentation-smolagents\n",
+        "%pip install langfuse datasets 'smolagents[gradio]' gradio"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 1: Instrument Your Agent\n",
+        "\n",
+        "In this notebook, we will use [Langfuse](https://langfuse.com/) as our observability tool, but you can use **any other OpenTelemetry-compatible service**. The code below shows how to set environment variables for Langfuse (or any OTel endpoint) and how to instrument your smolagent.\n",
+        "\n",
+        "**Note:** If you are using LlamaIndex or LangGraph, you can find documentation on instrumenting them [here](https://langfuse.com/docs/integrations/llama-index/workflows) and [here](https://langfuse.com/docs/integrations/langchain/example-python-langgraph). "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import base64\n",
+        "\n",
+        "# Get your own keys from https://cloud.langfuse.com\n",
+        "os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"pk-lf-...\" \n",
+        "os.environ[\"LANGFUSE_SECRET_KEY\"] = \"sk-lf-...\" \n",
+        "os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\"  # 🇪🇺 EU region example\n",
+        "# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\"  # 🇺🇸 US region example\n",
+        "\n",
+        "LANGFUSE_AUTH = base64.b64encode(\n",
+        "    f\"{os.environ.get('LANGFUSE_PUBLIC_KEY')}:{os.environ.get('LANGFUSE_SECRET_KEY')}\".encode()\n",
+        ").decode()\n",
+        "\n",
+        "os.environ[\"OTEL_EXPORTER_OTLP_ENDPOINT\"] = os.environ.get(\"LANGFUSE_HOST\") + \"/api/public/otel\"\n",
+        "os.environ[\"OTEL_EXPORTER_OTLP_HEADERS\"] = f\"Authorization=Basic {LANGFUSE_AUTH}\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Set your Hugging Face and other tokens/secrets as environment variable\n",
+        "os.environ[\"HF_TOKEN\"] = \"hf_...\" "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from opentelemetry.sdk.trace import TracerProvider\n",
+        "from openinference.instrumentation.smolagents import SmolagentsInstrumentor\n",
+        "from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n",
+        "from opentelemetry.sdk.trace.export import SimpleSpanProcessor\n",
+        " \n",
+        "# Create a TracerProvider for OpenTelemetry\n",
+        "trace_provider = TracerProvider()\n",
+        "\n",
+        "# Add a SimpleSpanProcessor with the OTLPSpanExporter to send traces\n",
+        "trace_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter()))\n",
+        "\n",
+        "# Set the global default tracer provider\n",
+        "from opentelemetry import trace\n",
+        "trace.set_tracer_provider(trace_provider)\n",
+        "tracer = trace.get_tracer(__name__)\n",
+        "\n",
+        "# Instrument smolagents with the configured provider\n",
+        "SmolagentsInstrumentor().instrument(tracer_provider=trace_provider)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 2: Test Your Instrumentation\n",
+        "\n",
+        "Here is a simple CodeAgent from smolagents that calculates `1+1`. We run it to confirm that the instrumentation is working correctly. If everything is set up correctly, you will see logs/spans in your observability dashboard."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from smolagents import HfApiModel, CodeAgent\n",
+        "\n",
+        "# Create a simple agent to test instrumentation\n",
+        "agent = CodeAgent(\n",
+        "    tools=[],\n",
+        "    model=HfApiModel()\n",
+        ")\n",
+        "\n",
+        "agent.run(\"1+1=\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Check your [Langfuse Traces Dashboard](https://cloud.langfuse.com/traces) (or your chosen observability tool) to confirm that the spans and logs have been recorded.\n",
+        "\n",
+        "Example screenshot from Langfuse:\n",
+        "\n",
+        "![Example trace in Langfuse](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/first-example-trace.png)\n",
+        "\n",
+        "_[Link to the trace](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/1b94d6888258e0998329cdb72a371155?timestamp=2025-03-10T11%3A59%3A41.743Z)_"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 3: Observe and Evaluate a More Complex Agent\n",
+        "\n",
+        "Now that you have confirmed your instrumentation works, let's try a more complex query so we can see how advanced metrics (token usage, latency, costs, etc.) are tracked."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from smolagents import (CodeAgent, DuckDuckGoSearchTool, HfApiModel)\n",
+        "\n",
+        "search_tool = DuckDuckGoSearchTool()\n",
+        "agent = CodeAgent(tools=[search_tool], model=HfApiModel())\n",
+        "\n",
+        "agent.run(\"How many Rubik's Cubes could you fit inside the Notre Dame Cathedral?\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Trace Structure\n",
+        "\n",
+        "Most observability tools record a **trace** that contains **spans**, which represent each step of your agent’s logic. Here, the trace contains the overall agent run and sub-spans for:\n",
+        "- The tool calls (DuckDuckGoSearchTool)\n",
+        "- The LLM calls (HfApiModel)\n",
+        "\n",
+        "You can inspect these to see precisely where time is spent, how many tokens are used, and so on:\n",
+        "\n",
+        "![Trace tree in Langfuse](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/trace-tree.png)\n",
+        "\n",
+        "_[Link to the trace](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/1ac33b89ffd5e75d4265b62900c348ed?timestamp=2025-03-07T13%3A45%3A09.149Z&display=preview)_"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Online Evaluation\n",
+        "\n",
+        "In the previous section, we learned about the difference between online and offline evaluation. Now, we will see how to monitor your agent in production and evaluate it live.\n",
+        "\n",
+        "### Common Metrics to Track in Production\n",
+        "\n",
+        "1. **Costs** — The smolagents instrumentation captures token usage, which you can transform into approximate costs by assigning a price per token.\n",
+        "2. **Latency** — Observe the time it takes to complete each step, or the entire run.\n",
+        "3. **User Feedback** — Users can provide direct feedback (thumbs up/down) to help refine or correct the agent.\n",
+        "4. **LLM-as-a-Judge** — Use a separate LLM to evaluate your agent’s output in near real-time (e.g., checking for toxicity or correctness).\n",
+        "\n",
+        "Below, we show examples of these metrics."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 1. Costs\n",
+        "\n",
+        "Below is a screenshot showing usage for `Qwen2.5-Coder-32B-Instruct` calls. This is useful to see costly steps and optimize your agent. \n",
+        "\n",
+        "![Costs](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/smolagents-costs.png)\n",
+        "\n",
+        "_[Link to the trace](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/1ac33b89ffd5e75d4265b62900c348ed?timestamp=2025-03-07T13%3A45%3A09.149Z&display=preview)_"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 2. Latency\n",
+        "\n",
+        "We can also see how long it took to complete each step. In the example below, the entire conversation took 32 seconds, which you can break down by step. This helps you identify bottlenecks and optimize your agent.\n",
+        "\n",
+        "![Latency](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/smolagents-latency.png)\n",
+        "\n",
+        "_[Link to the trace](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/1ac33b89ffd5e75d4265b62900c348ed?timestamp=2025-03-07T13%3A45%3A09.149Z&display=preview)_"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 3. Additional Attributes\n",
+        "\n",
+        "You may also pass additional attributes—such as user IDs, session IDs, or tags—by setting them on the spans. For example, smolagents instrumentation uses OpenTelemetry to attach attributes like `langfuse.user.id` or custom tags."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from smolagents import (CodeAgent, DuckDuckGoSearchTool, HfApiModel)\n",
+        "from opentelemetry import trace\n",
+        "\n",
+        "search_tool = DuckDuckGoSearchTool()\n",
+        "agent = CodeAgent(\n",
+        "    tools=[search_tool],\n",
+        "    model=HfApiModel()\n",
+        ")\n",
+        "\n",
+        "with tracer.start_as_current_span(\"Smolagent-Trace\") as span:\n",
+        "    span.set_attribute(\"langfuse.user.id\", \"smolagent-user-123\")\n",
+        "    span.set_attribute(\"langfuse.session.id\", \"smolagent-session-123456789\")\n",
+        "    span.set_attribute(\"langfuse.tags\", [\"city-question\", \"testing-agents\"])\n",
+        "\n",
+        "    agent.run(\"What is the capital of Germany?\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Enhancing agent runs with additional metrics](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/smolagents-attributes.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 4. User Feedback\n",
+        "\n",
+        "If your agent is embedded into a user interface, you can record direct user feedback (like a thumbs-up/down in a chat UI). Below is an example using [Gradio](https://gradio.app/) to embed a chat with a simple feedback mechanism.\n",
+        "\n",
+        "In the code snippet below, when a user sends a chat message, we capture the OpenTelemetry trace ID. If the user likes/dislikes the last answer, we attach a score to the trace."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import gradio as gr\n",
+        "from opentelemetry.trace import format_trace_id\n",
+        "from smolagents import (CodeAgent, HfApiModel)\n",
+        "from langfuse import Langfuse\n",
+        "\n",
+        "langfuse = Langfuse()\n",
+        "model = HfApiModel()\n",
+        "agent = CodeAgent(tools=[], model=model, add_base_tools=True)\n",
+        "\n",
+        "formatted_trace_id = None  # We'll store the current trace_id globally for demonstration\n",
+        "\n",
+        "def respond(prompt, history):\n",
+        "    with trace.get_tracer(__name__).start_as_current_span(\"Smolagent-Trace\") as span:\n",
+        "        output = agent.run(prompt)\n",
+        "\n",
+        "        current_span = trace.get_current_span()\n",
+        "        span_context = current_span.get_span_context()\n",
+        "        trace_id = span_context.trace_id\n",
+        "        global formatted_trace_id\n",
+        "        formatted_trace_id = str(format_trace_id(trace_id))\n",
+        "        langfuse.trace(id=formatted_trace_id, input=prompt, output=output)\n",
+        "\n",
+        "    history.append({\"role\": \"assistant\", \"content\": str(output)})\n",
+        "    return history\n",
+        "\n",
+        "def handle_like(data: gr.LikeData):\n",
+        "    # For demonstration, we map user feedback to a 1 (like) or 0 (dislike)\n",
+        "    if data.liked:\n",
+        "        langfuse.score(\n",
+        "            value=1,\n",
+        "            name=\"user-feedback\",\n",
+        "            trace_id=formatted_trace_id\n",
+        "        )\n",
+        "    else:\n",
+        "        langfuse.score(\n",
+        "            value=0,\n",
+        "            name=\"user-feedback\",\n",
+        "            trace_id=formatted_trace_id\n",
+        "        )\n",
+        "\n",
+        "with gr.Blocks() as demo:\n",
+        "    chatbot = gr.Chatbot(label=\"Chat\", type=\"messages\")\n",
+        "    prompt_box = gr.Textbox(placeholder=\"Type your message...\", label=\"Your message\")\n",
+        "\n",
+        "    # When the user presses 'Enter' on the prompt, we run 'respond'\n",
+        "    prompt_box.submit(\n",
+        "        fn=respond,\n",
+        "        inputs=[prompt_box, chatbot],\n",
+        "        outputs=chatbot\n",
+        "    )\n",
+        "\n",
+        "    # When the user clicks a 'like' button on a message, we run 'handle_like'\n",
+        "    chatbot.like(handle_like, None, None)\n",
+        "\n",
+        "demo.launch()\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "User feedback is then captured in your observability tool:\n",
+        "\n",
+        "![User feedback is being captured in Langfuse](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/user-feedback-gradio.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 5. LLM-as-a-Judge\n",
+        "\n",
+        "LLM-as-a-Judge is another way to automatically evaluate your agent's output. You can set up a separate LLM call to gauge the output’s correctness, toxicity, style, or any other criteria you care about.\n",
+        "\n",
+        "**Workflow**:\n",
+        "1. You define an **Evaluation Template**, e.g., \"Check if the text is toxic.\"\n",
+        "2. Each time your agent generates output, you pass that output to your \"judge\" LLM with the template.\n",
+        "3. The judge LLM responds with a rating or label that you log to your observability tool.\n",
+        "\n",
+        "Example from Langfuse:\n",
+        "\n",
+        "![LLM-as-a-Judge Evaluation Template](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/evaluator-template.png)\n",
+        "![LLM-as-a-Judge Evaluator](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/evaluator.png)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Example: Checking if the agent’s output is toxic or not.\n",
+        "from smolagents import (CodeAgent, DuckDuckGoSearchTool, HfApiModel)\n",
+        "\n",
+        "search_tool = DuckDuckGoSearchTool()\n",
+        "agent = CodeAgent(tools=[search_tool], model=HfApiModel())\n",
+        "\n",
+        "agent.run(\"Can eating carrots improve your vision?\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "You can see that the answer of this example is judged as \"not toxic\".\n",
+        "\n",
+        "![LLM-as-a-Judge Evaluation Score](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/llm-as-a-judge-score.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 6. Observability Metrics Overview\n",
+        "\n",
+        "All of these metrics can be visualized together in dashboards. This enables you to quickly see how your agent performs across many sessions and helps you to track quality metrics over time.\n",
+        "\n",
+        "![Observability metrics overview](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/langfuse-dashboard.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Offline Evaluation\n",
+        "\n",
+        "Online evaluation is essential for live feedback, but you also need **offline evaluation**—systematic checks before or during development. This helps maintain quality and reliability before rolling changes into production."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Dataset Evaluation\n",
+        "\n",
+        "In offline evaluation, you typically:\n",
+        "1. Have a benchmark dataset (with prompt and expected output pairs)\n",
+        "2. Run your agent on that dataset\n",
+        "3. Compare outputs to the expected results or use an additional scoring mechanism\n",
+        "\n",
+        "Below, we demonstrate this approach with the [GSM8K dataset](https://huggingface.co/datasets/gsm8k), which contains math questions and solutions."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import pandas as pd\n",
+        "from datasets import load_dataset\n",
+        "\n",
+        "# Fetch GSM8K from Hugging Face\n",
+        "dataset = load_dataset(\"openai/gsm8k\", 'main', split='train')\n",
+        "df = pd.DataFrame(dataset)\n",
+        "print(\"First few rows of GSM8K dataset:\")\n",
+        "print(df.head())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Next, we create a dataset entity in Langfuse to track the runs. Then, we add each item from the dataset to the system. (If you’re not using Langfuse, you might simply store these in your own database or local file for analysis.)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from langfuse import Langfuse\n",
+        "langfuse = Langfuse()\n",
+        "\n",
+        "langfuse_dataset_name = \"gsm8k_dataset_huggingface\"\n",
+        "\n",
+        "# Create a dataset in Langfuse\n",
+        "langfuse.create_dataset(\n",
+        "    name=langfuse_dataset_name,\n",
+        "    description=\"GSM8K benchmark dataset uploaded from Huggingface\",\n",
+        "    metadata={\n",
+        "        \"date\": \"2025-03-10\", \n",
+        "        \"type\": \"benchmark\"\n",
+        "    }\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "for idx, row in df.iterrows():\n",
+        "    langfuse.create_dataset_item(\n",
+        "        dataset_name=langfuse_dataset_name,\n",
+        "        input={\"text\": row[\"question\"]},\n",
+        "        expected_output={\"text\": row[\"answer\"]},\n",
+        "        metadata={\"source_index\": idx}\n",
+        "    )\n",
+        "    if idx >= 9: # Upload only the first 10 items for demonstration\n",
+        "        break"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Dataset items in Langfuse](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/example-dataset.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Running the Agent on the Dataset\n",
+        "\n",
+        "We define a helper function `run_smolagent()` that:\n",
+        "1. Starts an OpenTelemetry span\n",
+        "2. Runs our agent on the prompt\n",
+        "3. Records the trace ID in Langfuse\n",
+        "\n",
+        "Then, we loop over each dataset item, run the agent, and link the trace to the dataset item. We can also attach a quick evaluation score if desired."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from opentelemetry.trace import format_trace_id\n",
+        "from smolagents import (CodeAgent, HfApiModel, LiteLLMModel)\n",
+        "\n",
+        "# Example: using HfApiModel or LiteLLMModel to access openai, anthropic, gemini, etc. models:\n",
+        "model = HfApiModel()\n",
+        "\n",
+        "agent = CodeAgent(\n",
+        "    tools=[],\n",
+        "    model=model,\n",
+        "    add_base_tools=True\n",
+        ")\n",
+        "\n",
+        "def run_smolagent(question):\n",
+        "    with tracer.start_as_current_span(\"Smolagent-Trace\") as span:\n",
+        "        span.set_attribute(\"langfuse.tag\", \"dataset-run\")\n",
+        "        output = agent.run(question)\n",
+        "\n",
+        "        current_span = trace.get_current_span()\n",
+        "        span_context = current_span.get_span_context()\n",
+        "        trace_id = span_context.trace_id\n",
+        "        formatted_trace_id = format_trace_id(trace_id)\n",
+        "\n",
+        "        langfuse_trace = langfuse.trace(\n",
+        "            id=formatted_trace_id, \n",
+        "            input=question, \n",
+        "            output=output\n",
+        "        )\n",
+        "    return langfuse_trace, output"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dataset = langfuse.get_dataset(langfuse_dataset_name)\n",
+        "\n",
+        "# Run our agent against each dataset item (limited to first 10 above)\n",
+        "for item in dataset.items:\n",
+        "    langfuse_trace, output = run_smolagent(item.input[\"text\"])\n",
+        "\n",
+        "    # Link the trace to the dataset item for analysis\n",
+        "    item.link(\n",
+        "        langfuse_trace,\n",
+        "        run_name=\"smolagent-notebook-run-01\",\n",
+        "        run_metadata={ \"model\": model.model_id }\n",
+        "    )\n",
+        "\n",
+        "    # Optionally, store a quick evaluation score for demonstration\n",
+        "    langfuse_trace.score(\n",
+        "        name=\"<example_eval>\",\n",
+        "        value=1,\n",
+        "        comment=\"This is a comment\"\n",
+        "    )\n",
+        "\n",
+        "# Flush data to ensure all telemetry is sent\n",
+        "langfuse.flush()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "You can repeat this process with different:\n",
+        "- Models (OpenAI GPT, local LLM, etc.)\n",
+        "- Tools (search vs. no search)\n",
+        "- Prompts (different system messages)\n",
+        "\n",
+        "Then compare them side-by-side in your observability tool:\n",
+        "\n",
+        "![Dataset run overview](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/dataset_runs.png)\n",
+        "![Dataset run comparison](https://huggingface.co/datasets/agents-course/course-images/resolve/main/en/bonus-unit2/dataset-run-comparison.png)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Final Thoughts\n",
+        "\n",
+        "In this notebook, we covered how to:\n",
+        "1. **Set up Observability** using smolagents + OpenTelemetry exporters\n",
+        "2. **Check Instrumentation** by running a simple agent\n",
+        "3. **Capture Detailed Metrics** (cost, latency, etc.) through an observability tools\n",
+        "4. **Collect User Feedback** via a Gradio interface\n",
+        "5. **Use LLM-as-a-Judge** to automatically evaluate outputs\n",
+        "6. **Perform Offline Evaluation** with a benchmark dataset\n",
+        "\n",
+        "🤗 Happy coding!"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.13.2"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}