Bhumi14 commited on
Commit
c062906
ยท
verified ยท
1 Parent(s): 886a417

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +236 -236
README.md CHANGED
@@ -1,236 +1,236 @@
1
- ---
2
- title: GAIA Agent - Q&A Chatbot
3
- emoji: ๐Ÿค–
4
- colorFrom: green
5
- colorTo: blue
6
- sdk: gradio
7
- sdk_version: 5.25.2
8
- app_file: app.py
9
- pinned: false
10
- hf_oauth: true
11
- # optional, default duration is 8 hours/480 minutes. Max duration is 30 days/43200 minutes.
12
- hf_oauth_expiration_minutes: 480
13
- ---
14
-
15
- # ๐Ÿค– **GAIA Agent - Advanced Q&A Chatbot**
16
-
17
- ## ๐ŸŒŸ **Introduction**
18
-
19
- **GAIA Agent** is a sophisticated AI-powered chatbot system designed to handle complex questions and tasks through an intuitive Q&A interface. Built on top of the GAIA benchmark framework, this agent combines advanced reasoning, code execution, web search, document processing, and multimodal understanding capabilities. The system features both a user-friendly chatbot interface and a comprehensive evaluation runner for benchmark testing.
20
-
21
- ## ๐Ÿš€ **Key Features**
22
-
23
- - **๐Ÿ” Multi-Modal Search**: Web search, Wikipedia, and arXiv paper search
24
- - **๐Ÿ’ป Code Execution**: Support for Python, Bash, SQL, C, and Java
25
- - **๐Ÿ–ผ๏ธ Image Processing**: Analysis, transformation, OCR, and generation
26
- - **๐Ÿ“„ Document Processing**: PDF, CSV, Excel, and text file analysis
27
- - **๐Ÿ“ File Upload Support**: Handle multiple file types with drag-and-drop
28
- - **๐Ÿงฎ Mathematical Operations**: Complete set of mathematical tools
29
- - **๐Ÿ’ฌ Conversational Interface**: Natural chat-based interaction
30
- - **๐Ÿ“Š Evaluation System**: Automated benchmark testing and submission
31
-
32
- ## ๐Ÿ—๏ธ **Project Structure**
33
-
34
- ```
35
- gaia-agent/
36
- โ”œโ”€โ”€ app.py # Main Q&A chatbot interface
37
- โ”œโ”€โ”€ evaluation_app.py # GAIA benchmark evaluation runner
38
- โ”œโ”€โ”€ agent.py # Core agent implementation with tools
39
- โ”œโ”€โ”€ code_interpreter.py # Multi-language code execution
40
- โ”œโ”€โ”€ image_processing.py # Image processing utilities
41
- โ”œโ”€โ”€ system_prompt.txt # System prompt for the agent
42
- โ”œโ”€โ”€ requirements.txt # Python dependencies
43
- โ”œโ”€โ”€ metadata.jsonl # GAIA benchmark metadata
44
- โ”œโ”€โ”€ explore_metadata.ipynb # Data exploration notebook
45
- โ””โ”€โ”€ README.md # This file
46
- ```
47
-
48
- ## ๐Ÿ› ๏ธ **Tool Categories**
49
-
50
- ### **๐ŸŒ Browser & Search Tools**
51
- - **Wikipedia Search**: Search Wikipedia with up to 2 results
52
- - **Web Search**: Tavily-powered web search with up to 3 results
53
- - **arXiv Search**: Academic paper search with up to 3 results
54
-
55
- ### **๐Ÿ’ป Code Interpreter Tools**
56
- - **Multi-Language Execution**: Python, Bash, SQL, C, Java support
57
- - **Plot Generation**: Matplotlib visualization support
58
- - **DataFrame Analysis**: Pandas data processing
59
- - **Error Handling**: Comprehensive error reporting
60
-
61
- ### **๐Ÿงฎ Mathematical Tools**
62
- - **Basic Operations**: Add, subtract, multiply, divide
63
- - **Advanced Functions**: Modulus, power, square root
64
- - **Complex Numbers**: Support for complex number operations
65
-
66
- ### **๐Ÿ“„ Document Processing Tools**
67
- - **File Operations**: Save, read, and download files
68
- - **CSV Analysis**: Pandas-based data analysis
69
- - **Excel Processing**: Excel file analysis and processing
70
- - **OCR**: Extract text from images using Tesseract
71
-
72
- ### **๐Ÿ–ผ๏ธ Image Processing & Generation Tools**
73
- - **Image Analysis**: Size, color, and property analysis
74
- - **Transformations**: Resize, rotate, crop, flip, adjust brightness/contrast
75
- - **Drawing Tools**: Add shapes, text, and annotations
76
- - **Image Generation**: Create gradients, noise patterns, and simple graphics
77
- - **Image Combination**: Stack and combine multiple images
78
-
79
- ## ๐ŸŽฏ **How to Use**
80
-
81
- ### **Q&A Chatbot Interface (app.py)**
82
-
83
- 1. **Start the Chatbot:**
84
- ```bash
85
- python app.py
86
- ```
87
-
88
- 2. **Access the Interface:**
89
- - Open `http://localhost:7860` in your browser
90
- - Upload files (images, documents, CSV, etc.) if needed
91
- - Ask questions in natural language
92
- - Get comprehensive answers with tool usage
93
-
94
- 3. **Supported Interactions:**
95
- - **Text Questions**: "What is the capital of France?"
96
- - **Math Problems**: "Calculate the square root of 144"
97
- - **Code Requests**: "Write a Python function to sort a list"
98
- - **Image Analysis**: Upload an image and ask "What do you see?"
99
- - **Data Analysis**: Upload a CSV and ask "What are the trends?"
100
- - **Web Search**: "What are the latest AI developments?"
101
-
102
- ### **Evaluation Runner (evaluation_app.py)**
103
-
104
- 1. **Run the Evaluation:**
105
- ```bash
106
- python evaluation_app.py
107
- ```
108
-
109
- 2. **Benchmark Testing:**
110
- - Log in with your Hugging Face account
111
- - Click "Run Evaluation & Submit All Answers"
112
- - Monitor progress as the agent processes GAIA benchmark questions
113
- - View results and scores automatically
114
-
115
- ## ๐Ÿ”ง **Technical Architecture**
116
-
117
- ### **LangGraph State Machine**
118
- ```
119
- START โ†’ Retriever โ†’ Assistant โ†’ Tools โ†’ Assistant
120
- โ†‘ โ†“
121
- โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
122
- ```
123
-
124
- 1. **Retriever Node**: Searches vector database for similar questions
125
- 2. **Assistant Node**: LLM processes question with available tools
126
- 3. **Tools Node**: Executes selected tools (web search, code, etc.)
127
- 4. **Conditional Routing**: Dynamically routes between assistant and tools
128
-
129
- ### **Vector Database Integration**
130
- - **Supabase Vector Store**: Stores GAIA benchmark Q&A pairs
131
- - **Semantic Search**: Finds similar questions for context
132
- - **HuggingFace Embeddings**: sentence-transformers/all-mpnet-base-v2
133
-
134
- ### **Multi-Modal File Support**
135
- - **Images**: JPG, PNG, GIF, BMP, WebP
136
- - **Documents**: PDF, DOC, DOCX, TXT, MD
137
- - **Data**: CSV, Excel, JSON
138
- - **Code**: Python, Bash, SQL, C, Java
139
-
140
- ## โš™๏ธ **Installation & Setup**
141
-
142
- ### **1. Clone Repository**
143
- ```bash
144
- git clone https://github.com/fisherman611/gaia-agent.git
145
- cd gaia-agent
146
- ```
147
-
148
- ### **2. Install Dependencies**
149
- ```bash
150
- pip install -r requirements.txt
151
- ```
152
-
153
- ### **3. Environment Variables**
154
- Create a `.env` file with your API keys:
155
- ```env
156
- SUPABASE_URL=your_supabase_url
157
- SUPABASE_SERVICE_ROLE_KEY=your_supabase_key
158
- GROQ_API_KEY=your_groq_api_key
159
- TAVILY_API_KEY=your_tavily_api_key
160
- HUGGINGFACEHUB_API_TOKEN=your_hf_token
161
- LANGSMITH_API_KEY=your_langsmith_key
162
-
163
- LANGSMITH_TRACING=true
164
- LANGSMITH_PROJECT=ai_agent_course
165
- LANGSMITH_ENDPOINT=https://api.smith.langchain.com
166
- ```
167
-
168
- ### **4. Database Setup (Supabase)**
169
- Execute this SQL in your Supabase database:
170
- ```sql
171
- -- Enable pgvector extension
172
- CREATE EXTENSION IF NOT EXISTS vector;
173
-
174
- -- Create match function for documents2 table
175
- CREATE OR REPLACE FUNCTION public.match_documents_2(
176
- query_embedding vector(768)
177
- )
178
- RETURNS TABLE(
179
- id bigint,
180
- content text,
181
- metadata jsonb,
182
- embedding vector(768),
183
- similarity double precision
184
- )
185
- LANGUAGE sql STABLE
186
- AS $$
187
- SELECT
188
- id,
189
- content,
190
- metadata,
191
- embedding,
192
- 1 - (embedding <=> query_embedding) AS similarity
193
- FROM public.documents2
194
- ORDER BY embedding <=> query_embedding
195
- LIMIT 10;
196
- $$;
197
-
198
- -- Grant permissions
199
- GRANT EXECUTE ON FUNCTION public.match_documents_2(vector) TO anon, authenticated;
200
- ```
201
-
202
- ## ๐Ÿš€ **Running the Application**
203
-
204
- ### **Chatbot Interface**
205
- ```bash
206
- python app.py
207
- ```
208
- Access at: `http://localhost:7860`
209
-
210
- ### **Evaluation Runner**
211
- ```bash
212
- python evaluation_app.py
213
- ```
214
- Access at: `http://localhost:7860`
215
-
216
- ### **Live Demo**
217
- Try it online: [Hugging Face Space](https://huggingface.co/spaces/fisherman611/gaia-agent)
218
-
219
- ## ๐Ÿ”— **Resources**
220
-
221
- - [GAIA Benchmark](https://huggingface.co/spaces/gaia-benchmark/leaderboard)
222
- - [Hugging Face Agents Course](https://huggingface.co/agents-course)
223
- - [LangGraph Documentation](https://langchain-ai.github.io/langgraph/)
224
- - [Supabase Vector Store](https://supabase.com/docs/guides/ai/vector-columns)
225
-
226
- ## ๐Ÿค **Contributing**
227
-
228
- Contributions are welcome! Areas for improvement:
229
- - **New Tools**: Add specialized tools for specific domains
230
- - **UI Enhancements**: Improve the chatbot interface
231
- - **Performance**: Optimize response times and accuracy
232
- - **Documentation**: Expand examples and use cases
233
-
234
- ## ๐Ÿ“„ **License**
235
-
236
- This project is licensed under the [MIT License](https://mit-license.org/).
 
1
+ ---
2
+ title: GAIA Agent - Q&A Chatbot
3
+ emoji: ๐Ÿค–
4
+ colorFrom: green
5
+ colorTo: blue
6
+ sdk: gradio
7
+ sdk_version: 5.25.2
8
+ app_file: app.py
9
+ pinned: false
10
+ hf_oauth: true
11
+ # optional, default duration is 8 hours/480 minutes. Max duration is 30 days/43200 minutes.
12
+ hf_oauth_expiration_minutes: 480
13
+ ---
14
+
15
+ # ๐Ÿค– **GAIA Agent - Advanced Q&A Chatbot**
16
+
17
+ ## ๐ŸŒŸ **Introduction**
18
+
19
+ **GAIA Agent** is a sophisticated AI-powered chatbot system designed to handle complex questions and tasks through an intuitive Q&A interface. Built on top of the GAIA benchmark framework, this agent combines advanced reasoning, code execution, web search, document processing, and multimodal understanding capabilities. The system features both a user-friendly chatbot interface and a comprehensive evaluation runner for benchmark testing.
20
+
21
+ ## ๐Ÿš€ **Key Features**
22
+
23
+ - **๐Ÿ” Multi-Modal Search**: Web search, Wikipedia, and arXiv paper search
24
+ - **๐Ÿ’ป Code Execution**: Support for Python, Bash, SQL, C, and Java
25
+ - **๐Ÿ–ผ๏ธ Image Processing**: Analysis, transformation, OCR, and generation
26
+ - **๐Ÿ“„ Document Processing**: PDF, CSV, Excel, and text file analysis
27
+ - **๐Ÿ“ File Upload Support**: Handle multiple file types with drag-and-drop
28
+ - **๐Ÿงฎ Mathematical Operations**: Complete set of mathematical tools
29
+ - **๐Ÿ’ฌ Conversational Interface**: Natural chat-based interaction
30
+ - **๐Ÿ“Š Evaluation System**: Automated benchmark testing and submission
31
+
32
+ ## ๐Ÿ—๏ธ **Project Structure**
33
+
34
+ ```
35
+ gaia-agent/
36
+ โ”œโ”€โ”€ app.py # Main Q&A chatbot interface
37
+ โ”œโ”€โ”€ evaluation_app.py # GAIA benchmark evaluation runner
38
+ โ”œโ”€โ”€ agent.py # Core agent implementation with tools
39
+ โ”œโ”€โ”€ code_interpreter.py # Multi-language code execution
40
+ โ”œโ”€โ”€ image_processing.py # Image processing utilities
41
+ โ”œโ”€โ”€ system_prompt.txt # System prompt for the agent
42
+ โ”œโ”€โ”€ requirements.txt # Python dependencies
43
+ โ”œโ”€โ”€ metadata.jsonl # GAIA benchmark metadata
44
+ โ”œโ”€โ”€ explore_metadata.ipynb # Data exploration notebook
45
+ โ””โ”€โ”€ README.md # This file
46
+ ```
47
+
48
+ ## ๐Ÿ› ๏ธ **Tool Categories**
49
+
50
+ ### **๐ŸŒ Browser & Search Tools**
51
+ - **Wikipedia Search**: Search Wikipedia with up to 2 results
52
+ - **Web Search**: Tavily-powered web search with up to 3 results
53
+ - **arXiv Search**: Academic paper search with up to 3 results
54
+
55
+ ### **๐Ÿ’ป Code Interpreter Tools**
56
+ - **Multi-Language Execution**: Python, Bash, SQL, C, Java support
57
+ - **Plot Generation**: Matplotlib visualization support
58
+ - **DataFrame Analysis**: Pandas data processing
59
+ - **Error Handling**: Comprehensive error reporting
60
+
61
+ ### **๐Ÿงฎ Mathematical Tools**
62
+ - **Basic Operations**: Add, subtract, multiply, divide
63
+ - **Advanced Functions**: Modulus, power, square root
64
+ - **Complex Numbers**: Support for complex number operations
65
+
66
+ ### **๐Ÿ“„ Document Processing Tools**
67
+ - **File Operations**: Save, read, and download files
68
+ - **CSV Analysis**: Pandas-based data analysis
69
+ - **Excel Processing**: Excel file analysis and processing
70
+ - **OCR**: Extract text from images using Tesseract
71
+
72
+ ### **๐Ÿ–ผ๏ธ Image Processing & Generation Tools**
73
+ - **Image Analysis**: Size, color, and property analysis
74
+ - **Transformations**: Resize, rotate, crop, flip, adjust brightness/contrast
75
+ - **Drawing Tools**: Add shapes, text, and annotations
76
+ - **Image Generation**: Create gradients, noise patterns, and simple graphics
77
+ - **Image Combination**: Stack and combine multiple images
78
+
79
+ ## ๐ŸŽฏ **How to Use**
80
+
81
+ ### **Q&A Chatbot Interface (app.py)**
82
+
83
+ 1. **Start the Chatbot:**
84
+ ```bash
85
+ python app.py
86
+ ```
87
+
88
+ 2. **Access the Interface:**
89
+ - Open `http://localhost:7860` in your browser
90
+ - Upload files (images, documents, CSV, etc.) if needed
91
+ - Ask questions in natural language
92
+ - Get comprehensive answers with tool usage
93
+
94
+ 3. **Supported Interactions:**
95
+ - **Text Questions**: "What is the capital of France?"
96
+ - **Math Problems**: "Calculate the square root of 144"
97
+ - **Code Requests**: "Write a Python function to sort a list"
98
+ - **Image Analysis**: Upload an image and ask "What do you see?"
99
+ - **Data Analysis**: Upload a CSV and ask "What are the trends?"
100
+ - **Web Search**: "What are the latest AI developments?"
101
+
102
+ ### **Evaluation Runner (evaluation_app.py)**
103
+
104
+ 1. **Run the Evaluation:**
105
+ ```bash
106
+ python evaluation_app.py
107
+ ```
108
+
109
+ 2. **Benchmark Testing:**
110
+ - Log in with your Hugging Face account
111
+ - Click "Run Evaluation & Submit All Answers"
112
+ - Monitor progress as the agent processes GAIA benchmark questions
113
+ - View results and scores automatically
114
+
115
+ ## ๐Ÿ”ง **Technical Architecture**
116
+
117
+ ### **LangGraph State Machine**
118
+ ```
119
+ START โ†’ Retriever โ†’ Assistant โ†’ Tools โ†’ Assistant
120
+ โ†‘ โ†“
121
+ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
122
+ ```
123
+
124
+ 1. **Retriever Node**: Searches vector database for similar questions
125
+ 2. **Assistant Node**: LLM processes question with available tools
126
+ 3. **Tools Node**: Executes selected tools (web search, code, etc.)
127
+ 4. **Conditional Routing**: Dynamically routes between assistant and tools
128
+
129
+ ### **Vector Database Integration**
130
+ - **Supabase Vector Store**: Stores GAIA benchmark Q&A pairs
131
+ - **Semantic Search**: Finds similar questions for context
132
+ - **HuggingFace Embeddings**: sentence-transformers/all-mpnet-base-v2
133
+
134
+ ### **Multi-Modal File Support**
135
+ - **Images**: JPG, PNG, GIF, BMP, WebP
136
+ - **Documents**: PDF, DOC, DOCX, TXT, MD
137
+ - **Data**: CSV, Excel, JSON
138
+ - **Code**: Python, Bash, SQL, C, Java
139
+
140
+ ## โš™๏ธ **Installation & Setup**
141
+
142
+ ### **1. Clone Repository**
143
+ ```bash
144
+ git clone https://github.com/Bhumi14/gaia-agent.git
145
+ cd gaia-agent
146
+ ```
147
+
148
+ ### **2. Install Dependencies**
149
+ ```bash
150
+ pip install -r requirements.txt
151
+ ```
152
+
153
+ ### **3. Environment Variables**
154
+ Create a `.env` file with your API keys:
155
+ ```env
156
+ SUPABASE_URL=your_supabase_url
157
+ SUPABASE_SERVICE_ROLE_KEY=your_supabase_key
158
+ GROQ_API_KEY=your_groq_api_key
159
+ TAVILY_API_KEY=your_tavily_api_key
160
+ HUGGINGFACEHUB_API_TOKEN=your_hf_token
161
+ LANGSMITH_API_KEY=your_langsmith_key
162
+
163
+ LANGSMITH_TRACING=true
164
+ LANGSMITH_PROJECT=ai_agent_course
165
+ LANGSMITH_ENDPOINT=https://api.smith.langchain.com
166
+ ```
167
+
168
+ ### **4. Database Setup (Supabase)**
169
+ Execute this SQL in your Supabase database:
170
+ ```sql
171
+ -- Enable pgvector extension
172
+ CREATE EXTENSION IF NOT EXISTS vector;
173
+
174
+ -- Create match function for documents2 table
175
+ CREATE OR REPLACE FUNCTION public.match_documents_2(
176
+ query_embedding vector(768)
177
+ )
178
+ RETURNS TABLE(
179
+ id bigint,
180
+ content text,
181
+ metadata jsonb,
182
+ embedding vector(768),
183
+ similarity double precision
184
+ )
185
+ LANGUAGE sql STABLE
186
+ AS $$
187
+ SELECT
188
+ id,
189
+ content,
190
+ metadata,
191
+ embedding,
192
+ 1 - (embedding <=> query_embedding) AS similarity
193
+ FROM public.documents2
194
+ ORDER BY embedding <=> query_embedding
195
+ LIMIT 10;
196
+ $$;
197
+
198
+ -- Grant permissions
199
+ GRANT EXECUTE ON FUNCTION public.match_documents_2(vector) TO anon, authenticated;
200
+ ```
201
+
202
+ ## ๐Ÿš€ **Running the Application**
203
+
204
+ ### **Chatbot Interface**
205
+ ```bash
206
+ python app.py
207
+ ```
208
+ Access at: `http://localhost:7860`
209
+
210
+ ### **Evaluation Runner**
211
+ ```bash
212
+ python evaluation_app.py
213
+ ```
214
+ Access at: `http://localhost:7860`
215
+
216
+ ### **Live Demo**
217
+ Try it online: [Hugging Face Space](https://huggingface.co/spaces/Bhumi14/gaia-agent)
218
+
219
+ ## ๐Ÿ”— **Resources**
220
+
221
+ - [GAIA Benchmark](https://huggingface.co/spaces/gaia-benchmark/leaderboard)
222
+ - [Hugging Face Agents Course](https://huggingface.co/agents-course)
223
+ - [LangGraph Documentation](https://langchain-ai.github.io/langgraph/)
224
+ - [Supabase Vector Store](https://supabase.com/docs/guides/ai/vector-columns)
225
+
226
+ ## ๐Ÿค **Contributing**
227
+
228
+ Contributions are welcome! Areas for improvement:
229
+ - **New Tools**: Add specialized tools for specific domains
230
+ - **UI Enhancements**: Improve the chatbot interface
231
+ - **Performance**: Optimize response times and accuracy
232
+ - **Documentation**: Expand examples and use cases
233
+
234
+ ## ๐Ÿ“„ **License**
235
+
236
+ This project is licensed under the [MIT License](https://mit-license.org/).