version-2 initial version

.env

@@ -1 +1,7 @@
-OPENAI_API_KEY = <openai_api_key>
+OPENAI_API_KEY = sk-fsuFBJSKfbashjvasSFBASJ
+
+LLM_PROVIDER = google
+
+GEMINI_API_KEY = AIfuwfnqf8qsfj3P9o
+
+LLL_MODEL = gemini-2.0-flash

.gitattributes

@@ -37,3 +37,4 @@ TTS-Spaces-Arena-25-Dec-2024.png filter=lfs diff=lfs merge=lfs -text
 HEARME.wav filter=lfs diff=lfs merge=lfs -text
 demo/af_sky.wav filter=lfs diff=lfs merge=lfs -text
 output.wav filter=lfs diff=lfs merge=lfs -text
+frontend/bun.lockb filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,12 @@
+__pycache__
+qdrant_storage
+node_modules
+models/
+__pycache__
+test3.py
+test4.py
+virtual/
+langchain_parser*
+qdrant_search_test*
+
+.env.prod

README.md

@@ -1,208 +1,314 @@
----
-license: mit
-base_model:
-- hexgrad/Kokoro-82M
----
-# **VocRT**  
-This repository contains the complete codebase for building your personal Realtime Voice-to-Voice (V2V) solution. It integrates a powerful TTS model, gRPC communication, an Express server, and a React-based client. Follow this guide to set up and explore the system effectively.
+# **VocRT - Personal Realtime Voice-to-Voice AI Solution**
 
----
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Docker](https://img.shields.io/badge/Docker-Available-blue.svg)](https://hub.docker.com/r/anuragsingh922/vocrt)
+[![Python 3.10](https://img.shields.io/badge/Python-3.10-green.svg)](https://www.python.org/)
+
+VocRT is a comprehensive, privacy-first **Realtime Voice-to-Voice (V2V)** solution that enables natural conversations with AI. Built with cutting-edge TTS models, RAG capabilities, and seamless integration, VocRT processes your voice input and responds with high-quality synthesized speech in real-time.
+
+## **🚀 Key Features**
+
+### **Real-time Voice Processing**
+
+- **Ultra-low latency** voice-to-voice conversion
+- **High-quality speech synthesis** using Kokoro-82M model
+- **Customizable voice selection** with multiple voice options
+- **Adjustable threshold and silence duration** for optimal user experience
+
+### **Advanced RAG Capabilities**
+
+- **Multi-format document support**: PDF, CSV, TXT, PPT, PPTX, DOC, DOCX, XLS, XLSX
+- **URL content extraction**: Process web pages, Medium blogs, and online PDFs
+- **Unlimited document uploads** without usage limits or billing concerns
+- **100% privacy-first approach** with local processing
+
+### **Privacy & Cost Benefits**
+
+- **No API usage limits** or recurring charges
+- **Complete data privacy** - all processing happens locally
+- **Offline capability** use local LLM model if resources allow
+- **No data sharing** with external AI services
+
+## **🏗️ Architecture Overview**
 
-## **Repository Structure**
 ```
-├── backend/         # Express server for handling API requests
-├── frontend/        # React client for user interaction
-├── .env             # Environment variables (OpenAI API key, etc.)
-├── voices           # All available voices
-├── demo             # Contains sample audio and demo files
-├── other...
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│   React Client  │◄──►│  Express Server │◄──►│  VocRT Engine   │
+│   (Frontend)    │    │   (Backend)     │    │   (Python)      │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+                                                        |
+                                                        |
+                                 _______________________|
+                                │                       │
+                                ▼                       ▼
+                       ┌──────────────────┐    ┌─────────────────┐
+                       │    Embeddings    │    │   Whisper STT   │
+                       │   (e5-base-v2)   │    │   Kokoro TTS    │
+                       │    Qdrant DB     │    │                 │
+                       │   (Vector Store) │    └─────────────────┘
+                       └──────────────────┘
 ```
 
----
+## **📁 Repository Structure**
+
+```
+VocRT/
+├── backend/         # Express.js server
+├── frontend/        # React client application
+├── models/          # AI models directory
+├── voices/          # Available voice profiles
+├── demo/            # Sample audio and demo files
+├── .env             # Environment configuration
+├── requirements.txt # Python dependencies
+└── README.md        # Project documentation
+```
 
-## **Docker**
+## **🐳 Quick Start with Docker**
 
-🐳 VocRT on Docker Hub: https://hub.docker.com/r/anuragsingh922/vocrt
+**Docker Hub**: [anuragsingh922/vocrt](https://hub.docker.com/r/anuragsingh922/vocrt)
 
+```
+# Pull and run VocRT container
+docker pull anuragsingh922/vocrt
+docker run -p 3000:3000 -p 8080:8080 anuragsingh922/vocrt
+```
 
-## **Repository**
-## **Setup Guide**
+## **🛠️ Manual Installation**
 
-### **Step 1: Clone the Repository**
-Clone this repository to your local machine:
-```bash
+### **Prerequisites**
+
+- **Python 3.10** (required)
+- **Node.js 16+** and npm
+- **Docker** (for Qdrant vector database)
+- **Git** for cloning repositories
+
+### **Step 1: Clone Repository**
+
+```
 git clone https://huggingface.co/anuragsingh922/VocRT
 cd VocRT
 ```
 
----
+### **Step 2: Python Environment Setup**
 
-### **Step 2: Python Virtual Environment Setup**
-Create a virtual environment to manage dependencies:
+#### **macOS/Linux:**
 
-#### macOS/Linux:
-```bash
-python3 -m venv venv
+```
+python3.10 -m venv venv
 source venv/bin/activate
 ```
 
-#### Windows:
-```bash
-python -m venv venv
+#### **Windows:**
+
+```
+python3.10 -m venv venv
 venv\Scripts\activate
 ```
 
----
-
 ### **Step 3: Install Python Dependencies**
-With the virtual environment activated, install the required dependencies:
-```bash
+
+```
 pip install --upgrade pip setuptools wheel
 pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
-pip install phonemizer transformers scipy munch python-dotenv openai grpcio grpcio-tools
-```
-
-### **Installing eSpeak**
-`eSpeak` is a necessary dependency for the VocRT system. Follow the instructions below to install it on your platform:
-
-#### **Ubuntu/Linux**
-Use the `apt-get` package manager to install `eSpeak`:
-```bash
-sudo apt-get update
-sudo apt-get install espeak
-```
-
-#### **macOS**
-Install `eSpeak` using [Homebrew](https://brew.sh/):
-1. Ensure Homebrew is installed on your system:
-   ```bash
-   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
-   ```
-2. Install `espeak`:
-   ```bash
-   brew install espeak
-   ```
-
-#### **Windows**
-For Windows, follow these steps to install `eSpeak`:
-1. Download the eSpeak installer from the official website: [eSpeak Downloads](http://espeak.sourceforge.net/download.html).
-2. Run the installer and follow the on-screen instructions to complete the installation.
-3. Add the `eSpeak` installation path to your system's `PATH` environment variable:
-   - Open **System Properties** → **Advanced** → **Environment Variables**.
-   - In the "System Variables" section, find the `Path` variable and edit it.
-   - Add the path to the `espeak.exe` file (e.g., `C:\Program Files (x86)\eSpeak`).
-4. Verify the installation:
-   Open Command Prompt and run:
-   ```cmd
-   espeak --version
-   ```
+pip install -r requirements.txt
+```
 
----
+### **Step 4: Install eSpeak**
 
-### **Verification**
-After installing `eSpeak`, verify it is correctly set up by running:
-```bash
-espeak "Hello, world!"
+#### **Ubuntu/Debian:**
+
+```
+sudo apt-get update && sudo apt-get install espeak
 ```
 
-This should output "Hello, world!" as audio on your system.
+#### **macOS:**
 
----
+```
+# Install Homebrew if not present
+/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
+# Install eSpeak
+brew install espeak
+```
 
-### **Step 4: Backend Setup (Express Server)**
-1. Navigate to the `backend` directory:
-   ```bash
-   cd backend
-   ```
-2. Install Node.js dependencies:
-   ```bash
-   npm install
-   ```
-3. Update the `config.env` file with your Deepgram API key:
-   - Open `config.env` in a text editor.
-   - Replace `<deepgram_api_key>` with your actual Deepgram API key.
-
-4. Start the Express server:
-   ```bash
-   node app.js
-   ```
+#### **Windows:**
 
----
+1. Download from [eSpeak official website](http://espeak.sourceforge.net/download.html)
+2. Run installer and follow instructions
+3. Add installation path to system PATH environment variable
+4. Verify installation: `espeak --version`
 
-### **Step 5: Frontend Setup (React Client)**
-1. Open a new terminal and navigate to the `frontend` directory:
-   ```bash
-   cd frontend
-   ```
-2. Install client dependencies:
-   ```bash
-   npm install
-   ```
-3. Start the client:
-   ```bash
-   npm start
-   ```
+#### **Verification:**
 
----
+```
+espeak "VocRT installation successful!"
+```
 
-### **Step 6: Start the VocRT Server**
-1. Add your OpenAI API key to the `.env` file:
-   - Open `.env` in a text editor.
-   - Replace `<openai_api_key>` with your actual OpenAI API key.
+### **Step 5: Backend Setup (Express.js)**
 
-2. Start the VocRT server:
-   ```bash
-   python3 app.py
-   ```
+```
+cd backend
+npm install
+node app.js
+```
 
----
+### **Step 6: Frontend Setup (React)**
+
+```
+cd frontend
+npm install
+npm run dev
+```
 
-### **Step 7: Test the Full System**
-- Once all servers are running:
-  1. Access the React client at [http://localhost:3000](http://localhost:3000).
-  2. Interact with the VocRT system via the web interface.
+### **Step 7: Qdrant Vector Database Setup**
 
----
+**Documentation**: [Qdrant Quickstart Guide](https://qdrant.tech/documentation/quickstart)
 
-## **Model Used**
-VocRT uses [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) for text-to-speech synthesis, processing user inputs into high-quality voice responses.
+```
+# Pull Qdrant image
+docker pull qdrant/qdrant
 
----
+# Start Qdrant container
+docker run -p 6333:6333 -p 6334:6334 \
+  -v "$(pwd)/qdrant_storage:/qdrant/storage:z" \
+  qdrant/qdrant
+```
 
-## **Key Features**
-1. **Realtime voice response generation**: Convert speech input into speech with minimal latency.
-2. **React Client**: A user-friendly frontend for interaction.
-3. **Express Backend**: Handles API requests and integrates the VocRT system with external services.
-4. **gRPC Communication**: Seamless communication between the VocRT server and other components.
-5. **Configurable APIs**: Integrates with OpenAI and Deepgram APIs for speech recognition and text generation.
+**Access Points:**
 
----
+- **REST API**: [http://localhost:6333](http://localhost:6333)
+- **Web Dashboard**: [http://localhost:6333/dashboard](http://localhost:6333/dashboard)
+- **gRPC API**: [http://localhost:6334](http://localhost:6334)
+
+### **Step 8: Download Required Models**
+
+#### **Embedding Model:**
+
+Clone [e5-base-v2](https://huggingface.co/intfloat/e5-base-v2) to `models/e5-base-v2`
+
+![alt text](image.png)
+
+#### **Whisper STT Model:**
+
+Choose your preferred Whisper model size:
+
+- **tiny**: Fastest, lower accuracy
+- **base**: Balanced performance
+- **small**: Better accuracy
+- **medium/large**: Highest accuracy, slower processing
+
+![alt text](image-3.png)
+
+### **Step 9: Environment Configuration**
+
+Edit `.env` file with your API credentials:
+
+```
+# LLM Configuration
+OPENAI_API_KEY=your_openai_api_key_here
+GEMINI_API_KEY=your_gemini_api_key_here
+LLM_PROVIDER=google  # or 'google' for Gemini
+LLM_MODEL=gemini-2.0-flash  # or your preferred model
+```
+
+### **Step 10: Launch VocRT Server**
+
+```
+python3 app.py
+```
+
+## **🎯 Usage Guide**
+
+1. **Access the application**: Navigate to [http://localhost:3000](http://localhost:3000)
+2. **Select voice profile**: Choose from available voice options
+3. **Configure settings**: Adjust silence duration for optimal performance
+4. **Add context**: Upload documents, provide URLs, or enter text for AI context
+5. **Start conversation**: Begin speaking and enjoy real-time voice responses
+
+## **📊 Supported Document Formats**
+
+| Format         | Extension       | Description                        |
+| -------------- | --------------- | ---------------------------------- |
+| **PDF**        | `.pdf`          | Portable Document Format           |
+| **Text**       | `.txt`          | Plain text files                   |
+| **Word**       | `.doc`, `.docx` | Microsoft Word documents           |
+| **Excel**      | `.xls`, `.xlsx` | Microsoft Excel spreadsheets       |
+| **PowerPoint** | `.ppt`, `.pptx` | Microsoft PowerPoint presentations |
+| **CSV**        | `.csv`          | Comma-separated values             |
+| **URLs**       | Web links       | Online content, blogs, PDFs        |
+
+## **🤖 AI Models & Technology Stack**
+
+### **Core Models**
+
+- **TTS**: [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) - High-quality text-to-speech
+- **STT**: OpenAI Whisper - Accurate speech recognition
+- **Embeddings**: [e5-base-v2](https://huggingface.co/intfloat/e5-base-v2) - Semantic text understanding
+- **LLM**: OpenAI GPT / Google Gemini - Natural language processing
+
+### **Technology Stack**
+
+- **Backend**: Python, Express.js, gRPC
+- **Frontend**: React, Vite
+- **Database**: Qdrant (Vector Database)
+- **Containerization**: Docker
+- **Audio Processing**: Whisper, eSpeak, phonemizer
+
+## **🔧 Performance Optimization**
+
+### **Hardware Recommendations**
+
+- **CPU**: Multi-core processor (4+ cores recommended)
+- **RAM**: 4GB+ for optimal performance
+- **Storage**: SSD for faster model loading
+- **GPU**: Optional, for accelerated inference can reduce latency upto 60%
+
+### **Configuration Tips**
+
+- Modify **silence duration** for natural conversation flow
+- Use **smaller Whisper models** for faster STT processing
+- Enable **GPU acceleration** if available
+
+## **🤝 Contributing**
+
+We welcome contributions from the community! Here's how you can help:
+
+### **Ways to Contribute**
+
+- 🐛 **Bug Reports**: Submit issues with detailed reproduction steps
+- 💡 **Feature Requests**: Suggest new capabilities and improvements
+- 📝 **Documentation**: Improve guides, tutorials, and API docs
+- 🔧 **Code Contributions**: Submit pull requests with enhancements
+
+### **Development Setup**
+
+1. Fork the repository
+2. Create a feature branch: `git checkout -b feature/amazing-feature`
+3. Commit changes: `git commit -m 'Add amazing feature'`
+4. Push to branch: `git push origin feature/amazing-feature`
+5. Open a Pull Request
+
+## **📄 License**
+
+This project is licensed under the **MIT License**
+
+## **🙏 Acknowledgments**
 
-## **Dependencies**
+Special thanks to the amazing open-source communities:
 
-### Python:
-  - torch, torchvision, torchaudio
-  - phonemizer
-  - transformers
-  - scipy
-  - munch
-  - python-dotenv
-  - openai
-  - grpcio, grpcio-tools
-  - espeak
+- **[Hugging Face](https://huggingface.co/)** - For hosting and maintaining AI models
+- **[Kokoro-82M Team](https://huggingface.co/hexgrad/Kokoro-82M)** - Exceptional TTS model
+- **[OpenAI Whisper](https://github.com/openai/whisper)** - Revolutionary speech recognition
+- **[Qdrant](https://qdrant.tech/)** - High-performance vector database
+- **[React](https://reactjs.org/)** & **[Node.js](https://nodejs.org/)** communities
 
+## **📞 Support & Contact**
 
-### Node.js:
-- Express server dependencies (`npm install` in `backend`).
-- React client dependencies (`npm install` in `frontend`).
+- **Email**: [anuragjadu922@gmail.com](mailto:[email protected])
 
 ---
 
-## **Contributing**
-Contributions are welcome! Feel free to fork this repository and create a pull request with your improvements.
+**⭐ If VocRT helps your projects, please consider giving it a star!**
 
 ---
 
-## **Acknowledgments**
-- [Hugging Face](https://huggingface.co/) for hosting the Kokoro-82M model.
-- The amazing communities behind PyTorch, OpenAI, and Deepgram APIs.
+_Built with ❤️ for the open-source community_

app.py

@@ -1,206 +1,472 @@
 from concurrent import futures
+import asyncio
 import torch
 from models import build_model
-import numpy as np
-import re
-import wave
-from kokoro import generate
-from openai import OpenAI
 from collections import deque
 import grpc
 import text_to_speech_pb2
 import text_to_speech_pb2_grpc
-import io
-from dotenv import load_dotenv
+from chat_database import save_chat_entry
+import fastAPI
+from providers.audio_provider import get_audio_bytes, dummy_bytes, generate_audio_stream
+from providers.llm_provider import getResponseWithRagAsync, getResponseAsync
+import numpy as np
 import os
-from chat_database import save_chat_entry, get_chat_history
+import re
+import time
+from silero_vad import load_silero_vad, VADIterator
+import random
+from providers.filler_words import filler_phrases
+from scipy.io.wavfile import write
+from faster_whisper import WhisperModel
+
+sampling_rate = 16_000
+vad_model = load_silero_vad()
+vad_iter = VADIterator(vad_model, sampling_rate=sampling_rate)
+frame_size = 512
 
-load_dotenv()
 
-# Device configuration
 device = 'cuda' if torch.cuda.is_available() else 'cpu'
 
-# Load the Kokoro model
-MODEL = build_model('kokoro-v0_19.pth', device)
 
-# Specify the voice name and load the voice pack
+whisper_model = WhisperModel(
+    "small",
+    device=device,
+    compute_type="int8",
+    cpu_threads=os.cpu_count(),
+    download_root="./models"
+)
+
+MODEL = build_model('kokoro-v0_19.pth', device)
 VOICE_NAME = [
     'af',
     'af_bella', 'af_sarah', 'am_adam', 'am_michael',
     'bf_emma', 'bf_isabella', 'bm_george', 'bm_lewis',
     'af_nicole', 'af_sky',
 ][0]
-VOICEPACK = torch.load(f'voices/{VOICE_NAME}.pt', weights_only=True).to(device)
 
 
-client = OpenAI(
-    api_key= os.getenv("OPENAI_API_KEY")
-)
+VOICEPACK = torch.load(
+    f'voices/{VOICE_NAME}.pt', weights_only=True).to(device)
 
-def chunk_text(text, max_chars=2040):
-    sentences = re.split(r'(?<=[.!?])\s+', text)
-    chunks = []
-    current_chunk = []
-    current_length = 0
-    for sentence in sentences:
-        sentence_length = len(sentence)
-        if current_length + sentence_length <= max_chars:
-            current_chunk.append(sentence)
-            current_length += sentence_length
-        else:
-            if current_chunk:
-                chunks.append(' '.join(current_chunk))
-            current_chunk = [sentence]
-            current_length = sentence_length
-    if current_chunk:
-        chunks.append(' '.join(current_chunk))
-    return chunks
-
-def generate_audio_from_chunks(text, model, voicepack, voice_name):
-    chunks = chunk_text(text)
-    combined_audio = np.array([])
-    for chunk in chunks:
-        try:
-            audio, _ = generate(model, chunk, voicepack, lang=voice_name[0])
-            combined_audio = np.concatenate([combined_audio, audio]) if combined_audio.size > 0 else audio
-        except Exception:
-            pass
-    return combined_audio
-
-def save_audio_to_file(audio_data, file_number, sample_rate=24000):
-    filename = f"output-{file_number}.wav"
-    with wave.open(filename, 'wb') as wav_file:
-        wav_file.setnchannels(1)
-        wav_file.setsampwidth(2)
-        wav_file.setframerate(sample_rate)
-        audio_int16 = (audio_data * 32767).astype(np.int16)
-        wav_file.writeframes(audio_int16.tobytes())
-    return filename
-
-def getResponse(text , session_id):
-    try:
-        chat_history = get_chat_history(session_id)
-        response = client.chat.completions.create(
-            model='gpt-3.5-turbo',
-            messages=chat_history,
-            stream=True
-        )
-        return response
-    except Exception as e:
-        print("Error in getResponse : "  , e)
-
-def get_audio_bytes(audio_data, sample_rate=24000):
-    wav_bytes = io.BytesIO()
-    with wave.open(wav_bytes, 'wb') as wav_file:
-        wav_file.setnchannels(1)
-        wav_file.setsampwidth(2)
-        wav_file.setframerate(sample_rate)
-        audio_int16 = (audio_data * 32767).astype(np.int16)
-        wav_file.writeframes(audio_int16.tobytes())
-    wav_bytes.seek(0)
-    return wav_bytes.read()
-
-def dummy_bytes():
-    buffer = io.BytesIO()
-    dummy_data = b"This is a test of dummy byte data."
-    buffer.write(dummy_data)
-    buffer.seek(0)
-    byte_value = buffer.getvalue()
-    return byte_value
+AUDIO_FILES_DIR = 'audio_files'
+os.makedirs(AUDIO_FILES_DIR, exist_ok=True)
+
+PRE_CHUNK_LIMIT_BYTES = frame_size * 2 * 20
+
+transcription_pool = futures.ThreadPoolExecutor(max_workers=10)
+
+terminators = ['.', '?', '!', '...', '…', '?!', '!?', '‽', '。', '؟', '۔']
+
+BLACKLIST = {
+    "Give me a minute.",
+    "Let me check the details.",
+    "Give me a minute. Let me check the details."
+}
+
+
+dummy_audio = np.frombuffer(
+    np.zeros(int(16_000 * 5.0), dtype=np.float32), dtype=np.int16).astype(np.float32) / 32768.0
+
+
+def _fw_transcribe_block(audio_f32: np.ndarray) -> dict:
+    segments, info = whisper_model.transcribe(
+        audio_f32,
+        language="en",
+        beam_size=1,
+        vad_filter=False,
+        initial_prompt="Indian English accent; do not make up words.",
+        no_speech_threshold=0.25,
+        log_prob_threshold=-0.6,
+        temperature=0
+    )
+    text = "".join(seg.text for seg in segments)
+    return {"text": text, "language": info.language,
+            "language_probability": info.language_probability}
+
+
+async def safe_transcribe(audio_float32: np.ndarray):
+    loop = asyncio.get_running_loop()
+    return await loop.run_in_executor(
+        transcription_pool,
+        lambda: _fw_transcribe_block(audio_float32)
+    )
 
 
 class TextToSpeechServicer(text_to_speech_pb2_grpc.TextToSpeechServiceServicer):
-    def ProcessText(self, request_iterator, context):
+
+    def __init__(self):
+        super().__init__()
+        self._transcribe_lock = asyncio.Lock()
+
+    async def ProcessText(self, request_iterator, context):
         try:
-            print("Received new request")
+            global VOICEPACK
+
+            print("New connection")
+
+            tts_queue = asyncio.Queue()
+            response_queue = asyncio.Queue()
+
             parameters = {
                 "processing_active": False,
                 "queue": deque(),
                 "file_number": 0,
                 "session_id": "",
-                "interrupt_seq" : 0
+                "interrupt_seq": 0,
+                "temperature": 1,
+                "activeVoice": "af",
+                "in_speech": False,
+                "maxTokens": 500,
+                "audio_buffer": bytearray(),
+                "pre_chunks": bytearray(),
+                "silence_counter": 0.0,
+                "silence_duration": 0.8,  # default duration in seconds
+                "silence_threshold": 800,  # default amplitude threshold
+                "VOICEPACK": VOICEPACK,
+                "audio_count": 0,
+                "user_sequence": 0,
+                "last_file_number": 0
             }
-            for request in request_iterator:
-                field = request.WhichOneof('request_data')
-                if field == 'metadata':
-                    parameters["session_id"] = request.metadata.session_id
-                    continue
-                elif field == 'text':
-                    text = request.text
-                    if not text:
+
+            reader = asyncio.create_task(
+                self._read_requests(request_iterator, tts_queue, response_queue, parameters))
+
+            tts = asyncio.create_task(self._tts_queue_worker(
+                tts_queue, response_queue, parameters))
+
+            try:
+                while True:
+                    resp = await response_queue.get()
+                    if resp is None:
+                        break
+                    yield resp
+            finally:
+                reader.cancel()
+                tts.cancel()
+
+        except Exception as e:
+            print("Error in ProcessText:", e)
+
+    async def _read_requests(self, request_iterator, tts_queue: asyncio.Queue, response_queue: asyncio.Queue, parameters):
+        async for request in request_iterator:
+            field = request.WhichOneof('request_data')
+            if field == 'metadata':
+                meta = request.metadata
+                # print("\n\nMetadata : ", meta)
+                if meta.session_id:
+                    parameters["session_id"] = meta.session_id
+                if meta.temperature:
+                    parameters["temperature"] = meta.temperature
+                if meta.maxTokens:
+                    parameters["maxTokens"] = meta.maxTokens
+                if meta.activeVoice:
+                    parameters["activeVoice"] = meta.activeVoice
+                    parameters["VOICEPACK"] = torch.load(
+                        f'voices/{parameters["activeVoice"]}.pt', weights_only=True).to(device)
+                    print("\n\nVoice model loaded successfully")
+                if meta.silenceDuration:
+                    silence_duration = meta.silenceDuration / 1000
+                    parameters["silence_duration"] = silence_duration
+                if meta.threshold:
+                    parameters["silence_threshold"] = meta.threshold
+
+                print("\n\nPatameter : ", parameters)
+
+                # output = await safe_transcribe("output2.wav")
+
+                resp = text_to_speech_pb2.ProcessTextResponse(
+                    buffer=dummy_bytes(),
+                    session_id=parameters["session_id"],
+                    sequence_id="-10",
+                    transcript="",
+                )
+                await response_queue.put(resp)
+
+                continue
+            elif field == 'audio_data':
+
+                buffer = request.audio_data.buffer
+
+                audio_data = np.frombuffer(buffer, dtype=np.int16)
+
+                float_chunk = audio_data.astype(np.float32) / 32768.0
+
+                vad_result = vad_iter(float_chunk)
+
+                parameters["pre_chunks"].extend(buffer)
+                if len(parameters["pre_chunks"]) > PRE_CHUNK_LIMIT_BYTES:
+                    overflow = len(
+                        parameters["pre_chunks"]) - PRE_CHUNK_LIMIT_BYTES
+                    del parameters["pre_chunks"][:overflow]
+
+                if vad_result:
+                    if "start" in vad_result:
+                        parameters["in_speech"] = True
+                        parameters["audio_buffer"].extend(
+                            parameters["pre_chunks"])
+                    if "end" in vad_result:
+                        parameters["in_speech"] = False
+
+                if parameters["in_speech"]:
+                    parameters["audio_buffer"].extend(buffer)
+                    parameters["silence_counter"] = 0.0
+                    parameters["audio_count"] += 1
+                else:
+                    sample_rate = 16000
+                    duration = len(audio_data) / sample_rate
+                    parameters["silence_counter"] += duration
+
+                if parameters["silence_counter"] >= parameters["silence_duration"]:
+                    parameters["silence_counter"] = 0.0
+                    if parameters["audio_count"] < 2:
+                        parameters["audio_count"] = 0
                         continue
-                    save_chat_entry(parameters["session_id"] , "user" , text)
-                    parameters["queue"].clear()
-                    yield text_to_speech_pb2.ProcessTextResponse(
-                        buffer = dummy_bytes(),
+                    parameters["audio_count"] = 0
+                    print("Silence ")
+
+                    resp = text_to_speech_pb2.ProcessTextResponse(
+                        buffer=dummy_bytes(),
                         session_id=parameters["session_id"],
-                        sequence_id = "-2",
-                        transcript=text,
+                        sequence_id="-3",
+                        transcript="",
                     )
-                    final_response = ""
-                    response = getResponse(text , parameters["session_id"])
-                    for chunk in response:
-                        msg = chunk.choices[0].delta.content
-                        if msg:
-                            final_response += msg
-                            if final_response.endswith(('.', '!', '?')):
+                    await response_queue.put(resp)
+
+                    # resp = text_to_speech_pb2.ProcessTextResponse(
+                    #     buffer=dummy_bytes(),
+                    #     session_id=parameters["session_id"],
+                    #     sequence_id="0",
+                    #     transcript="",
+                    # )
+
+                    # await response_queue.put(resp)
+
+                    sample_rate = 16000
+
+                    audio_float = np.frombuffer(
+                        parameters["audio_buffer"], dtype=np.int16).astype(np.float32) / 32768.0
+
+                    parameters["audio_buffer"] = bytearray()
+
+                    whisper_start_time = time.time()
+                    result = ""
+                    try:
+                        result = await safe_transcribe(audio_float)
+                    except Exception as e:
+                        await tts_queue.put(("Sorry! I am not able to catch that can you repeat again please!", parameters["file_number"]))
+                        print("Error in transcribing text : ", e)
+                        continue
+
+                    whisper_end_time = time.time()
+                    time_taken_to_transcribe = whisper_end_time - whisper_start_time
+                    print(
+                        f"Transcribing time: {time_taken_to_transcribe:.4f} seconds")
+                    transcribed_text = result["text"]
+                    print(
+                        f"Transcribed Text :", transcribed_text)
+
+                    if not transcribed_text.strip():
+                        resp = text_to_speech_pb2.ProcessTextResponse(
+                            buffer=dummy_bytes(),
+                            session_id=parameters["session_id"],
+                            sequence_id="-5",
+                            transcript="",
+                        )
+                        await response_queue.put(resp)
+                        continue
+
+
+# Transcript Detected ------------------------------------------------------------------------------------
+
+                    if transcribed_text:
+                        parameters["queue"].clear()
+                        parameters["user_sequence"] += 1
+                        parameters["last_file_number"] = parameters["file_number"]
+                        while not response_queue.empty():
+                            try:
+                                response_queue.get_nowait()
+                                response_queue.task_done()
+                            except asyncio.QueueEmpty:
+                                break
+                        while not tts_queue.empty():
+                            try:
+                                tts_queue.get_nowait()
+                                tts_queue.task_done()
+                            except asyncio.QueueEmpty:
+                                break
+
+                        resp = text_to_speech_pb2.ProcessTextResponse(
+                            buffer=dummy_bytes(),
+                            session_id=parameters["session_id"],
+                            sequence_id="-4",
+                            transcript="",
+                        )
+                        await response_queue.put(resp)
+                        resp = text_to_speech_pb2.ProcessTextResponse(
+                            buffer=dummy_bytes(),
+                            session_id=parameters["session_id"],
+                            sequence_id="-2",
+                            transcript=transcribed_text,
+                        )
+                        save_chat_entry(
+                            parameters["session_id"], "user", transcribed_text)
+                        await response_queue.put(resp)
+
+                        try:
+                            filler = random.choice(filler_phrases)
+                            # await tts_queue.put((filler, parameters["file_number"]))
+                            loop = asyncio.get_event_loop()
+
+                            loop.call_later(
+                                0,
+                                # 1.0,
+                                lambda: asyncio.create_task(
+                                    tts_queue.put(
+                                        (filler, parameters["file_number"]))
+                                )
+                            )
+
+                        except Exception as e:
+                            print("Error in sendign error : ", e)
+                        final_response = ""
+                        complete_response = ""
+                        current_user_sequence = parameters["user_sequence"]
+                        response = await getResponseAsync(
+                            transcribed_text, parameters["session_id"])
+                        if response is None:
+                            continue
+                        for chunk in response:
+                            if (current_user_sequence != parameters["user_sequence"]):
+                                break
+                            msg = chunk.choices[0].delta.content
+                            if msg:
+                                complete_response += msg
+                                m = re.search(r'[.?!]', msg)
+                                if m:
+                                    idx = m.start()
+                                    segment = msg[:idx+1]
+                                    leftover = msg[idx+1:]
+                                else:
+                                    segment, leftover = msg, ''
+
+                                final_response += segment
+
+                                if segment.endswith(('.', '!', '?')):
+                                    parameters["file_number"] += 1
+                                    parameters["queue"].append(
+                                        (final_response, parameters["file_number"]))
+                                    await tts_queue.put((final_response, parameters["file_number"]))
+                                    final_response = leftover
+
+                        if final_response.strip():
+                            parameters["file_number"] += 1
+                            parameters["queue"].append(
+                                (final_response, parameters["file_number"]))
+                            await tts_queue.put((final_response, parameters["file_number"]))
+
+                        if ("Let me check" in complete_response):
+                            final_response = ""
+                            complete_response = ""
+                            current_user_sequence = parameters["user_sequence"]
+                            response = await getResponseWithRagAsync(
+                                transcribed_text, parameters["session_id"])
+                            for chunk in response:
+                                if (current_user_sequence != parameters["user_sequence"]):
+                                    break
+                                msg = chunk.choices[0].delta.content
+                                if msg:
+                                    m = re.search(r'[.?!]', msg)
+                                    if m:
+                                        idx = m.start()
+                                        segment = msg[:idx+1]
+                                        leftover = msg[idx+1:]
+                                    else:
+                                        segment, leftover = msg, ''
+
+                                    final_response += segment
+                                    complete_response += segment
+
+                                    if segment.endswith(('.', '!', '?')):
+                                        parameters["file_number"] += 1
+                                        parameters["queue"].append(
+                                            (final_response, parameters["file_number"]))
+                                        await tts_queue.put((final_response, parameters["file_number"]))
+                                        final_response = leftover
+
+                            if final_response.strip():
                                 parameters["file_number"] += 1
-                                parameters["queue"].append((final_response, parameters["file_number"]))
-                                final_response = ""
-                                if not parameters["processing_active"]:
-                                    yield from self.process_queue(parameters)
-
-                    if final_response:
-                        parameters["file_number"] += 1
-                        parameters["queue"].append((final_response, parameters["file_number"]))
-                        if not parameters["processing_active"]:
-                            yield from self.process_queue(parameters)
-                            
-                elif field == 'status':
-                    transcript = request.status.transcript
-                    played_seq = request.status.played_seq
-                    interrupt_seq = request.status.interrupt_seq
-                    parameters["interrupt_seq"] = interrupt_seq
-                    save_chat_entry(parameters["session_id"] , "assistant" , transcript)
-                    continue
-                else:
-                    continue
-        except Exception as e:
-            print("Error in ProcessText:", e)
+                                parameters["queue"].append(
+                                    (final_response, parameters["file_number"]))
+                                await tts_queue.put((final_response, parameters["file_number"]))
 
-    def process_queue(self , parameters):
-        try:
-            while True:
-                if not parameters["queue"]:
-                    parameters["processing_active"] = False
+                continue
+
+            elif field == 'status':
+                transcript = request.status.transcript
+                played_seq = request.status.played_seq
+                interrupt_seq = request.status.interrupt_seq
+                parameters["interrupt_seq"] = interrupt_seq
+                text = transcript.strip() if transcript else ""
+                if text and text not in BLACKLIST:
+                    save_chat_entry(
+                        parameters["session_id"],
+                        "assistant",
+                        transcript
+                    )
+                continue
+            else:
+                continue
+
+    async def _tts_queue_worker(self, tts_queue: asyncio.Queue,
+                                response_queue: asyncio.Queue,
+                                params: dict):
+        """
+        Pull (text, seq) off tts_queue, run generate_audio_stream, wrap each chunk
+        in ProcessTextResponse, and push into response_queue.
+        """
+        while True:
+            item = await tts_queue.get()
+            tts_queue.task_done()
+            if item is None:
+                break
+
+            sentence, seq = item
+            # drop anything the client has already played:
+            if seq <= int(params["interrupt_seq"]):
+                continue
+
+            # stream the audio chunks, pack into gRPC responses
+            async for audio_chunk in generate_audio_stream(
+                    sentence, MODEL, params["VOICEPACK"], VOICE_NAME
+            ):
+                audio_bytes = get_audio_bytes(audio_chunk)
+                if seq <= int(params["last_file_number"]):
                     break
-                parameters["processing_active"] = True
-                sentence, file_number = parameters["queue"].popleft()
-                if file_number <= int(parameters["interrupt_seq"]):
-                    continue
-                combined_audio = generate_audio_from_chunks(sentence, MODEL, VOICEPACK, VOICE_NAME)
-                audio_bytes = get_audio_bytes(combined_audio)
-                # filename = save_audio_to_file(combined_audio, file_number)
-                yield text_to_speech_pb2.ProcessTextResponse(
+                resp = text_to_speech_pb2.ProcessTextResponse(
                     buffer=audio_bytes,
-                    session_id=parameters["session_id"],
-                    sequence_id=str(file_number),
+                    session_id=params["session_id"],
+                    sequence_id=str(seq),
                     transcript=sentence,
                 )
-        except Exception as e:
-            parameters["processing_active"] = False
-            print("Error in process_queue:", e)
+                await response_queue.put(resp)
 
 
-def serve():
+async def serve():
     print("Starting gRPC server...")
-    server = grpc.server(futures.ThreadPoolExecutor(max_workers=1))
-    text_to_speech_pb2_grpc.add_TextToSpeechServiceServicer_to_server(TextToSpeechServicer(), server)
+
+    # Use grpc.aio.server for the gRPC async server
+    server = grpc.aio.server(futures.ThreadPoolExecutor(max_workers=10))
+    text_to_speech_pb2_grpc.add_TextToSpeechServiceServicer_to_server(
+        TextToSpeechServicer(), server)
     server.add_insecure_port('[::]:8081')
-    server.start()
+
+    await server.start()
     print("gRPC server is running on port 8081")
-    server.wait_for_termination()
 
+    # The serve method should wait for the server to terminate asynchronously
+    await server.wait_for_termination()
 
 if __name__ == "__main__":
-    serve()
+    # Use asyncio.run to run the asynchronous serve function
+    asyncio.run(serve())

app2.py

@@ -0,0 +1,456 @@
+from concurrent import futures
+import asyncio
+import torch
+from models import build_model
+from collections import deque
+import grpc
+import text_to_speech_pb2
+import text_to_speech_pb2_grpc
+from chat_database import save_chat_entry
+import fastAPI
+from providers.audio_provider import get_audio_bytes, dummy_bytes, generate_audio_stream
+from providers.llm_provider import getResponseWithRagAsync, getResponseAsync
+import whisper
+import numpy as np
+import os
+import re
+import time
+from silero_vad import load_silero_vad, VADIterator
+import random
+from providers.filler_words import filler_phrases
+from scipy.io.wavfile import write
+
+sampling_rate = 16_000
+vad_model = load_silero_vad()
+vad_iter = VADIterator(vad_model, sampling_rate=sampling_rate)
+frame_size = 512
+
+
+device = 'cuda' if torch.cuda.is_available() else 'cpu'
+whisper_model = whisper.load_model("small", device=device).to(device).eval()
+# whisper_model = torch.compile(whisper_model)
+
+MODEL = build_model('kokoro-v0_19.pth', device)
+VOICE_NAME = [
+    'af',
+    'af_bella', 'af_sarah', 'am_adam', 'am_michael',
+    'bf_emma', 'bf_isabella', 'bm_george', 'bm_lewis',
+    'af_nicole', 'af_sky',
+][0]
+
+
+VOICEPACK = torch.load(
+    f'voices/{VOICE_NAME}.pt', weights_only=True).to(device)
+
+AUDIO_FILES_DIR = 'audio_files'
+os.makedirs(AUDIO_FILES_DIR, exist_ok=True)
+
+PRE_CHUNK_LIMIT_BYTES = frame_size * 2 * 20
+
+transcription_pool = futures.ThreadPoolExecutor(max_workers=10)
+
+# terminators = ['.', '?', '!']
+
+terminators = ['.', '?', '!', '...', '…', '?!', '!?', '‽', '。', '؟', '۔']
+
+BLACKLIST = {
+    "Give me a minute.",
+    "Let me check the details.",
+    "Give me a minute. Let me check the details."
+}
+
+
+dummy_audio = np.frombuffer(
+    np.zeros(int(16_000 * 5.0), dtype=np.float32), dtype=np.int16).astype(np.float32) / 32768.0
+
+
+async def safe_transcribe(audio_float32):
+    loop = asyncio.get_running_loop()
+    return await loop.run_in_executor(
+        transcription_pool,
+        lambda: whisper_model.transcribe(audio_float32,
+                                         language="en",
+                                         fp16=False,
+                                         no_speech_threshold=0.25,
+                                         logprob_threshold=-0.6,
+                                         prompt="Indian English accent; do not make up words.")
+    )
+
+
+class TextToSpeechServicer(text_to_speech_pb2_grpc.TextToSpeechServiceServicer):
+
+    def __init__(self):
+        super().__init__()
+        self._transcribe_lock = asyncio.Lock()
+
+    async def ProcessText(self, request_iterator, context):
+        try:
+            global VOICEPACK
+
+            print("New connection")
+
+            tts_queue = asyncio.Queue()
+            response_queue = asyncio.Queue()
+
+            parameters = {
+                "processing_active": False,
+                "queue": deque(),
+                "file_number": 0,
+                "session_id": "",
+                "interrupt_seq": 0,
+                "temperature": 1,
+                "activeVoice": "af",
+                "in_speech": False,
+                "maxTokens": 500,
+                "audio_buffer": bytearray(),
+                "pre_chunks": bytearray(),
+                "silence_counter": 0.0,
+                "silence_duration": 0.8,  # default duration in seconds
+                "silence_threshold": 800,  # default amplitude threshold
+                "VOICEPACK": VOICEPACK,
+                "audio_count": 0,
+                "user_sequence": 0,
+                "last_file_number": 0
+            }
+
+            reader = asyncio.create_task(
+                self._read_requests(request_iterator, tts_queue, response_queue, parameters))
+
+            tts = asyncio.create_task(self._tts_queue_worker(
+                tts_queue, response_queue, parameters))
+
+            try:
+                while True:
+                    resp = await response_queue.get()
+                    if resp is None:
+                        break
+                    yield resp
+            finally:
+                reader.cancel()
+                tts.cancel()
+
+        except Exception as e:
+            print("Error in ProcessText:", e)
+
+    async def _read_requests(self, request_iterator, tts_queue: asyncio.Queue, response_queue: asyncio.Queue, parameters):
+        async for request in request_iterator:
+            field = request.WhichOneof('request_data')
+            if field == 'metadata':
+                meta = request.metadata
+                # print("\n\nMetadata : ", meta)
+                if meta.session_id:
+                    parameters["session_id"] = meta.session_id
+                if meta.temperature:
+                    parameters["temperature"] = meta.temperature
+                if meta.maxTokens:
+                    parameters["maxTokens"] = meta.maxTokens
+                if meta.activeVoice:
+                    parameters["activeVoice"] = meta.activeVoice
+                    parameters["VOICEPACK"] = torch.load(
+                        f'voices/{parameters["activeVoice"]}.pt', weights_only=True).to(device)
+                    print("\n\nVoice model loaded successfully")
+                if meta.silenceDuration:
+                    silence_duration = meta.silenceDuration / 1000
+                    parameters["silence_duration"] = silence_duration
+                if meta.threshold:
+                    parameters["silence_threshold"] = meta.threshold
+
+                print("\n\nPatameter : ", parameters)
+
+                # output = await safe_transcribe("output2.wav")
+
+                resp = text_to_speech_pb2.ProcessTextResponse(
+                    buffer=dummy_bytes(),
+                    session_id=parameters["session_id"],
+                    sequence_id="-10",
+                    transcript="",
+                )
+                await response_queue.put(resp)
+
+                continue
+            elif field == 'audio_data':
+
+                buffer = request.audio_data.buffer
+
+                audio_data = np.frombuffer(buffer, dtype=np.int16)
+
+                float_chunk = audio_data.astype(np.float32) / 32768.0
+
+                vad_result = vad_iter(float_chunk)
+
+                parameters["pre_chunks"].extend(buffer)
+                if len(parameters["pre_chunks"]) > PRE_CHUNK_LIMIT_BYTES:
+                    overflow = len(
+                        parameters["pre_chunks"]) - PRE_CHUNK_LIMIT_BYTES
+                    del parameters["pre_chunks"][:overflow]
+
+                if vad_result:
+                    if "start" in vad_result:
+                        parameters["in_speech"] = True
+                        parameters["audio_buffer"].extend(
+                            parameters["pre_chunks"])
+                    if "end" in vad_result:
+                        parameters["in_speech"] = False
+
+                if parameters["in_speech"]:
+                    parameters["audio_buffer"].extend(buffer)
+                    parameters["silence_counter"] = 0.0
+                    parameters["audio_count"] += 1
+                else:
+                    sample_rate = 16000
+                    duration = len(audio_data) / sample_rate
+                    parameters["silence_counter"] += duration
+
+                if parameters["silence_counter"] >= parameters["silence_duration"]:
+                    parameters["silence_counter"] = 0.0
+                    if parameters["audio_count"] < 2:
+                        parameters["audio_count"] = 0
+                        continue
+                    parameters["audio_count"] = 0
+                    print("Silence ")
+
+                    resp = text_to_speech_pb2.ProcessTextResponse(
+                        buffer=dummy_bytes(),
+                        session_id=parameters["session_id"],
+                        sequence_id="-3",
+                        transcript="",
+                    )
+                    await response_queue.put(resp)
+
+                    # resp = text_to_speech_pb2.ProcessTextResponse(
+                    #     buffer=dummy_bytes(),
+                    #     session_id=parameters["session_id"],
+                    #     sequence_id="0",
+                    #     transcript="",
+                    # )
+
+                    # await response_queue.put(resp)
+
+                    sample_rate = 16000
+
+                    audio_float = np.frombuffer(
+                        parameters["audio_buffer"], dtype=np.int16).astype(np.float32) / 32768.0
+
+                    parameters["audio_buffer"] = bytearray()
+
+                    whisper_start_time = time.time()
+                    result = ""
+                    try:
+                        result = await safe_transcribe(audio_float)
+                    except Exception as e:
+                        await tts_queue.put(("Sorry! I am not able to catch that can you repeat again please!", parameters["file_number"]))
+                        print("Error in transcribing text : ", e)
+                        continue
+
+                    whisper_end_time = time.time()
+                    time_taken_to_transcribe = whisper_end_time - whisper_start_time
+                    print(
+                        f"Transcribing time: {time_taken_to_transcribe:.4f} seconds")
+                    transcribed_text = result["text"]
+                    print(
+                        f"Transcribed Text :", transcribed_text)
+
+                    if not transcribed_text.strip():
+                        resp = text_to_speech_pb2.ProcessTextResponse(
+                            buffer=dummy_bytes(),
+                            session_id=parameters["session_id"],
+                            sequence_id="-5",
+                            transcript="",
+                        )
+                        await response_queue.put(resp)
+                        continue
+
+
+# Transcript Detected ------------------------------------------------------------------------------------
+
+                    if transcribed_text:
+                        parameters["queue"].clear()
+                        parameters["user_sequence"] += 1
+                        parameters["last_file_number"] = parameters["file_number"]
+                        while not response_queue.empty():
+                            try:
+                                response_queue.get_nowait()
+                                response_queue.task_done()
+                            except asyncio.QueueEmpty:
+                                break
+                        while not tts_queue.empty():
+                            try:
+                                tts_queue.get_nowait()
+                                tts_queue.task_done()
+                            except asyncio.QueueEmpty:
+                                break
+
+                        resp = text_to_speech_pb2.ProcessTextResponse(
+                            buffer=dummy_bytes(),
+                            session_id=parameters["session_id"],
+                            sequence_id="-4",
+                            transcript="",
+                        )
+                        await response_queue.put(resp)
+                        resp = text_to_speech_pb2.ProcessTextResponse(
+                            buffer=dummy_bytes(),
+                            session_id=parameters["session_id"],
+                            sequence_id="-2",
+                            transcript=transcribed_text,
+                        )
+                        save_chat_entry(
+                            parameters["session_id"], "user", transcribed_text)
+                        await response_queue.put(resp)
+
+                        try:
+                            filler = random.choice(filler_phrases)
+                            # await tts_queue.put((filler, parameters["file_number"]))
+                            loop = asyncio.get_event_loop()
+
+                            loop.call_later(
+                                0,
+                                # 1.0,
+                                lambda: asyncio.create_task(
+                                    tts_queue.put(
+                                        (filler, parameters["file_number"]))
+                                )
+                            )
+
+                        except Exception as e:
+                            print("Error in sendign error : ", e)
+                        final_response = ""
+                        complete_response = ""
+                        current_user_sequence = parameters["user_sequence"]
+                        response = await getResponseAsync(
+                            transcribed_text, parameters["session_id"])
+                        if response is None:
+                            continue
+                        for chunk in response:
+                            if (current_user_sequence != parameters["user_sequence"]):
+                                break
+                            msg = chunk.choices[0].delta.content
+                            if msg:
+                                complete_response += msg
+                                m = re.search(r'[.?!]', msg)
+                                if m:
+                                    idx = m.start()
+                                    segment = msg[:idx+1]
+                                    leftover = msg[idx+1:]
+                                else:
+                                    segment, leftover = msg, ''
+
+                                final_response += segment
+
+                                if segment.endswith(('.', '!', '?')):
+                                    parameters["file_number"] += 1
+                                    parameters["queue"].append(
+                                        (final_response, parameters["file_number"]))
+                                    await tts_queue.put((final_response, parameters["file_number"]))
+                                    final_response = leftover
+
+                        if final_response.strip():
+                            parameters["file_number"] += 1
+                            parameters["queue"].append(
+                                (final_response, parameters["file_number"]))
+                            await tts_queue.put((final_response, parameters["file_number"]))
+
+                        if ("Let me check" in complete_response):
+                            final_response = ""
+                            complete_response = ""
+                            current_user_sequence = parameters["user_sequence"]
+                            response = await getResponseWithRagAsync(
+                                transcribed_text, parameters["session_id"])
+                            for chunk in response:
+                                if (current_user_sequence != parameters["user_sequence"]):
+                                    break
+                                msg = chunk.choices[0].delta.content
+                                if msg:
+                                    m = re.search(r'[.?!]', msg)
+                                    if m:
+                                        idx = m.start()
+                                        segment = msg[:idx+1]
+                                        leftover = msg[idx+1:]
+                                    else:
+                                        segment, leftover = msg, ''
+
+                                    final_response += segment
+                                    complete_response += segment
+
+                                    if segment.endswith(('.', '!', '?')):
+                                        parameters["file_number"] += 1
+                                        parameters["queue"].append(
+                                            (final_response, parameters["file_number"]))
+                                        await tts_queue.put((final_response, parameters["file_number"]))
+                                        final_response = leftover
+
+                            if final_response.strip():
+                                parameters["file_number"] += 1
+                                parameters["queue"].append(
+                                    (final_response, parameters["file_number"]))
+                                await tts_queue.put((final_response, parameters["file_number"]))
+
+                continue
+
+            elif field == 'status':
+                transcript = request.status.transcript
+                played_seq = request.status.played_seq
+                interrupt_seq = request.status.interrupt_seq
+                parameters["interrupt_seq"] = interrupt_seq
+                text = transcript.strip() if transcript else ""
+                if text and text not in BLACKLIST:
+                    save_chat_entry(
+                        parameters["session_id"],
+                        "assistant",
+                        transcript
+                    )
+                continue
+            else:
+                continue
+
+    async def _tts_queue_worker(self, tts_queue: asyncio.Queue,
+                                response_queue: asyncio.Queue,
+                                params: dict):
+        """
+        Pull (text, seq) off tts_queue, run generate_audio_stream, wrap each chunk
+        in ProcessTextResponse, and push into response_queue.
+        """
+        while True:
+            item = await tts_queue.get()
+            tts_queue.task_done()
+            if item is None:
+                break
+
+            sentence, seq = item
+            # drop anything the client has already played:
+            if seq <= int(params["interrupt_seq"]):
+                continue
+
+            # stream the audio chunks, pack into gRPC responses
+            async for audio_chunk in generate_audio_stream(
+                    sentence, MODEL, params["VOICEPACK"], VOICE_NAME
+            ):
+                audio_bytes = get_audio_bytes(audio_chunk)
+                if seq <= int(params["last_file_number"]):
+                    break
+                resp = text_to_speech_pb2.ProcessTextResponse(
+                    buffer=audio_bytes,
+                    session_id=params["session_id"],
+                    sequence_id=str(seq),
+                    transcript=sentence,
+                )
+                await response_queue.put(resp)
+
+
+async def serve():
+    print("Starting gRPC server...")
+
+    # Use grpc.aio.server for the gRPC async server
+    server = grpc.aio.server(futures.ThreadPoolExecutor(max_workers=10))
+    text_to_speech_pb2_grpc.add_TextToSpeechServiceServicer_to_server(
+        TextToSpeechServicer(), server)
+    server.add_insecure_port('[::]:8081')
+
+    await server.start()
+    print("gRPC server is running on port 8081")
+
+    # The serve method should wait for the server to terminate asynchronously
+    await server.wait_for_termination()
+
+if __name__ == "__main__":
+    # Use asyncio.run to run the asynchronous serve function
+    asyncio.run(serve())

app_old.py

@@ -0,0 +1,190 @@
+from concurrent import futures
+import torch
+from models import build_model
+from collections import deque
+import grpc
+import text_to_speech_pb2
+import text_to_speech_pb2_grpc
+from chat_database import save_chat_entry
+import fastAPI
+from providers.audio_provider import get_audio_bytes, dummy_bytes, generate_audio_from_chunks
+from providers.llm_provider import getResponseWithRAG, getResponse
+
+
+device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+MODEL = build_model('kokoro-v0_19.pth', device)
+
+VOICE_NAME = [
+    'af',
+    'af_bella', 'af_sarah', 'am_adam', 'am_michael',
+    'bf_emma', 'bf_isabella', 'bm_george', 'bm_lewis',
+    'af_nicole', 'af_sky',
+][0]
+
+
+VOICEPACK = torch.load(
+    f'voices/{VOICE_NAME}.pt', weights_only=True).to(device)
+
+
+class TextToSpeechServicer(text_to_speech_pb2_grpc.TextToSpeechServiceServicer):
+    def ProcessText(self, request_iterator , context):
+        try:
+            global VOICEPACK
+            print("Received new request")
+            parameters = {
+                "processing_active": False,
+                "queue": deque(),
+                "file_number": 0,
+                "session_id": "",
+                "interrupt_seq": 0,
+                "temperature": 1,
+                "activeVoice": "af",
+                "maxTokens": 500,
+            }
+            for request in request_iterator:
+                field = request.WhichOneof('request_data')
+                if field == 'metadata':
+                    meta = request.metadata
+                    print("Metadata received:")
+                    print("  session_id:", meta.session_id)
+                    print("  silenceDuration:", meta.silenceDuration)
+                    print("  threshold:", meta.threshold)
+                    print("  temperature:", meta.temperature)
+                    print("  activeVoice:", meta.activeVoice)
+                    print("  maxTokens:", meta.maxTokens)
+                    print("Metadata : ", request.metadata)
+                    if meta.session_id:
+                        parameters["session_id"] = meta.session_id
+                    if meta.temperature:
+                        parameters["temperature"] = meta.temperature
+                    if meta.maxTokens:
+                        parameters["maxTokens"] = meta.maxTokens
+                    if meta.activeVoice:
+                        parameters["activeVoice"] = meta.activeVoice
+                        VOICEPACK = torch.load(
+                            f'voices/{parameters["activeVoice"]}.pt', weights_only=True).to(device)
+                    continue
+                elif field == 'text':
+                    text = request.text
+                    if not text:
+                        continue
+                    # yield text_to_speech_pb2.ProcessTextResponse(
+                    #     buffer=dummy_bytes(),
+                    #     session_id=parameters["session_id"],
+                    #     sequence_id="0",
+                    #     transcript="",
+                    # )
+                    # intent = check_for_rag(
+                    #     text, parameters["session_id"])
+                    # print("Intent : ", intent.intent)
+                    # print("Intent : ", intent.rag)
+                    save_chat_entry(parameters["session_id"], "user", text)
+                    parameters["queue"].clear()
+                    yield text_to_speech_pb2.ProcessTextResponse(
+                        buffer=dummy_bytes(),
+                        session_id=parameters["session_id"],
+                        sequence_id="-2",
+                        transcript=text,
+                    )
+                    final_response = ""
+                    complete_response = ""
+                    response = getResponse(text, parameters["session_id"])
+                    for chunk in response:
+                        msg = chunk.choices[0].delta.content
+                        if msg:
+                            final_response += msg
+                            complete_response += msg
+                            if final_response.endswith(('.', '!', '?')):
+                                parameters["file_number"] += 1
+                                parameters["queue"].append(
+                                    (final_response, parameters["file_number"]))
+                                final_response = ""
+                                if not parameters["processing_active"]:
+                                    yield from self.process_queue(parameters)
+
+                    if final_response:
+                        parameters["file_number"] += 1
+                        parameters["queue"].append(
+                            (final_response, parameters["file_number"]))
+                        if not parameters["processing_active"]:
+                            yield from self.process_queue(parameters)
+
+                    if ("Let me check" in complete_response):
+                        final_response = ""
+                        complete_response = ""
+                        response = getResponseWithRAG(
+                            text, parameters["session_id"])
+                        for chunk in response:
+                            msg = chunk.choices[0].delta.content
+                            if msg:
+                                final_response += msg
+                                complete_response += msg
+                                if final_response.endswith(('.', '!', '?')):
+                                    parameters["file_number"] += 1
+                                    parameters["queue"].append(
+                                        (final_response, parameters["file_number"]))
+                                    final_response = ""
+                                    if not parameters["processing_active"]:
+                                        yield from self.process_queue(parameters)
+
+                        if final_response:
+                            parameters["file_number"] += 1
+                            parameters["queue"].append(
+                                (final_response, parameters["file_number"]))
+                            if not parameters["processing_active"]:
+                                yield from self.process_queue(parameters)
+
+                elif field == 'status':
+                    transcript = request.status.transcript
+                    played_seq = request.status.played_seq
+                    interrupt_seq = request.status.interrupt_seq
+                    parameters["interrupt_seq"] = interrupt_seq
+                    save_chat_entry(
+                        parameters["session_id"], "assistant", transcript)
+                    continue
+                else:
+                    continue
+        except Exception as e:
+            print("Error in ProcessText:", e)
+
+    def process_queue(self, parameters):
+        global VOICEPACK
+        try:
+            while True:
+                if not parameters["queue"]:
+                    parameters["processing_active"] = False
+                    break
+                parameters["processing_active"] = True
+                sentence, file_number = parameters["queue"].popleft()
+                if file_number <= int(parameters["interrupt_seq"]):
+                    continue
+
+                combined_audio = generate_audio_from_chunks(
+                    sentence, MODEL, VOICEPACK, VOICE_NAME)
+                audio_bytes = get_audio_bytes(combined_audio)
+                # filename = save_audio_to_file(combined_audio, file_number)
+                yield text_to_speech_pb2.ProcessTextResponse(
+                    buffer=audio_bytes,
+                    session_id=parameters["session_id"],
+                    sequence_id=str(file_number),
+                    transcript=sentence,
+                )
+        except Exception as e:
+            parameters["processing_active"] = False
+            print("Error in process_queue:", e)
+
+
+def serve():
+    print("Starting gRPC server...")
+    server = grpc.server(futures.ThreadPoolExecutor(max_workers=1))
+    text_to_speech_pb2_grpc.add_TextToSpeechServiceServicer_to_server(
+        TextToSpeechServicer(), server)
+    server.add_insecure_port('[::]:8081')
+    server.start()
+    print("gRPC server is running on port 8081")
+    server.wait_for_termination()
+
+
+if __name__ == "__main__":
+    serve()

backend/.gitignore

@@ -1,2 +0,0 @@
-/node_modules
-.DS_Store

backend/app.js

@@ -9,14 +9,27 @@ app.use(express.urlencoded({ extended: true }));
 app.use(bodyParser.json());
 const port = 8080;
 
-const { audio_stream } = require("./handle-realtime-tts/sttModelSocket.js");
+const {
+  audio_stream,
+} = require("./handle-realtime-tts/sttModelSocket_whisper.js");
+const chatRouter = require("./routes/chat.routes.js");
+const ragRouter = require("./routes/rag.routes.js");
+const promptRouter = require("./routes/prompt.routes.js");
 
 app.get("/health", (req, res) => {
   res.send("Green");
 });
 
+app.get("/", (req, res) => {
+  res.send("VocRT express server working fine.");
+});
+
 app.ws("/v2v", audio_stream);
 
+app.use("/chat", chatRouter);
+app.use("/rag", ragRouter);
+app.use("/prompt", promptRouter);
+
 app.listen(port, () => {
-  console.log(`Example app listening at http://localhost:${port}`);
+  console.log(`VocRT express server listening at http://localhost:${port}`);
 });

backend/config.env

@@ -1 +0,0 @@
-DEEPGRAM_KEY = <deepgram_api_key>

backend/config.js

@@ -1,7 +0,0 @@
-require("dotenv").config({ path: "./config.env" });
-
-const deepgram_key = process.env.DEEPGRAM_KEY;
-
-module.exports = {
-  deepgram_key
-};

backend/controller/chat.js

@@ -0,0 +1,105 @@
+const axios = require("axios");
+
+const getChats = async (req, res) => {
+  try {
+    const response = await axios.get(`http://localhost:8082/get-chats`);
+    console.log(response.data);
+    if (!response.data) {
+      return res.status(200).json({
+        success: false,
+        message: "failed to get chats",
+      });
+    }
+    const chats = response.data;
+    const all_chats = chats.sort(
+      (a, b) => new Date(b.lastUpdatedAt) - new Date(a.lastUpdatedAt)
+    );
+    return res.status(200).json({
+      success: true,
+      message: "All chats",
+      chats: all_chats,
+    });
+  } catch (error) {
+    console.log(error);
+    return res
+      .status(500)
+      .json({ success: false, message: "Failed to get chat" });
+  }
+};
+const renameChats = async (req, res) => {
+  try {
+    const { sessionId, title } = req.body;
+    const response = await axios.post(`http://localhost:8082/rename-chat`, {
+      sessionId,
+      title,
+    });
+    console.log(response.data);
+    if (!response.data) {
+      return res.status(200).json({
+        success: false,
+        message: "failed to rename chats",
+      });
+    }
+    return res.status(200).json({
+      success: true,
+      message: "chat renamed!",
+      rename: { id: sessionId, title: title },
+    });
+  } catch (error) {
+    console.log(error);
+    return res
+      .status(500)
+      .json({ success: false, message: "Failed to get chat" });
+  }
+};
+
+const createChat = async (req, res) => {
+  try {
+    const { sessionId } = req.body;
+    console.log(sessionId);
+    const response = await axios.post(
+      `http://localhost:8082/create-chat/${sessionId}`
+    );
+    if (!response.data) {
+      return res
+        .status(500)
+        .json({ success: false, message: "Failed to create chat" });
+    }
+    return res.status(200).json({
+      success: true,
+      message: "Chat created successfully",
+      sessionId: sessionId,
+    });
+  } catch (error) {
+    console.log(error);
+    return res
+      .status(500)
+      .json({ success: false, message: "Failed to create chat" });
+  }
+};
+
+const deleteChat = async (req, res) => {
+  try {
+    const { sessionId } = req.body;
+    const response = await axios.post(`http://localhost:8082/delete-chat`, {
+      sessionId,
+    });
+    if (!response.data) {
+      return res
+        .status(500)
+        .json({ success: false, message: "Failed to delete chat" });
+    }
+    return res.status(200).json({
+      success: true,
+      message: "Chat deleted successfully",
+      sessionId: sessionId,
+    });
+  } catch (error) {
+    console.log(error);
+    return res
+      .status(500)
+      .json({ success: false, message: "Failed to delete chat" });
+  }
+};
+
+module.exports = { createChat, getChats, renameChats, deleteChat };

backend/controller/file.js

@@ -0,0 +1,189 @@
+const axios = require("axios");
+const FormData = require("form-data");
+
+const uploadPDF = async (req, res) => {
+  try {
+    const file = req.file;
+
+    if (!file) {
+      return res
+        .status(400)
+        .json({ success: false, message: "No file uploaded" });
+    }
+
+    const { name, sessionId, title, summary, categories } = req.body;
+
+    const formData = new FormData();
+    formData.append("pdf_file", file.buffer, file.originalname);
+    formData.append("name", name);
+    formData.append("sessionId", sessionId);
+    formData.append("title", title);
+    formData.append("summary", summary);
+    formData.append("categories", categories);
+
+    const response = await axios.post(
+      "http://localhost:8082/upload-pdf",
+      formData,
+      {
+        headers: {
+          ...formData.getHeaders(),
+        },
+      }
+    );
+
+    // fs.unlink(file.path, (err) => {
+    //   if (err) console.error("Error deleting the file:", err);
+    //   else console.log("File deleted successfully");
+    // });
+
+    return res.status(200).json({
+      success: true,
+      message: "PDF uploaded and processed successfully",
+      data: response.data,
+    });
+  } catch (error) {
+    console.warn("Error in uploadPDF:", error);
+
+    return res.status(500).json({
+      success: false,
+      message: "Failed to process PDF",
+      error: error.message,
+    });
+  }
+};
+
+const uploadLink = async (req, res) => {
+  try {
+    const { link, sessionId, title, summary, categories } = req.body;
+    const response = await axios.post("http://localhost:8082/process-link", {
+      link,
+      sessionId,
+      title,
+      summary,
+      categories,
+    });
+
+    console.info("Response : ", response.data);
+
+    if (response.data.status === "success") {
+      // If the FastAPI endpoint indicates success
+      return res.status(200).json({
+        success: true,
+        message: "Link processed successfully",
+        data: response.data, // Include any data if needed
+      });
+    } else {
+      // If the FastAPI endpoint indicates an error
+      return res.status(400).json({
+        success: false,
+        message:
+          response.data.detail || "Failed to process link. Please try again",
+      });
+    }
+  } catch (error) {
+    console.error("Error in uploading link : ", error);
+
+    if (error.response) {
+      const { status, data } = error.response;
+      return res.status(status).json({
+        success: false,
+        message: data.detail || "Failed to process link. Please try again",
+      });
+    } else if (error.request) {
+      console.error("No response received from the server:", error.request);
+      return res.status(500).json({
+        success: false,
+        message:
+          "No response received from the server. Please try again later.",
+      });
+    } else {
+      // Something happened in setting up the request that triggered an Error
+      console.error("Error in setting up the request:", error.message);
+      return res.status(500).json({
+        success: false,
+        message: "An unexpected error occurred. Please try again",
+      });
+    }
+  }
+};
+
+const uploadText = async (req, res) => {
+  try {
+    const { text, sessionId, name, title, summary, categories } = req.body;
+    const response = await axios.post("http://localhost:8082/process-text", {
+      text,
+      sessionId,
+      title,
+      name,
+      summary,
+      categories,
+    });
+
+    console.info("Response : ", response.data);
+
+    if (response.data.status === "success") {
+      // If the FastAPI endpoint indicates success
+      return res.status(200).json({
+        success: true,
+        message: "Link processed successfully",
+        data: response.data, // Include any data if needed
+      });
+    } else {
+      // If the FastAPI endpoint indicates an error
+      return res.status(400).json({
+        success: false,
+        message:
+          response.data.detail || "Failed to process link. Please try again",
+      });
+    }
+  } catch (error) {
+    console.error("Error in uploading link : ", error);
+
+    if (error.response) {
+      const { status, data } = error.response;
+      return res.status(status).json({
+        success: false,
+        message: data.detail || "Failed to process link. Please try again",
+      });
+    } else if (error.request) {
+      console.error("No response received from the server:", error.request);
+      return res.status(500).json({
+        success: false,
+        message:
+          "No response received from the server. Please try again later.",
+      });
+    } else {
+      // Something happened in setting up the request that triggered an Error
+      console.error("Error in setting up the request:", error.message);
+      return res.status(500).json({
+        success: false,
+        message: "An unexpected error occurred. Please try again",
+      });
+    }
+  }
+};
+
+const clearContext = async (req, res) => {
+  try {
+    const { sessionId } = req.body;
+    const response = await axios.post("http://localhost:8082/clear-context", {
+      sessionId,
+    });
+    const { data } = response;
+    if (data.status === "success") {
+      return res
+        .status(200)
+        .json({ success: true, message: "context cleared successfully" });
+    }
+    return res
+      .status(400)
+      .json({ message: "failed to clear context", success: false });
+  } catch (error) {
+    console.error("Error in clearing all context : ", error);
+    return res
+      .status(500)
+      .json({ message: "failed to clear context", success: false });
+  }
+};
+
+module.exports = { uploadPDF, uploadLink, clearContext, uploadText };

backend/controller/prompt.js

@@ -0,0 +1,31 @@
+const axios = require("axios");
+
+const savePrompt = async (req, res) => {
+  try {
+    const { sessionId, prompt } = req.body;
+    // console.log({ sessionId, prompt });
+    const response = await axios.post(`http://localhost:8082/save-prompt`, {
+      sessionId,
+      prompt,
+    });
+    console.log(response.data);
+    if (!response.data) {
+      return res.status(200).json({
+        success: false,
+        message: "failed to save prompt",
+      });
+    }
+    return res.status(200).json({
+      success: true,
+      message: "prompt saved!",
+      rename: { id: sessionId, prompt: prompt },
+    });
+  } catch (error) {
+    console.log(error);
+    return res
+      .status(500)
+      .json({ success: false, message: "Failed to save prompt" });
+  }
+};
+
+module.exports = { savePrompt };

backend/handle-realtime-tts/makegRPCconnection.js

@@ -5,7 +5,7 @@ const path = require("path");
 const getgRPCConnection = (session) => {
   return new Promise((resolve, reject) => {
     protoLoader
-      .load(path.join(__dirname, "text_to_speech.proto"), {
+      .load(path.join(__dirname, "text_to_speech_whisper.proto"), {
         keepCase: true,
         longs: String,
         enums: String,
@@ -13,7 +13,8 @@ const getgRPCConnection = (session) => {
         oneofs: true,
       })
       .then((packageDefinition) => {
-        const textToSpeechProto = grpc.loadPackageDefinition(packageDefinition).texttospeech;
+        const textToSpeechProto =
+          grpc.loadPackageDefinition(packageDefinition).texttospeech;
         const client = new textToSpeechProto.TextToSpeechService(
           "localhost:8081",
           grpc.credentials.createInsecure()
@@ -24,10 +25,8 @@ const getgRPCConnection = (session) => {
         console.log("Made connection");
         session.client = client;
 
-
         const call = client.ProcessText();
         resolve(call);
-
       })
       .catch((error) => {
         session.client = null;
@@ -37,4 +36,4 @@ const getgRPCConnection = (session) => {
   });
 };
 
-module.exports = { getgRPCConnection };
+module.exports = { getgRPCConnection };

backend/handle-realtime-tts/sttModelSocket.js

@@ -1,12 +1,12 @@
 const isBuffer = require("is-buffer");
 const { Buffer } = require("buffer");
-const {deepgram_key} = require("../config");
+// const { deepgram_key } = require("../config");
 const Session = require("../utils/session.js");
 const { cleanupConnection } = require("./cleangRPCconnections.js");
 const { getgRPCConnection } = require("./makegRPCconnection.js");
 const { updateChathistory } = require("../providers/updateChathistory.js");
 const { createClient, LiveTranscriptionEvents } = require("@deepgram/sdk");
-const deepgram = createClient(deepgram_key);
+// const deepgram = createClient(deepgram_key);
 
 const audio_stream = async (wss, req) => {
   try {
@@ -14,15 +14,20 @@ const audio_stream = async (wss, req) => {
 
     wss.send(JSON.stringify({ type: "initial", msg: "connected" }));
 
-
     const connection = deepgram.listen.live({
       punctuate: true,
       interim_results: true,
       speech_final: true,
       encoding: "linear16",
       sample_rate: 16000,
+      // model: "nova-2-conversationalai",
+      // utterance_end_ms: "500",
+      endpointing: 500,
+      // model: "nova-2-phonecall",
       model: "nova-2",
-      speech_final: true,
+      // model: "nova-2-general",
+      language: "en",
+      // language: "en-IN",
       version: "latest",
     });
 
@@ -32,8 +37,7 @@ const audio_stream = async (wss, req) => {
       } catch (error) {
         console.error("Error in calling ml server : ", error);
       }
-    }
-
+    };
 
     connection.on(LiveTranscriptionEvents.Open, () => {
       console.log(LiveTranscriptionEvents.Open);
@@ -59,7 +63,6 @@ const audio_stream = async (wss, req) => {
       });
     });
 
-
     wss.on("message", async (message) => {
       try {
         if (isBuffer(message) && session.call) {
@@ -105,14 +108,34 @@ const audio_stream = async (wss, req) => {
 
                   const {
                     sessionId,
+                    silenceDuration,
+                    threshold,
+                    temperature,
+                    activeVoice,
+                    maxTokens,
                   } = JSON.parse(msg);
+                  console.log({
+                    sessionId,
+                    silenceDuration,
+                    threshold,
+                    temperature,
+                    activeVoice,
+                    maxTokens,
+                  });
                   const metadata = {
                     metadata: {
-                      session_id: sessionId,
+                      session_id: String(sessionId),
+                      silenceDuration: parseInt(silenceDuration, 10) || 100,
+                      threshold: parseInt(threshold, 10) || 100,
+                      temperature: parseFloat(temperature) || 0.7,
+                      activeVoice: String(activeVoice),
+                      maxTokens: parseInt(maxTokens, 10) || 500,
                     },
                   };
+
+                  console.log(metadata);
                   if (session.call) {
-                    console.log("Sending metadata.")
+                    console.log("Sending metadata.");
                     session.call.write(metadata);
                   }
                 } catch (err) {
@@ -122,7 +145,8 @@ const audio_stream = async (wss, req) => {
                 session.call.on("data", (response) => {
                   console.log("Data : ", response);
 
-                  const {session_id , sequence_id , transcript , buffer} = response;
+                  const { session_id, sequence_id, transcript, buffer } =
+                    response;
 
                   const metadata = JSON.stringify({
                     session_id: session_id,
@@ -134,12 +158,12 @@ const audio_stream = async (wss, req) => {
                     session.latency = Date.now();
                     wss.send(JSON.stringify({ type: "clear", msg: "clear" }));
                     session.chathistory = [...session.chathistorybackup];
-                    wss.send(
-                      JSON.stringify({
-                        type: "chathistory",
-                        msg: session.chathistorybackup,
-                      })
-                    );
+                    // wss.send(
+                    //   JSON.stringify({
+                    //     type: "chathistory",
+                    //     msg: session.chathistorybackup,
+                    //   })
+                    // );
                     const wavBuffer = Buffer.concat([
                       Buffer.from(metadata),
                       Buffer.from([0]),
@@ -156,8 +180,11 @@ const audio_stream = async (wss, req) => {
                     });
                     wss.send(
                       JSON.stringify({
-                        type: "chathistory",
-                        msg: session.chathistory,
+                        type: "chat",
+                        msg: {
+                          role: "user",
+                          content: transcript,
+                        },
                       })
                     );
                     session.chathistorybackup.push({
@@ -212,8 +239,11 @@ const audio_stream = async (wss, req) => {
 
                   wss.send(
                     JSON.stringify({
-                      type: "chathistory",
-                      msg: session.chathistory,
+                      type: "chat",
+                      msg: {
+                        role: "ai",
+                        content: transcript,
+                      },
                     })
                   );
                 });
@@ -223,7 +253,7 @@ const audio_stream = async (wss, req) => {
                   await cleanupConnection(session);
                   try {
                     wss.send(JSON.stringify({ type: "end", msg: "end" }));
-                  } catch (err) { }
+                  } catch (err) {}
                   console.log("Stream ended");
                 });
 
@@ -231,7 +261,7 @@ const audio_stream = async (wss, req) => {
                   console.error(`Stream error: ${error}`);
                   try {
                     wss.send(JSON.stringify({ type: "end", msg: "end" }));
-                  } catch (err) { }
+                  } catch (err) {}
                   await cleanupConnection(session);
                 });
                 break;
@@ -240,7 +270,7 @@ const audio_stream = async (wss, req) => {
                 const { session_id, sequence_id, transcript } = msg;
                 const status = {
                   status: {
-                    transcript : transcript,
+                    transcript: transcript,
                     played_seq: sequence_id,
                     interrupt_seq: sequence_id,
                   },
@@ -280,9 +310,9 @@ const audio_stream = async (wss, req) => {
     });
   } catch (err) {
     try {
-      console.log(err)
+      console.log(err);
       wss.send(JSON.stringify({ type: "end", msg: "end" }));
-    } catch (err) { }
+    } catch (err) {}
   }
 };
 

backend/handle-realtime-tts/sttModelSocket_whisper.js

@@ -0,0 +1,300 @@
+const isBuffer = require("is-buffer");
+const { Buffer } = require("buffer");
+const Session = require("../utils/session.js");
+const { cleanupConnection } = require("./cleangRPCconnections.js");
+const { getgRPCConnection } = require("./makegRPCconnection.js");
+const { updateChathistory } = require("../providers/updateChathistory.js");
+
+const audio_stream = async (wss, req) => {
+  try {
+    const session = new Session();
+
+    wss.send(JSON.stringify({ type: "initial", msg: "connected" }));
+
+    wss.on("message", async (message) => {
+      try {
+        if (isBuffer(message) && session.call) {
+          try {
+            const audio_message = {
+              audio_data: {
+                buffer: message,
+              },
+            };
+
+            try {
+              // Whisper
+              session.call.write(audio_message);
+            } catch (error) {
+              console.log("Error sending buffer to deepgram : ", error);
+            }
+          } catch (err) {
+            console.error("Error writing to stream: ", err);
+          }
+        }
+
+        if (typeof message === "string") {
+          try {
+            const data = JSON.parse(message);
+
+            const { type, msg } = data;
+
+            switch (type) {
+              case "start":
+                session.starttime = Date.now();
+                session.chathistory = [];
+                session.chathistorybackup = [];
+                console.log("Making Connection with gRPC...");
+                try {
+                  console.time("grpcconnection");
+                  session.call = await getgRPCConnection(session);
+                  console.timeEnd("grpcconnection");
+                  const state = session.channel.getConnectivityState(false);
+                  console.log(`Client : ${state}`);
+                  session.saved = false;
+                  wss.send(JSON.stringify({ type: "ready", msg: "connected" }));
+                  console.log("Connected to gRPC.");
+
+                  const {
+                    sessionId,
+                    silenceDuration,
+                    threshold,
+                    temperature,
+                    activeVoice,
+                    maxTokens,
+                  } = JSON.parse(msg);
+                  console.log({
+                    sessionId,
+                    silenceDuration,
+                    threshold,
+                    temperature,
+                    activeVoice,
+                    maxTokens,
+                  });
+
+                  console.log(silenceDuration);
+                  const metadata = {
+                    metadata: {
+                      session_id: String(sessionId),
+                      silenceDuration: parseInt(silenceDuration * 1000) || 800,
+                      threshold: parseInt(threshold, 10) || 100,
+                      temperature: parseFloat(temperature, 10) || 0.7,
+                      activeVoice: String(activeVoice),
+                      maxTokens: parseInt(maxTokens, 10) || 500,
+                    },
+                  };
+
+                  console.log(metadata);
+                  if (session.call) {
+                    console.log("Sending metadata.");
+                    session.call.write(metadata);
+                  }
+                } catch (err) {
+                  await cleanupConnection(session);
+                  console.error("Error in making gRPC Connection. : ", err);
+                }
+                session.call.on("data", (response) => {
+                  const { session_id, sequence_id, transcript, buffer } =
+                    response;
+
+                  const metadata = JSON.stringify({
+                    session_id: session_id,
+                    sequence_id: sequence_id,
+                    transcript: transcript,
+                  });
+
+                  if (sequence_id === "-2") {
+                    session.latency = Date.now();
+                    wss.send(JSON.stringify({ type: "clear", msg: "clear" }));
+                    session.chathistory = [...session.chathistorybackup];
+                    // wss.send(
+                    //   JSON.stringify({
+                    //     type: "chathistory",
+                    //     msg: session.chathistorybackup,
+                    //   })
+                    // );
+                    const wavBuffer = Buffer.concat([
+                      Buffer.from(metadata),
+                      Buffer.from([0]),
+                      buffer,
+                    ]);
+
+                    const base64buffer = wavBuffer.toString("base64");
+                    wss.send(
+                      JSON.stringify({ type: "media", msg: base64buffer })
+                    );
+                    session.chathistory.push({
+                      speaker: "USER",
+                      content: transcript,
+                    });
+                    wss.send(
+                      JSON.stringify({
+                        type: "chat",
+                        msg: {
+                          role: "user",
+                          content: transcript,
+                        },
+                      })
+                    );
+                    session.chathistorybackup.push({
+                      speaker: "USER",
+                      content: transcript,
+                    });
+                    return;
+                  }
+
+                  if (sequence_id === "0") {
+                    wss.send(JSON.stringify({ type: "pause", msg: "pause" }));
+                    session.cansend = false;
+                    return;
+                  }
+
+                  if (sequence_id === "-3") {
+                    wss.send(
+                      JSON.stringify({
+                        type: "transcribing",
+                        msg: "transcribing",
+                      })
+                    );
+                    return;
+                  }
+                  if (sequence_id === "-5") {
+                    wss.send(
+                      JSON.stringify({
+                        type: "stop_transcribing",
+                        msg: "stop_transcribing",
+                      })
+                    );
+                    return;
+                  }
+                  if (sequence_id === "-10") {
+                    wss.send(
+                      JSON.stringify({
+                        type: "connected",
+                        msg: "connected",
+                      })
+                    );
+                    return;
+                  }
+                  if (sequence_id === "-4") {
+                    wss.send(
+                      JSON.stringify({ type: "thinking", msg: "thinking" })
+                    );
+                    return;
+                  }
+
+                  if (sequence_id === "-1") {
+                    wss.send(
+                      JSON.stringify({ type: "continue", msg: "continue" })
+                    );
+                    return;
+                  }
+
+                  if (sequence_id === "1") {
+                    const latency = Date.now() - session.latency;
+                    session.latency = 0;
+                    // wss.send(JSON.stringify({ type: "clear", msg: "clear" }));
+                    session.cansend = true;
+                  }
+
+                  if (!buffer) {
+                    return;
+                  }
+
+                  if (!session.cansend && sequence_id !== "0") {
+                    return;
+                  }
+
+                  // Combine header and PCM data into a single Buffer
+                  const wavBuffer = Buffer.concat([
+                    Buffer.from(metadata),
+                    Buffer.from([0]),
+                    buffer,
+                  ]);
+
+                  const base64buffer = wavBuffer.toString("base64");
+                  wss.send(
+                    JSON.stringify({ type: "media", msg: base64buffer })
+                  );
+
+                  updateChathistory(transcript, false, session);
+
+                  wss.send(
+                    JSON.stringify({
+                      type: "chat",
+                      msg: {
+                        role: "ai",
+                        content: transcript,
+                      },
+                    })
+                  );
+                });
+
+                session.call.on("end", async () => {
+                  console.log("Ended");
+                  await cleanupConnection(session);
+                  try {
+                    wss.send(JSON.stringify({ type: "end", msg: "end" }));
+                  } catch (err) {}
+                  console.log("Stream ended");
+                });
+
+                session.call.on("error", async (error) => {
+                  console.error(`Stream error: ${error}`);
+                  try {
+                    wss.send(JSON.stringify({ type: "end", msg: "end" }));
+                  } catch (err) {}
+                  await cleanupConnection(session);
+                });
+                break;
+
+              case "status":
+                const { session_id, sequence_id, transcript } = msg;
+                const status = {
+                  status: {
+                    transcript: transcript,
+                    played_seq: sequence_id,
+                    interrupt_seq: sequence_id,
+                  },
+                };
+
+                if (session.call) {
+                  session.call.write(status);
+                }
+
+                updateChathistory(transcript, true, session);
+                break;
+
+              case "stop":
+                console.log("Client Stoped the stream.");
+                await cleanupConnection(session);
+                break;
+              default:
+                console.log("Type not handled.");
+            }
+          } catch (err) {
+            console.log(`Not a valid json : ${err}`);
+          }
+        }
+      } catch (err) {
+        console.error(`Error in wss.onmessage : ${err}`);
+      }
+    });
+
+    wss.on("close", async () => {
+      await cleanupConnection(session);
+      console.log("WebSocket connection closed.");
+    });
+
+    wss.on("error", async (err) => {
+      console.error(`WebSocket error: ${err}`);
+      await cleanupConnection(session);
+    });
+  } catch (err) {
+    try {
+      console.log(err);
+      wss.send(JSON.stringify({ type: "end", msg: "end" }));
+    } catch (err) {}
+  }
+};
+
+module.exports = { audio_stream };

backend/handle-realtime-tts/text_to_speech.proto

@@ -23,6 +23,11 @@ message ProcessTextResponse {
 
 message Meta {
   string session_id = 1;
+  int32 silenceDuration = 2;
+  int32 threshold = 3;
+  float temperature = 4;
+  string activeVoice = 5;
+  int32 maxTokens = 6;
 }
 
 message Status {

backend/handle-realtime-tts/text_to_speech_whisper.proto

@@ -0,0 +1,41 @@
+syntax = "proto3";
+
+package texttospeech;
+
+service TextToSpeechService {
+  rpc ProcessText (stream ProcessTextRequest) returns (stream ProcessTextResponse);
+}
+
+message ProcessTextRequest {
+  oneof request_data {
+    Audio audio_data = 1;
+    Meta metadata = 2;
+    Status status = 3;
+  }
+}
+
+message ProcessTextResponse {
+  bytes buffer = 1;
+  string session_id = 2;
+  string sequence_id = 3;
+  string transcript = 4;
+}
+
+message Meta {
+  string session_id = 1;
+  int32 silenceDuration = 2;
+  int32 threshold = 3;
+  float temperature = 4;
+  string activeVoice = 5;
+  int32 maxTokens = 6;
+}
+
+message Status {
+  string transcript = 1;
+  string played_seq = 2;
+  string interrupt_seq = 3;
+}
+
+message Audio {
+  bytes buffer = 1;
+}

backend/package-lock.json

@@ -12,7 +12,7 @@
         "@deepgram/sdk": "^3.9.0",
         "@geckos.io/server": "^3.0.0",
         "@grpc/grpc-js": "^1.11.3",
-        "axios": "^1.7.9",
+        "axios": "^1.8.4",
         "bcryptjs": "^2.4.3",
         "cors": "^2.8.5",
         "crypto": "^1.0.1",
@@ -21,7 +21,8 @@
         "express-ws": "^5.0.2",
         "is-buffer": "^2.0.5",
         "jsonwebtoken": "^9.0.2",
-        "module": "^1.2.5"
+        "module": "^1.2.5",
+        "multer": "^1.4.5-lts.2"
       }
     },
     "node_modules/@deepgram/captions": {
@@ -276,6 +277,11 @@
         "url": "https://github.com/chalk/ansi-styles?sponsor=1"
       }
     },
+    "node_modules/append-field": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/append-field/-/append-field-1.0.0.tgz",
+      "integrity": "sha512-klpgFSWLW1ZEs8svjfb7g4qWY0YS5imI82dTg+QahUvJ8YqAY0P10Uk8tTyh9ZGuYEZEMaeJYCF5BFuX552hsw=="
+    },
     "node_modules/arr-diff": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/arr-diff/-/arr-diff-2.0.0.tgz",
@@ -325,9 +331,9 @@
       }
     },
     "node_modules/axios": {
-      "version": "1.7.9",
-      "resolved": "https://registry.npmjs.org/axios/-/axios-1.7.9.tgz",
-      "integrity": "sha512-LhLcE7Hbiryz8oMDdDptSrWowmB4Bl6RCt6sIJKpRB4XtVf0iEgewX3au/pJqm+Py1kCASkb/FFKjxQaLtxJvw==",
+      "version": "1.8.4",
+      "resolved": "https://registry.npmjs.org/axios/-/axios-1.8.4.tgz",
+      "integrity": "sha512-eBSYY4Y68NNlHbHBMdeDmKNtDgXWhQsJcGqzO3iLUM0GraQFSS9cVgPX5I9b3lbdFKyYoAEGAZF1DwhTaljNAw==",
       "dependencies": {
         "follow-redirects": "^1.15.6",
         "form-data": "^4.0.0",
@@ -446,6 +452,22 @@
       "resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
       "integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA=="
     },
+    "node_modules/buffer-from": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
+      "integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ=="
+    },
+    "node_modules/busboy": {
+      "version": "1.6.0",
+      "resolved": "https://registry.npmjs.org/busboy/-/busboy-1.6.0.tgz",
+      "integrity": "sha512-8SFQbg/0hQ9xy3UNTB0YEnsNBbWfhf7RtnzpL7TkBiTBRfrQ9Fxcnz7VJsleJpyp6rVLvXiuORqjlHi5q+PYuA==",
+      "dependencies": {
+        "streamsearch": "^1.1.0"
+      },
+      "engines": {
+        "node": ">=10.16.0"
+      }
+    },
     "node_modules/bytes": {
       "version": "3.1.2",
       "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
@@ -2301,6 +2323,74 @@
       "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
       "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="
     },
+    "node_modules/multer": {
+      "version": "1.4.5-lts.2",
+      "resolved": "https://registry.npmjs.org/multer/-/multer-1.4.5-lts.2.tgz",
+      "integrity": "sha512-VzGiVigcG9zUAoCNU+xShztrlr1auZOlurXynNvO9GiWD1/mTBbUljOKY+qMeazBqXgRnjzeEgJI/wyjJUHg9A==",
+      "dependencies": {
+        "append-field": "^1.0.0",
+        "busboy": "^1.0.0",
+        "concat-stream": "^1.5.2",
+        "mkdirp": "^0.5.4",
+        "object-assign": "^4.1.1",
+        "type-is": "^1.6.4",
+        "xtend": "^4.0.0"
+      },
+      "engines": {
+        "node": ">= 6.0.0"
+      }
+    },
+    "node_modules/multer/node_modules/concat-stream": {
+      "version": "1.6.2",
+      "resolved": "https://registry.npmjs.org/concat-stream/-/concat-stream-1.6.2.tgz",
+      "integrity": "sha512-27HBghJxjiZtIk3Ycvn/4kbJk/1uZuJFfuPEns6LaEvpvG1f0hTea8lilrouyo9mVc2GWdcEZ8OLoGmSADlrCw==",
+      "engines": [
+        "node >= 0.8"
+      ],
+      "dependencies": {
+        "buffer-from": "^1.0.0",
+        "inherits": "^2.0.3",
+        "readable-stream": "^2.2.2",
+        "typedarray": "^0.0.6"
+      }
+    },
+    "node_modules/multer/node_modules/process-nextick-args": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.1.tgz",
+      "integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag=="
+    },
+    "node_modules/multer/node_modules/readable-stream": {
+      "version": "2.3.8",
+      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz",
+      "integrity": "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA==",
+      "dependencies": {
+        "core-util-is": "~1.0.0",
+        "inherits": "~2.0.3",
+        "isarray": "~1.0.0",
+        "process-nextick-args": "~2.0.0",
+        "safe-buffer": "~5.1.1",
+        "string_decoder": "~1.1.1",
+        "util-deprecate": "~1.0.1"
+      }
+    },
+    "node_modules/multer/node_modules/safe-buffer": {
+      "version": "5.1.2",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
+      "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+    },
+    "node_modules/multer/node_modules/string_decoder": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz",
+      "integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==",
+      "dependencies": {
+        "safe-buffer": "~5.1.0"
+      }
+    },
+    "node_modules/multer/node_modules/typedarray": {
+      "version": "0.0.6",
+      "resolved": "https://registry.npmjs.org/typedarray/-/typedarray-0.0.6.tgz",
+      "integrity": "sha512-/aCDEGatGvZ2BIk+HmLf4ifCJFwvKFNb9/JeZPMulfgFracn9QFcAf5GO8B/mweUjSoblS5In0cWhqpfs/5PQA=="
+    },
     "node_modules/napi-build-utils": {
       "version": "1.0.2",
       "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-1.0.2.tgz",
@@ -3195,6 +3285,14 @@
       "resolved": "https://registry.npmjs.org/stream-shift/-/stream-shift-1.0.3.tgz",
       "integrity": "sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ=="
     },
+    "node_modules/streamsearch": {
+      "version": "1.1.0",
+      "resolved": "https://registry.npmjs.org/streamsearch/-/streamsearch-1.1.0.tgz",
+      "integrity": "sha512-Mcc5wHehp9aXz1ax6bZUyY5afg9u2rv5cqQI3mRrYkGC8rW2hM02jWuwjtL++LS5qinSyhj2QfLyNsuc+VsExg==",
+      "engines": {
+        "node": ">=10.0.0"
+      }
+    },
     "node_modules/string_decoder": {
       "version": "1.3.0",
       "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",

backend/package.json

@@ -10,10 +10,9 @@
   "author": "",
   "license": "ISC",
   "dependencies": {
-    "@deepgram/sdk": "^3.9.0",
     "@geckos.io/server": "^3.0.0",
     "@grpc/grpc-js": "^1.11.3",
-    "axios": "^1.7.9",
+    "axios": "^1.8.4",
     "bcryptjs": "^2.4.3",
     "cors": "^2.8.5",
     "crypto": "^1.0.1",
@@ -22,6 +21,7 @@
     "express-ws": "^5.0.2",
     "is-buffer": "^2.0.5",
     "jsonwebtoken": "^9.0.2",
-    "module": "^1.2.5"
+    "module": "^1.2.5",
+    "multer": "^1.4.5-lts.2"
   }
 }

backend/routes/chat.routes.js

@@ -0,0 +1,15 @@
+const express = require("express");
+const {
+  createChat,
+  getChats,
+  renameChats,
+  deleteChat,
+} = require("../controller/chat");
+const chatRouter = express.Router();
+
+chatRouter.post("/create-chat", createChat);
+chatRouter.post("/get-chats", getChats);
+chatRouter.post("/rename-chat", renameChats);
+chatRouter.post("/delete-chat", deleteChat);
+
+module.exports = chatRouter;

backend/routes/prompt.routes.js

@@ -0,0 +1,7 @@
+const express = require("express");
+const promptRouter = express.Router();
+const { savePrompt } = require("../controller/prompt");
+
+promptRouter.post("/", savePrompt);
+
+module.exports = promptRouter;

backend/routes/rag.routes.js

@@ -0,0 +1,66 @@
+const express = require("express");
+const {
+  uploadPDF,
+  uploadLink,
+  clearContext,
+  uploadText,
+} = require("../controller/file");
+const multer = require("multer");
+const path = require("path");
+const ragRouter = express.Router();
+
+// const storage = multer.diskStorage({
+//   destination: function (req, file, cb) {
+//     cb(null, "uploads/");
+//   },
+//   filename: function (req, file, cb) {
+//     cb(null, Date.now() + "-" + file.originalname);
+//   },
+// });
+
+const storage = multer.memoryStorage();
+
+const fileFilter = function (req, file, cb) {
+  const allowedExt = /\.(pdf|csv|ppt|pptx|doc|docx|xls|xlsx|txt)$/i;
+
+  // Allowed MIME types
+  const allowedMime = [
+    "application/pdf",
+    "text/csv",
+    "application/vnd.ms-powerpoint",
+    "application/vnd.openxmlformats-officedocument.presentationml.presentation",
+    "application/msword",
+    "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+    "application/vnd.ms-excel",
+    "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
+    "text/plain",
+  ];
+
+  // Check extension
+  const extname = allowedExt.test(path.extname(file.originalname));
+  // Check mime
+  const mimetype = allowedMime.includes(file.mimetype);
+
+  if (extname && mimetype) {
+    cb(null, true);
+  } else {
+    cb(
+      new Error(
+        "Invalid file type. Only document files are allowed: PDF, CSV, PPT(X), DOC(X), XLS(X), TXT."
+      ),
+      false
+    );
+  }
+};
+
+const upload = multer({
+  storage: storage,
+  fileFilter: fileFilter,
+});
+
+ragRouter.post("/pdf", upload.single("pdfFile"), uploadPDF);
+ragRouter.post("/link", uploadLink);
+ragRouter.post("/text", uploadText);
+ragRouter.post("/clear-context", clearContext);
+
+module.exports = ragRouter;

chat_database.py

@@ -1,5 +1,6 @@
 import pickle
 import os
+from datetime import datetime, timezone
 
 FILE_PATH = "chat_history.pkl"
 
@@ -7,28 +8,107 @@ if not os.path.exists(FILE_PATH):
     with open(FILE_PATH, "wb") as file:
         pickle.dump({}, file)
 
+
+async def save_context_detail(session_id, name, title, summary, categories):
+    try:
+        try:
+            with open(FILE_PATH, "rb") as file:
+                data = pickle.load(file)
+        except (FileNotFoundError, EOFError):
+            data = {}
+
+        now = datetime.now(timezone.utc).isoformat()
+
+        if session_id not in data:
+            print("Session id not in data")
+            data[session_id] = {
+                "title": "New Chat",
+                "createdAt": now,
+                "lastUpdatedAt": now,
+                "chat": [],
+                "context": [],
+                "prompt": "",
+            }
+
+        session = data.get(session_id)
+        contexts = session.get("context", [])
+
+        contexts.append({"name": name, "title": title,
+                        "summary": summary, "categories": categories})
+
+        data[session_id]["lastUpdatedAt"] = now
+
+        with open(FILE_PATH, "wb") as file:
+            pickle.dump(data, file)
+
+    except Exception as e:
+        print(f"Error saving context entry: {e}")
+
+
+def clear_context_detail(session_id):
+    try:
+        try:
+            with open(FILE_PATH, "rb") as file:
+                data = pickle.load(file)
+        except (FileNotFoundError, EOFError):
+            data = {}
+
+        now = datetime.now(timezone.utc).isoformat()
+
+        if session_id not in data:
+            print("Session id not in data")
+            return False
+
+        data[session_id]["context"] = []
+
+        data[session_id]["lastUpdatedAt"] = now
+
+        with open(FILE_PATH, "wb") as file:
+            pickle.dump(data, file)
+
+    except Exception as e:
+        print(f"Error saving context entry: {e}")
+
+
 def save_chat_entry(session_id, role, transcript):
     try:
-        with open(FILE_PATH, "rb") as file:
-            data = pickle.load(file)
+        try:
+            with open(FILE_PATH, "rb") as file:
+                data = pickle.load(file)
+        except (FileNotFoundError, EOFError):
+            data = {}
+
+        now = datetime.now(timezone.utc).isoformat()
 
         if session_id not in data:
-            data[session_id] = []
+            print("Session id not in data")
+            data[session_id] = {
+                "title": "New Chat",
+                "createdAt": now,
+                "lastUpdatedAt": now,
+                "chat": [],
+                "context": [],
+                "prompt": "",
+            }
+
+        messages = data[session_id]["chat"]
 
         if role == "user":
-            data[session_id].append({
+            messages.append({
                 "role": role,
                 "transcript": transcript
             })
         else:
-            if data[session_id] and data[session_id][-1]['role'] == "assistant":
-                data[session_id][-1]['transcript'] += " " + transcript
+            if messages and messages[-1]["role"] == "assistant":
+                messages[-1]["transcript"] += " " + transcript
             else:
-                data[session_id].append({
+                messages.append({
                     "role": role,
                     "transcript": transcript
                 })
 
+        data[session_id]["lastUpdatedAt"] = now
+
         with open(FILE_PATH, "wb") as file:
             pickle.dump(data, file)
 
@@ -36,29 +116,191 @@ def save_chat_entry(session_id, role, transcript):
         print(f"Error saving chat entry: {e}")
 
 
-def get_chat_history(session_id):
+def get_chat_history(session_id, limit=15):
     try:
         with open(FILE_PATH, "rb") as file:
             data = pickle.load(file)
 
-        chat_history = data.get(session_id, [])
-
-        if not chat_history:
+        session = data.get(session_id)
+        if not session or not isinstance(session, dict):
             return []
 
-        message_history = []
-        for entry in chat_history:
-            role = entry.get('role', '')
-            transcript = entry.get('transcript', '') 
-            if role and transcript:
-                message_history.append({"role": role, "content": transcript})
+        # or "messages" if you’ve standardized on that
+        # messages = session.get("chat", [])
+
+        # message_history = []
+        # for entry in messages:
+        #     role = entry.get('role', '')
+        #     transcript = entry.get('transcript', '')
+        #     if role and transcript:
+        #         message_history.append({"role": role, "content": transcript})
 
-        return message_history
+        # return message_history[-15:]
+
+        tail = session.get("chat", [])[-limit:]
+        chat_history = [
+            {"role": msg["role"], "content": msg["transcript"]}
+            for msg in tail
+            if msg.get("role") and msg.get("transcript")
+        ]
+
+        user_prompt = session.get("prompt", "")
+        return chat_history, user_prompt
 
     except (FileNotFoundError, pickle.UnpicklingError) as e:
         print(f"Error reading or parsing the file: {e}")
-        return [] 
+        return []
     except Exception as e:
         print(f"Unexpected error: {e}")
-        return [] 
+        return []
+
 
+def get_all_chat_details():
+    try:
+        with open(FILE_PATH, "rb") as file:
+            data = pickle.load(file)
+
+        chat_list = []
+
+        for session_id, chat in data.items():
+            if not isinstance(chat, dict):
+                continue
+
+            messages = []
+            for entry in chat.get("chat", []):
+                role = entry.get("role", "")
+                transcript = entry.get("transcript", "")
+                if role and transcript:
+                    messages.append({
+                        "role": role,
+                        "content": transcript
+                    })
+
+            chat_list.append({
+                "id": session_id,
+                "title": chat.get("title", "Untitled"),
+                "createdAt": chat.get("createdAt"),
+                "lastUpdatedAt": chat.get("lastUpdatedAt"),
+                "chat": messages,
+                "context": chat.get("context", []),
+                "prompt": chat.get("prompt", ""),
+            })
+
+        return chat_list
+
+    except (FileNotFoundError, EOFError):
+        return []
+
+    except Exception as e:
+        print(f"Error reading chats: {e}")
+        return []
+
+
+def create_chat_entry(session_id):
+    try:
+        # Load existing data or initialize an empty dict
+        try:
+            with open(FILE_PATH, "rb") as file:
+                data = pickle.load(file)
+        except (FileNotFoundError, EOFError):
+            data = {}
+
+        now = datetime.now(timezone.utc).isoformat()
+
+        if session_id not in data:
+            data[session_id] = {
+                "title": "New Chat",
+                "createdAt": now,
+                "lastUpdatedAt": now,
+                "chat": [],
+                "context": [],
+            }
+
+            # Save the updated data back to file
+            with open(FILE_PATH, "wb") as file:
+                pickle.dump(data, file)
+
+        return True
+
+    except Exception as e:
+        print(f"Error create chat entry : {e}")
+        return False
+
+
+def rename_chat_title(session_id, title):
+    try:
+        try:
+            with open(FILE_PATH, "rb") as file:
+                data = pickle.load(file)
+        except (FileNotFoundError, EOFError):
+            data = {}
+
+        if session_id not in data:
+            return False
+
+        data[session_id]["title"] = title
+        data[session_id]["lastUpdatedAt"] = datetime.now(
+            timezone.utc).isoformat()
+
+        with open(FILE_PATH, "wb") as file:
+            pickle.dump(data, file)
+
+        print(f"Renamed chat: {data[session_id]}")
+        return True
+
+    except Exception as e:
+        print(f"Error renaming chat title: {e}")
+        return False
+
+
+def save_system_prompt(session_id, prompt):
+    try:
+        try:
+            with open(FILE_PATH, "rb") as file:
+                data = pickle.load(file)
+        except (FileNotFoundError, EOFError):
+            data = {}
+
+        now = datetime.now(timezone.utc).isoformat()
+
+        if session_id not in data:
+            return False
+
+        data[session_id]["prompt"] = prompt
+        data[session_id]["lastUpdatedAt"] = now
+
+        with open(FILE_PATH, "wb") as file:
+            pickle.dump(data, file)
+
+        print(f"Saved Prompt : {data[session_id]}")
+        return True
+
+    except Exception as e:
+        print(f"Error saving context entry: {e}")
+        return False
+
+
+def delete_chat(session_id):
+    try:
+        try:
+            with open(FILE_PATH, "rb") as file:
+                data = pickle.load(file)
+        except (FileNotFoundError, EOFError):
+            data = {}
+
+        if session_id not in data:
+            return True
+
+        data.pop(session_id)
+
+        with open(FILE_PATH, "wb") as file:
+            pickle.dump(data, file)
+
+        if session_id not in data:
+            return True
+
+        return False
+
+    except Exception as e:
+        print(f"Error deleting chat: {e}")
+        return False

chat_history.pkl

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d229c43b598eacb4620b5bf033308ff27c1e9979af506afd1e58d7e6ba24c9da
-size 12508
+oid sha256:f5a6e279b1243be5d92db335883c01146efde4af08a42a887b5fd326ed2a3636
+size 73280