initial commit

README.md

@@ -11,4 +11,89 @@ license: mit
 short_description: ' SmolVLM2 on transformers'
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🎥 Real‑Time Webcam Captioning with SmolVLM2
+
+This project provides a Gradio‑based web app that captures your webcam feed and generates natural‑language captions in real time using the SmolVLM2 image‑to‑text model family. It supports CPU, CUDA GPUs, and Intel XPU accelerators, with configurable capture intervals and custom prompts.
+
+## 🚀 Features
+
+* **Real‑time captions** from your webcam stream
+* Support for **CPU**, **CUDA** (NVIDIA GPU), and **XPU** (Intel accelerator)
+* Adjustable **capture interval** (default: 3000 ms)
+* Custom **system** and **user** prompts for fine‑tuning the caption style
+* **Debug logging** of preprocessing, tokenization, inference, and decoding times
+* Model caching to avoid repeated reloads on parameter changes
+
+## 🛠️ Prerequisites
+
+* **Python** 3.9+ (tested on 3.12)
+* **Git** (to clone this repo)
+* **Intel oneAPI** and **PyTorch XPU** build (if you plan to use XPU)
+
+### Python packages
+
+```
+pip install torch transformers gradio opencv-python pillow
+```
+
+*(If you have CUDA‑enabled hardware, install a CUDA‑compatible torch build; if you have XPU hardware, ensure you install the Intel XPU‑patched torch.)*
+
+## 📦 Installation
+
+1. **Clone** this repository:
+
+   ```bash
+   git clone https://github.com/<your‑org>/smolvml2‑webcam‑caption.git
+   cd smolvml2‑webcam‑caption
+   ```
+2. **Create** and **activate** a virtual environment:
+
+   ```bash
+   python -m venv .venv
+   source .venv/bin/activate
+   ```
+3. **Install** dependencies:
+
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+## ▶️ Usage
+
+1. **Run** the app:
+
+   ```bash
+   python app.py
+   ```
+2. **Open** the URL printed by Gradio in your browser (usually `http://127.0.0.1:7860`).
+3. **Select**:
+
+   * **Model ID**: choose between `SmolVLM2-256M`, `SmolVLM2-500M`, or `SmolVLM2-2.2B`.
+   * **Device**: `cpu`, `cuda`, or `xpu` (if available).
+4. **Adjust** the **Interval (ms)** slider to set how often frames are captioned.
+5. **Edit** the **System Prompt** and **User Prompt** to control the style and content of captions.
+6. **Allow** webcam access and watch live captions appear along with debug logs.
+
+## ⚙️ Configuration Options
+
+| Parameter     | Description                                     | Default                            |
+| ------------- | ----------------------------------------------- | ---------------------------------- |
+| Model ID      | HuggingFace repo for SmolVLM2 model             | `SmolVLM2-256M-Video-Instruct`     |
+| Device        | Compute device: `cpu`, `cuda`, or `xpu`         | `cuda` if available                |
+| Interval (ms) | Delay between frame captures (milliseconds)     | `3000`                             |
+| System Prompt | Instruction guiding the caption style           | `Describe the key action`          |
+| User Prompt   | Question shown to the model alongside the image | `What is happening in this image?` |
+
+## 🐞 Troubleshooting
+
+* **KeyError** in Gradio event handling: ensure `demo.queue()` is called before `demo.launch()`.
+* **XPU BFloat16 errors**: XPU currently handles float32 more reliably; the code defaults to float32 on XPU.
+* **Model reload slowness**: models are cached—changing only prompts or intervals won’t trigger a reload.
+
+## 📜 License
+
+This project is licensed under the **MIT License**. See [LICENSE](LICENSE) for details.
+
+---
+
+*Developed by Luigi Liu (2025)*

app.py

@@ -0,0 +1,143 @@
+import time
+import logging
+import gradio as gr
+import cv2
+import os
+from transformers import AutoProcessor, AutoModelForImageTextToText
+import torch
+from PIL import Image
+
+# Cache for loaded model and processor
+default_cache = {'model_id': None, 'processor': None, 'model': None, 'device': None}
+model_cache = default_cache.copy()
+
+# Check for XPU availability
+has_xpu = hasattr(torch, 'xpu') and torch.xpu.is_available()
+
+def update_model(model_id, device):
+    if model_cache['model_id'] != model_id or model_cache['device'] != device:
+        logging.info(f'Loading model {model_id} on {device}')
+        processor = AutoProcessor.from_pretrained(model_id)
+        # Load model with appropriate precision for each device
+        if device == 'cuda':
+            # Use bfloat16 for CUDA for performance
+            model = AutoModelForImageTextToText.from_pretrained(
+                model_id,
+                torch_dtype=torch.bfloat16,
+                _attn_implementation='flash_attention_2'
+            ).to('cuda')
+        elif device == 'xpu' and has_xpu:
+            # Use float32 on XPU to avoid bfloat16 layernorm issues
+            model = AutoModelForImageTextToText.from_pretrained(
+                model_id,
+                torch_dtype=torch.float32
+            ).to('xpu')
+        else:
+            # Default to float32 on CPU
+            model = AutoModelForImageTextToText.from_pretrained(model_id).to('cpu')
+        model.eval()
+        model_cache.update({'model_id': model_id, 'processor': processor, 'model': model, 'device': device})
+
+
+def caption_frame(frame, model_id, interval_ms, sys_prompt, usr_prompt, device):
+    debug_msgs = []
+    update_model(model_id, device)
+    processor = model_cache['processor']
+    model = model_cache['model']
+
+    # Control capture interval
+    time.sleep(interval_ms / 1000)
+
+    # Preprocess frame
+    t0 = time.time()
+    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+    pil_img = Image.fromarray(rgb)
+    temp_path = 'frame.jpg'
+    pil_img.save(temp_path, format='JPEG', quality=50)
+    debug_msgs.append(f'Preprocess: {int((time.time()-t0)*1000)} ms')
+
+    # Prepare multimodal chat messages
+    messages = [
+        {'role': 'system', 'content': [{'type': 'text', 'text': sys_prompt}]},
+        {'role': 'user', 'content': [
+            {'type': 'image', 'url': temp_path},
+            {'type': 'text', 'text': usr_prompt}
+        ]}
+    ]
+
+    # Tokenize and encode
+    t1 = time.time()
+    inputs = processor.apply_chat_template(
+        messages,
+        add_generation_prompt=True,
+        tokenize=True,
+        return_dict=True,
+        return_tensors='pt'
+    ).to(model.device)
+    debug_msgs.append(f'Tokenize: {int((time.time()-t1)*1000)} ms')
+
+    # Inference
+    t2 = time.time()
+    outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
+    debug_msgs.append(f'Inference: {int((time.time()-t2)*1000)} ms')
+
+    # Decode and strip history
+    t3 = time.time()
+    raw = processor.batch_decode(outputs, skip_special_tokens=True)[0]
+    debug_msgs.append(f'Decode: {int((time.time()-t3)*1000)} ms')
+    if "Assistant:" in raw:
+        caption = raw.split("Assistant:")[-1].strip()
+    else:
+        lines = raw.splitlines()
+        caption = lines[-1].strip() if len(lines) > 1 else raw.strip()
+
+    return caption, '\n'.join(debug_msgs)
+
+
+def main():
+    logging.basicConfig(level=logging.INFO)
+    model_choices = [
+        'HuggingFaceTB/SmolVLM2-256M-Video-Instruct',
+        'HuggingFaceTB/SmolVLM2-500M-Video-Instruct',
+        'HuggingFaceTB/SmolVLM2-2.2B-Instruct'
+    ]
+    # Determine available devices
+    device_options = ['cpu']
+    if torch.cuda.is_available():
+        device_options.append('cuda')
+    if has_xpu:
+        device_options.append('xpu')
+
+    default_device = 'cuda' if torch.cuda.is_available() else ('xpu' if has_xpu else 'cpu')
+
+    with gr.Blocks() as demo:
+        gr.Markdown('## 🎥 Real-Time Webcam Captioning with SmolVLM2 (Transformers)')
+
+        with gr.Row():
+            model_dd  = gr.Dropdown(model_choices, value=model_choices[0], label='Model ID')
+            device_dd = gr.Dropdown(device_options, value=default_device, label='Device')
+
+        interval   = gr.Slider(100, 20000, step=100, value=3000, label='Interval (ms)')
+        sys_p      = gr.Textbox(lines=2, value='Describe the key action', label='System Prompt')
+        usr_p      = gr.Textbox(lines=1, value='What is happening in this image?', label='User Prompt')
+
+        cam        = gr.Image(sources=['webcam'], streaming=True, label='Webcam Feed')
+        caption_tb = gr.Textbox(interactive=False, label='Caption')
+        log_tb     = gr.Textbox(lines=4, interactive=False, label='Debug Log')
+
+        cam.stream(
+            fn=caption_frame,
+            inputs=[cam, model_dd, interval, sys_p, usr_p, device_dd],
+            outputs=[caption_tb, log_tb],
+            time_limit=600
+        )
+
+    # Enable Gradio's async event queue to register callback IDs and prevent KeyErrors
+    demo.queue()
+
+    # Launch the app
+    demo.launch()
+
+
+if __name__ == '__main__':
+    main()

packages.txt

@@ -0,0 +1 @@
+python3-opencv

requirements.txt

@@ -0,0 +1,8 @@
+gradio>=5.0
+opencv-python
+pillow
+huggingface-hub
+termcolor
+transformers
+torch
+num2words