Spaces:
Running
on
Zero
Running
on
Zero
File size: 3,626 Bytes
f3fcd77 5dfbac5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
---
title: SmolVLM2 On Transformers
emoji: 🐠
colorFrom: blue
colorTo: pink
sdk: gradio
sdk_version: 5.34.0
app_file: app.py
pinned: false
license: mit
short_description: ' SmolVLM2 on transformers'
---
# 🎥 Real‑Time Webcam Captioning with SmolVLM2
This project provides a Gradio‑based web app that captures your webcam feed and generates natural‑language captions in real time using the SmolVLM2 image‑to‑text model family. It supports CPU, CUDA GPUs, and Intel XPU accelerators, with configurable capture intervals and custom prompts.
## 🚀 Features
* **Real‑time captions** from your webcam stream
* Support for **CPU**, **CUDA** (NVIDIA GPU), and **XPU** (Intel accelerator)
* Adjustable **capture interval** (default: 3000 ms)
* Custom **system** and **user** prompts for fine‑tuning the caption style
* **Debug logging** of preprocessing, tokenization, inference, and decoding times
* Model caching to avoid repeated reloads on parameter changes
## 🛠️ Prerequisites
* **Python** 3.9+ (tested on 3.12)
* **Git** (to clone this repo)
* **Intel oneAPI** and **PyTorch XPU** build (if you plan to use XPU)
### Python packages
```
pip install torch transformers gradio opencv-python pillow
```
*(If you have CUDA‑enabled hardware, install a CUDA‑compatible torch build; if you have XPU hardware, ensure you install the Intel XPU‑patched torch.)*
## 📦 Installation
1. **Clone** this repository:
```bash
git clone https://github.com/<your‑org>/smolvml2‑webcam‑caption.git
cd smolvml2‑webcam‑caption
```
2. **Create** and **activate** a virtual environment:
```bash
python -m venv .venv
source .venv/bin/activate
```
3. **Install** dependencies:
```bash
pip install -r requirements.txt
```
## ▶️ Usage
1. **Run** the app:
```bash
python app.py
```
2. **Open** the URL printed by Gradio in your browser (usually `http://127.0.0.1:7860`).
3. **Select**:
* **Model ID**: choose between `SmolVLM2-256M`, `SmolVLM2-500M`, or `SmolVLM2-2.2B`.
* **Device**: `cpu`, `cuda`, or `xpu` (if available).
4. **Adjust** the **Interval (ms)** slider to set how often frames are captioned.
5. **Edit** the **System Prompt** and **User Prompt** to control the style and content of captions.
6. **Allow** webcam access and watch live captions appear along with debug logs.
## ⚙️ Configuration Options
| Parameter | Description | Default |
| ------------- | ----------------------------------------------- | ---------------------------------- |
| Model ID | HuggingFace repo for SmolVLM2 model | `SmolVLM2-256M-Video-Instruct` |
| Device | Compute device: `cpu`, `cuda`, or `xpu` | `cuda` if available |
| Interval (ms) | Delay between frame captures (milliseconds) | `3000` |
| System Prompt | Instruction guiding the caption style | `Describe the key action` |
| User Prompt | Question shown to the model alongside the image | `What is happening in this image?` |
## 🐞 Troubleshooting
* **KeyError** in Gradio event handling: ensure `demo.queue()` is called before `demo.launch()`.
* **XPU BFloat16 errors**: XPU currently handles float32 more reliably; the code defaults to float32 on XPU.
* **Model reload slowness**: models are cached—changing only prompts or intervals won’t trigger a reload.
## 📜 License
This project is licensed under the **MIT License**. See [LICENSE](LICENSE) for details.
---
*Developed by Luigi Liu (2025)*
|