GGUF Loader
**GGUF Loader is a plugin-based desktop platform for running and extending local LLMs — with built-in floating tools that give you instant AI anywhere on your screen Run offline LLMs with one click, load custom models, and summon a floating AI assistant on top of any app — all from your desktop.
The problem:
Running open-source LLMs locally is powerful but painful. It’s either command-line only or scattered across multiple tools. There’s no GUI that brings it together — no ecosystem, no UX, no quick way to make LLMs useful in daily tasks like summarizing PDFs, replying to emails, or writing inside your browser.
The solution:
GGUF Loader gives users a beautiful desktop interface, one-click model loading, and a plugin system inspired by Blender. But it goes further: with its built-in floating button, users can summon AI from anywhere on their screen — to summarize, reply, translate, or search — all using fully offline models. It’s a privacy-first productivity layer that turns LLMs into personal agents you can drag, click, and extend with plugins.
✍️ 2. Product Vision (2 Paragraphs) The problem: Running open-source LLMs locally is powerful but frustrating. Users face messy installs, scattered tools, CLI-only interfaces, and no way to extend functionality without code. Even power users lack a smooth workflow to manage models, summarize content, or automate tasks using their own machine.
The solution: GGUF Loader turns your PC into a local AI platform. With a modern GUI, one-click model loading, and a Blender-style plugin system, users can add summarizers, floating agents, RAG tools, and more — all running offline. Whether you're a developer, researcher, or AI tinkerer, GGUF Loader gives you a stable, extensible foundation for intelligent tools that respect your privacy and run 24/7 without cloud lock-in.
🎯 Mission Statement
We believe AI shouldn’t live in the cloud — it should live on your screen, always-on, fully yours. GGUF Loader is building the interface layer for the local LLM revolution: a plugin-based platform with floating assistants, developer extensibility, and a vision to empower millions with intelligent local tools.
📂 Repository & Website
- GitHub: GGUF Loader on GitHub
- Website: https://ggufloader.github.io
🚀 New in GGUF Loader v2.0.1 (July 2025) 🧩 Floating Assistant Button A persistent, system-wide AI helper that hovers over your screen — just select text anywhere, and GGUF Loader pops up with a smart summary or reply.
🧩 🎬 Demo Video: Addon System + Floating Tool in Local LLM (v2.0.1 Update)
Discover how to supercharge your local AI workflows using the new floating addon system! No coding needed. Works offline. 🔌 Add-on System (Blender-style Plugins) Build your own AI tools inside GGUF Loader! Addons now live directly in the chat UI with toggle switches — think PDF summarizers, spreadsheet bots, email assistants, and more.
💻 Zero-Terminal Workflow Still pip-installed, but now behaves like a true desktop app — no terminal required after setup.
🛠️ New Plugin API & UI Docking Plugin developers can now define custom UIs, dock panels into chat view, and use context from chats or system clipboard.
📌 Perfect for Power Users & Tinkerers Use GGUF Loader not just as a model runner — but as a base platform for building your own offline AI workspace.
🔖 Model Card
This “model” repository hosts the Model Card and optional demo Space for GGUF Loader, a desktop application that loads, manages, and chats with GGUF‑format large language models entirely offline.
📝 Description
GGUF Loader is simple software that lets you easily run advanced AI language models (LLMs) like Mistral, LLaMA, and DeepSeek on Windows, macOS, and Linux. It has a drag-and-drop graphical interface, so loading models is quick and easy.
- ✨ GUI‑First: No terminal commands; point‑and‑click interface
- 🔌 Plugin System: Extend with addons (PDF summarizer, email assistant, spreadsheet automator…)
- ⚡️ Lightweight: Runs on machines as modest as Intel i5 + 16 GB RAM
- 🔒 Offline & Private: All inference happens locally—no cloud calls
🎯 Intended Uses
- Local AI prototyping: Experiment with open GGUF models without API costs
- Privacy‑focused demos: Chat privately with LLMs on your own machine
- Plugin workflows: Build custom data‑processing addons (e.g. summarization, code assistant)
⚠️ Limitations
- No cloud integration: Purely local, no access to OpenAI or Hugging Face inference APIs
- GUI only: No headless server/CLI‑only mode (coming soon)
- Requires Python 3.8+ and dependencies (
llama-cpp-python,PySide6)
🚀 How to Use
1. Install
pip install ggufloader
2. Launch GUI
ggufloader
3. Load Your Model
- Drag & drop your
.ggufmodel file into the window - Select plugin(s) from the sidebar (e.g. “Summarize PDF”)
- Start chatting!
4. Python API
from ggufloader import chat
# Ensure you have a GGUF model in ./models/mistral.gguf
chat("Hello offline world!", model_path="./models/mistral.gguf")
📦 Features
| Feature | Description |
|---|---|
| GUI for GGUF LLMs | Point‑and‑click model loading & chatting |
| Plugin Addons | Summarization, code helper, email reply, more |
| Cross‑Platform | Windows, macOS, Linux |
| Multi‑Model Support | Mistral, LLaMA, DeepSeek, Yi, Gemma, OpenHermes |
| Memory‑Efficient | Designed to run on 16 GB RAM or higher |
💡 Comparison
| Tool | GUI | Plugins | Pip Install | Offline | Notes |
|---|---|---|---|---|---|
| GGUF Loader | ✅ | ✅ | ✅ | ✅ | Modular, drag‑and‑drop UI |
| LM Studio | ✅ | ❌ | ❌ | ✅ | More polished, less extensible |
| Ollama | ❌ | ❌ | ❌ | ✅ | CLI‑first, narrow use case |
| GPT4All | ✅ | ❌ | ✅ | ✅ | Limited plugin support |
🔗 Demo Space
Try a static demo or minimal Gradio embed (no live inference) here:
https://huggingface.co/spaces/Hussain2050/gguf-loader-demo
📚 Citation
If you use GGUF Loader in your research or project, please cite:
@misc{ggufloader2025,
title = {GGUF Loader: Local GUI & Plugin‑Based Runner for GGUF Format LLMs},
author = {Hussain Nazary},
year = {2025},
howpublished = {\url{https://github.com/GGUFloader/gguf-loader}},
note = {Version 1.0.2, PyPI: ggufloader}
}
license: mit
🧠 GGUF Loader Quickstart
📦 1. Install GGUF Loader via pip
pip install ggufloader
🚀 2. Launch the App
After installation, run the following command in your terminal:
ggufloader
This will start the GGUF Loader interface. You can now load and chat with any GGUF model locally.
Let me know if you want to support GUI launching, system tray, or shortcuts too.
🔽 Download GGUF Models
⚡ Click a link below to download the model file directly (no Hugging Face page in between).
🧠 Mistral-7B Instruct
🧠 Qwen 1.5-7B Chat
🧠 DeepSeek 7B Chat
🧠 LLaMA 3 8B Instruct
🗂️ More Model Collections
⚖️ License
This project is licensed under the MIT License. See LICENSE for details.
Last updated: July 11, 2025