Project Overview
This project showcases a production-grade, hybrid AI orchestration stack deployed entirely on a self-hosted Proxmox homelab. The primary objective was to overcome the strict hardware constraints of consumer-grade GPUs by cleanly decoupling the management plane from the heavy computational execution layers.
By designing an on-demand VRAM orchestration mechanism and pairing it with local-to-cloud fallback gateways, this stack delivers enterprise-level agentic performance, absolute data privacy, and web-grounded Retrieval-Augmented Generation (RAG) without breaking the bank on infrastructure costs.
System Architecture & Flow
The framework divides responsibilities between highly isolated, low-overhead container environments and a dedicated hardware-accelerated execution node.
- Management & Frontend Layer (LXC): Requests enter the stack via Open WebUI or a custom Telegram client interface. The core management ecosystem—encompassing Redis for session caching, LiteLLM for routing, and SearXNG for privacy-focused metasearch—is containerized within lightweight Proxmox LXCs to minimize idle memory overhead.
- Intelligent API Routing (LiteLLM): LiteLLM functions as a centralized gateway for all incoming prompt traffic. It evaluates local model availability and dynamically routes requests, executing seamless fallback chains to high-capacity remote APIs (such as Gemini or OpenRouter) if local compute queues are saturated.
- Dynamic VRAM & Inference Engine (Hardware-Passthrough VM): Heavy compute workloads are delegated to a dedicated VM with direct PCIe passthrough of an AMD RX 580 GPU utilizing a Vulkan backend. To maximize the GPU’s 8GB VRAM limit,
llama-swapdynamically loads local GGUF models (like Qwen3.5) on-demand with a 5-minute Time-To-Live (TTL), while lightweightBGE-M3embedding models remain permanently anchored in CPU RAM for instant vector retrieval. - Agentic RAG Pipeline (Custom MCP Server): Autonomous execution is handled by a Nous Hermes agent. A custom Model Context Protocol (MCP) server written in Node.js bridges the agent with the internal SearXNG instance, enabling the model to safely query the web, synthesize live data, and return private answers directly to the user.
Key Technical Specifications
- VRAM Optimization: Automated model hot-swapping utilizing
llama-swapwith a 5-minute TTL, allowing multiple large GGUF models to share a single 8GB physical GPU frame buffer. - Failover Resilience: Zero-downtime routing policy driven by LiteLLM fallback matrices, guaranteeing high availability even during heavy local processing spikes.
- Retrieval Latency: Near-zero execution overhead for vector embeddings by pinning
BGE-M3directly to system CPU memory, leaving the GPU entirely open for text generation tokens. - Data Privacy: Complete network-level isolation of local inference endpoints, ensuring confidential agentic tasks and local documents never traverse the public cloud.
Deployment & Usage
The management plane and the inference nodes are initialized separately to maintain the architectural boundary between control and execution.