LlamaGPT is a self-hosted, offline chatbot application that replicates ChatGPT functionality while maintaining complete privacy by running entirely on the user's device. Built with TypeScript and powered by Llama 2 models, the project ensures that no data leaves the user's machine during operation. The application recently expanded its capabilities to include Code Llama support and Nvidia GPU acceleration, broadening its appeal to both general users and developers.
The repository supports multiple model variants across different sizes and specializations. Users can choose from Nous Hermes Llama 2 models in 7B, 13B, and 70B parameter configurations, all quantized to GGML q4_0 format. Additionally, Code Llama models are available in 7B, 13B, and 34B variants using GGUF Q4_K_M quantization. The 7B Nous Hermes model requires 6.29GB of RAM and downloads at 3.79GB, while the larger 70B variant demands 41.37GB of memory with a 38.87GB download size. This range of options allows deployment across diverse hardware configurations from resource-constrained devices to high-end servers.
Installation flexibility is a core strength of the project. Users can deploy LlamaGPT on umbrelOS home servers through a single-click installation via the Umbrel App Store, on M1 and M2 Macs with Docker support, on any x86 or arm64 system with Docker, or within Kubernetes clusters. The application includes an OpenAI-compatible API endpoint at localhost:3001 that provides drop-in replacement functionality, enabling integration with existing tools and workflows designed for OpenAI's API.
Performance benchmarks demonstrate significant variation based on hardware. On an M1 Max MacBook Pro with 64GB RAM, the 7B model achieves 54 tokens per second, while the same hardware generates only 4.8 tokens per second with the 70B model. More modest hardware shows proportionally reduced performance, with a Raspberry Pi 4 managing 0.9 tokens per second on the 7B model. These benchmarks span multiple device classes including Google Cloud Platform instances, consumer CPUs, and specialized home server hardware.
The project maintains an active development roadmap with several completed milestones including model volume separation from Docker images, Metal support for Apple Silicon, Code Llama integration, and CUDA support for Nvidia GPUs. Remaining priorities include the ability to load custom models and functionality allowing users to switch between different models at runtime.
GitGenius activity tracking reveals moderate engagement patterns with a median issue and pull request response latency of 4421.9 hours and a mean of 4906.5 hours across 16 tracked items. The most active contributors include Ualas with 4 recorded events, adevart with 3 events, and feasly with 2 events. The repository shares overlapping contributors with microsoft/typescript, sveltejs/svelte, and microsoft/vscode, indicating connections to broader web development and tooling ecosystems. The project is classified across numerous categories including AI models, chatbots, local deployment, privacy-focused applications, language modeling, and large language model research.
