DwarfStar is a specialized native inference engine written in C, designed specifically for running DeepSeek V4 Flash and PRO models locally, with additional support for GLM 5.2. Unlike general-purpose GGUF runners, it is deliberately narrow in scope, with model loading, prompt rendering, tool calls, KV state management, HTTP server functionality, and coding agent capabilities all built and tested together as an integrated system. The project is optimized first for Metal on Apple Silicon Macs with 96 GB or more of RAM, though it also supports NVIDIA CUDA including multi-GPU configurations and AMD ROCm on Strix Halo systems like the Framework Desktop. For machines with insufficient RAM, SSD streaming enables reasonable performance on smaller systems.
The repository is maintained by antirez and represents a significant engineering effort built with assistance from advanced language models including GPT 5.5, 5.6, and Claude Fable, though humans led the ideas, testing, and debugging. The project explicitly acknowledges its debt to llama.cpp and GGML, retaining certain source-level components under the MIT license including GGUF quantization layouts, CPU quantization logic, and specific kernels. While ds4.c does not link against GGML, the project exists because of the path opened by llama.cpp's kernels, quantization formats, and engineering knowledge.
DwarfStar enables several practical use cases. Consumer hardware users can run capable models on MacBooks, DGX Spark systems, or Strix Halo devices, with SSD streaming providing decent speeds even on memory-constrained machines. Server deployments can leverage CUDA multi-GPU support with micro-batching to convert older NVIDIA cards no longer supported by vLLM into multi-user LLM servers, with testing showing 120 tokens per second aggregate generation and 2000 tokens per second prefill on 8xL40S configurations. Advanced setups support tensor parallelism across two MacBook M5 Max or M3 Ultra systems via RDMA, and pipeline parallelism can combine multiple systems to aggregate their RAM for larger models.
The project's design reflects specific motivations: capable open-weight models now fit on high-end personal machines, DeepSeek V4 Flash and PRO along with GLM 5.2 tolerate aggressive routed-expert quantization, compressed KV caches and fast local SSDs make long contexts practical, and a specialized inference system for a few models offers advantages over general approaches. Model support is intentionally opportunistic, following the best open weights for useful local machine sizes, particularly 128 GB laptops and 512 GB workstations, with models removed when better replacements arrive.
The repository includes comprehensive tooling and documentation. The gguf-tools directory contains offline GGUF generation, imatrix collection, quantization tooling, and quality checks. Separate README files document GGUF tools, imatrix collection for routed-MoE models, calibration prompt corpus generation, quality testing against official continuations, directional steering data, speed benchmarking, and test vectors for regression checks. The project uses asymmetrical quantization strategies, quantizing only routed MoE experts while leaving shared experts, projections, and routing components untouched to guarantee quality.
DwarfStar includes experimental speculative decoding support through DSpark, an auxiliary draft model that reads hidden states from the main model and proposes up to five future tokens. The main Flash model verifies proposals and commits only accepted prefixes, potentially accelerating generation on predictable continuations like code, though low-yield prompts may see no speedup. The software is currently in beta quality with rapid changes, though QA runs precede each release. Comprehensive testing infrastructure includes remote Metal, CUDA, and ROCm machines, with contributor guidelines documented in CONTRIBUTING.md and the complete release test matrix in QA_BEFORE_RELEASES.md.
