ik_llama.cpp is a specialized fork of llama.cpp that prioritizes CPU and hybrid GPU/CPU inference performance for large language models. The repository is written in C++ and focuses on delivering state-of-the-art quantization techniques and optimized execution across different hardware backends.
A defining feature of ik_llama.cpp is its extensive collection of quantization types beyond the standard llama.cpp offerings. The repository implements Trellis quants including IQ1_KT, IQ2_KT, IQ3_KT, and IQ4_KT with support across CUDA, Metal, Neon, and CPU backends. The IQK quantization family includes variants like IQ2_K, IQ3_K, IQ4_K, IQ5_K, and IQ6_K with multiple implementations for different hardware. More recent additions include MXFP4 quantization as used in gpt-oss models and Hadamard transforms for both K-cache and V-cache optimization. The repository also implements low perplexity Q4_0 KV cache variants and provides options for re-quantized output tensors through the MTP feature.
The fork introduces specialized optimizations for particular model architectures. It provides first-class Bitnet support, improved DeepSeek performance through Multi-head Latent Attention (MLA) and FlashMLA implementations, and fused mixture-of-experts operations. Tensor overrides enable flexible hybrid GPU/CPU inference strategies, while row-interleaved quant packing improves memory efficiency. The repository explicitly documents important caveats regarding the row-interleaved tensor repacking option, warning that it can degrade performance when certain quantization types lack CUDA implementations.
The repository maintains clear documentation about supported compute backends, explicitly stating that only CPU backends with AVX2 or better and ARM_NEON or better, plus CUDA with Turing or newer GPUs, receive active development and support.
The project provides straightforward quickstart instructions for both CPU and GPU builds, with special guidance for AVX-512-capable processors to activate IQK quantized GEMM kernels. Users can download GGUF model files and launch a server with simple commands, accessing a web interface at localhost:8080. The repository includes comprehensive build documentation, parameter guides, and Docker/Podman container instructions for deployment flexibility.
