mergekit is a toolkit for merging pretrained large language models.
The tool addresses the challenge of combining multiple specialized language models into a single model without the computational overhead of ensembling or the need for retraining. It uses an out-of-core approach that enables merges to run on CPU or with as little as 8 GB of VRAM, making it practical for resource-constrained environments. The toolkit supports multiple merging algorithms and can perform operations like piecewise assembly of models from individual layers, extraction of LoRA adapters from merged models, and merging of Mixture of Experts architectures.
Developers should consider mergekit when they need to combine capabilities from different models, transfer knowledge between models without access to training data, or explore trade-offs between different model behaviors while maintaining single-model inference costs. The tool supports a range of model architectures including Llama, Mistral, GPT-NeoX, and StableLM. It offers both simple merging workflows and complex multi-stage merging for elaborate combinations, as well as raw PyTorch model merging and tokenizer transplantation capabilities for advanced use cases.
The project maintains active community engagement through GitHub Issues and Discord discussions, and explicitly welcomes contributions including new merge methods and features. Development activity shows ongoing expansion of supported merge algorithms and refinement of the toolkit's capabilities.
