The mistral-finetune repository is a Python-based codebase designed to enable memory-efficient and performant fine-tuning of Mistral's language models. The repository implements LoRA (Low-Rank Adaptation), a training paradigm where most model weights remain frozen while only 1-2% of additional weights in the form of low-rank matrix perturbations are trained. The codebase is optimized for multi-GPU single-node training setups, though smaller models like the 7B variant can run on a single GPU. For maximum efficiency, the developers recommend using A100 or H100 GPUs.
According to the README, the repository is now archived and no longer actively maintained, with a note that Mistral may launch a new library or major update in the future if community needs warrant it or if they believe they can bring additional value to the fine-tuning ecosystem. Despite this archived status, the repository provides a simple and guided entry point for fine-tuning Mistral models, though it is intentionally opinionated, particularly around data formatting requirements, and does not aim to be exhaustive across multiple model architectures or hardware types.
The codebase supports fine-tuning across multiple Mistral model variants, including the 7B Base V3, 7B Instruct v3, Mixtral 8x7B models, Mixtral 8x22B variants, Mistral Nemo 12B, and Mistral Large 2 123B models. Recent updates documented in the repository include compatibility with Mistral Large v2 as of August 2024 and Mistral Nemo as of July 2024, with specific guidance on memory requirements and hyperparameter adjustments for each model variant.
The repository enforces strict data formatting requirements, supporting two primary data types: pretrain data stored as plain text in a "text" key within JSONL format files, and instruct data supporting both conversational formats with user-assistant-system roles and function calling formats with tool integration. The codebase includes utility scripts for data validation and reformatting, such as validate_data.py for checking dataset correctness and estimating training time, and reformat_data.py for correcting common data formatting issues.
GitGenius activity tracking shows the repository has a median issue and pull request response latency of 1.3 hours across 45 tracked items, with a mean latency of 2056.4 hours. The most active contributor tracked is juliendenize with 71 events, followed by CorentinWicht and patrickvonplaten with 9 events each. Bug reports represent the most active issue label category with 22 tracked items. The repository shares overlapping contributors with vllm-project/vllm, ollama/ollama, and pytorch/pytorch, indicating connections to the broader ecosystem of language model inference and training projects.
The codebase is classified across 20 GitGenius categories spanning pre-trained models, NLP tasks, dataset integration, neural network optimization, fine-tuning, model adaptation, machine learning, language models, text processing, AI development, transfer learning, and related domains. The repository includes example configuration files and tutorials, such as a Colab notebook for fine-tuning the 7B model, making it accessible for users wanting to adapt Mistral models to specific tasks and domains.
