LLaMA Factory is a unified framework for efficient fine-tuning of over 100 large language models and vision-language models, recognized with an ACL 2024 publication. The repository provides both command-line and graphical interfaces for fine-tuning, making advanced model customization accessible without requiring code.
The framework supports an extensive range of models including LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, and Phi variants. It implements multiple training methodologies including continuous pre-training, multimodal supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, and other advanced techniques. The repository demonstrates rapid adoption of cutting-edge models, with day-zero support for models like Qwen3, Qwen2.5-VL, Gemma 3, and GLM-4.1V, and day-one support for Llama 4 and other recent releases.
LLaMA Factory offers diverse quantization and efficiency options spanning 16-bit full-tuning, freeze-tuning, LoRA, and 2 through 8-bit QLoRA implementations via AQLM, AWQ, GPTQ, LLM.int8, HQQ, and EETQ. The framework integrates advanced optimization algorithms including GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ, and PiSSA. Performance enhancements are achieved through FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune, and rsLoRA.
The repository supports diverse task types including multi-turn dialogue, tool usage, image understanding, visual grounding, video recognition, and audio understanding. Experiment monitoring is facilitated through LlamaBoard, TensorBoard, Wandb, MLflow, and SwanLab integration. For inference, the framework provides OpenAI-style API compatibility, Gradio UI, and CLI access with vLLM or SGLang workers for accelerated deployment.
The repository has established adoption among major technology companies including Amazon, NVIDIA, and Aliyun, with documented use cases in production environments.
The project provides comprehensive documentation at llamafactory.readthedocs.io, maintains an official blog at blog.llamafactory.net, and offers free cloud training options through Google Colab and Aliyun PAI-DSW. Docker support, ModelScope Hub integration, and Weights and Biases logging capabilities are included. The framework also supports specialized hardware backends including AMD GPU documentation and ASCEND NPU support, demonstrating broad hardware compatibility across different computing environments.
