The llm-course repository is a comprehensive educational resource designed to help learners enter the field of Large Language Models through structured roadmaps and practical Google Colab notebooks. The course is divided into three distinct parts: LLM Fundamentals covering mathematics, Python, and neural networks; The LLM Scientist focusing on building state-of-the-art LLMs using latest techniques; and The LLM Engineer concentrating on creating and deploying LLM-based applications.
The repository provides extensive hands-on learning materials through numerous Colab notebooks organized into several categories. The Tools section includes specialized notebooks like LLM AutoEval for automatic LLM evaluation using RunPod, LazyMergekit for model merging, LazyAxolotl for cloud-based fine-tuning, AutoQuant for quantization in multiple formats including GGUF, GPTQ, EXL2, AWQ, and HQQ, and ZeroSpace for creating Gradio chat interfaces. Additional tools cover model family tree visualization and automated abliteration and deduplication of datasets.
The Fine-tuning section offers multiple notebooks demonstrating different approaches and models. These include fine-tuning Llama 3.1 with Unsloth for efficient supervised fine-tuning, fine-tuning Llama 3 with ORPO for cheaper single-stage fine-tuning, fine-tuning Mistral-7b with DPO to boost performance, fine-tuning Mistral-7b with QLoRA in free-tier Google Colab, fine-tuning CodeLlama using Axolotl, and fine-tuning Llama 2 with QLoRA. The Quantization section covers introduction to weight quantization using 8-bit methods, 4-bit quantization using GPTQ for consumer hardware, quantization with GGUF and llama.cpp, and ExLlamaV2 for running the fastest quantized models.
The repository also includes notebooks on merging LLMs with MergeKit, creating Mixture of Experts with MergeKit, uncensoring LLMs through abliteration, improving ChatGPT with knowledge graphs, and understanding decoding strategies in language models. Each notebook is accompanied by detailed articles on the author's blog explaining the concepts and implementations.
The author has also published the LLM Engineer's Handbook, a hands-on book covering end-to-end LLM application development from design to deployment, though the course itself remains free. The repository is classified across multiple domains including LLMs, AI courses, deep learning, NLP, transformers, model training, model deployment, generative AI, and prompt engineering, reflecting its comprehensive coverage of the LLM ecosystem.
