LLM Foundry is a comprehensive training and deployment codebase developed by MosaicML for Databricks foundation models, written in Python and distributed via PyPI. The repository provides end-to-end infrastructure for training, finetuning, evaluating, and deploying large language models using the Composer framework and the MosaicML platform. The codebase is organized into distinct functional modules including source code for models and datasets in the llmfoundry directory, training scripts supporting HuggingFace and MPT models ranging from 125M to 70B parameters, data preparation utilities for converting text to StreamingDataset format, inference capabilities with model conversion to HuggingFace and ONNX formats, evaluation tools for academic and custom in-context-learning tasks, and benchmarking utilities for profiling both training throughput and inference latency.
The repository is closely associated with two major model families. DBRX is a state-of-the-art open source LLM using Mixture-of-Experts architecture with 132B total parameters and 36B active parameters, available in base and instruct variants with 32768 token context length. The MPT series includes multiple GPT-style models incorporating Flash Attention, ALiBi for context extrapolation, and stability improvements, ranging from 7B to 30B parameters with context lengths from 2048 to 65536 tokens. Several MPT variants support commercial use while others are restricted to research applications.
The repository's contributor network overlaps with major projects including microsoft/vscode, microsoft/typescript, and rust-lang/rust, suggesting cross-pollination with broader software development communities.
The codebase is classified across multiple domains including evaluation, AI development, research infrastructure, training, benchmarking, model management, workflow orchestration, scalability, data processing, and pipeline automation. Hardware support is actively tested on NVIDIA A100 and H100 GPUs with PyTorch 2.4, though the codebase may function on other devices including consumer NVIDIA and AMD cards. MosaicML provides prebuilt Docker images pinned to specific PyTorch and CUDA versions, with dedicated llm-foundry images built on every commit to the main branch and available in variants for standard and AWS deployments. The repository includes comprehensive documentation through a TUTORIAL.md file covering deeper workflows and frequently asked questions, alongside scripts for interactive model generation and community contributions including integrations with tools like Langchain, GPT4All, and optimized inference implementations for consumer hardware.
