LLM Foundry is a comprehensive training and deployment codebase developed by MosaicML for Databricks foundation models, written in Python and distributed via PyPI. The repository provides end-to-end infrastructure for training, finetuning, evaluating, and deploying large language models using the Composer framework and the MosaicML platform. The codebase is organized into distinct functional modules including source code for models and datasets in the llmfoundry directory, training scripts supporting HuggingFace and MPT models ranging from 125M to 70B parameters, data preparation utilities for converting text to StreamingDataset format, inference capabilities with model conversion to HuggingFace and ONNX formats, evaluation tools for academic and custom in-context-learning tasks, and benchmarking utilities for profiling both training throughput and inference latency.
The repository is closely associated with two major model families. DBRX is a state-of-the-art open source LLM using Mixture-of-Experts architecture with 132B total parameters and 36B active parameters, available in base and instruct variants with 32768 token context length. The MPT series includes multiple GPT-style models incorporating Flash Attention, ALiBi for context extrapolation, and stability improvements, ranging from 7B to 30B parameters with context lengths from 2048 to 65536 tokens. Several MPT variants support commercial use while others are restricted to research applications.
According to GitGenius activity tracking across 71 issues and pull requests, the repository shows a median response latency of 0.0 hours with a mean of 1293.1 hours, indicating variable response times across different issues. Bug reports represent the most active issue category with 31 tracked items, followed by questions with 15 items and enhancement requests with 10 items. The primary contributor tracked by GitGenius is dakinggg with 87 events, followed by LalchandPandia with 7 events and eldarkurtic with 5 events. The repository's contributor network overlaps with major projects including microsoft/vscode, microsoft/typescript, and rust-lang/rust, suggesting cross-pollination with broader software development communities.
The codebase is classified across multiple domains including evaluation, AI development, research infrastructure, training, benchmarking, model management, workflow orchestration, scalability, data processing, and pipeline automation. Hardware support is actively tested on NVIDIA A100 and H100 GPUs with PyTorch 2.4, though the codebase may function on other devices including consumer NVIDIA and AMD cards. MosaicML provides prebuilt Docker images pinned to specific PyTorch and CUDA versions, with dedicated llm-foundry images built on every commit to the main branch and available in variants for standard and AWS deployments. The repository includes comprehensive documentation through a TUTORIAL.md file covering deeper workflows and frequently asked questions, alongside scripts for interactive model generation and community contributions including integrations with tools like Langchain, GPT4All, and optimized inference implementations for consumer hardware.
