OctoPack is a comprehensive instruction tuning framework for code large language models, centered around the paper published at arxiv.org/abs/2308.07124. The repository provides all components necessary for instruction-tuning code LLMs, including datasets, trained models, and evaluation benchmarks. The project is primarily implemented in Jupyter Notebooks and serves as a resource for researchers and practitioners working on improving code generation capabilities in large language models.
The core of OctoPack consists of two main datasets. CommitPack is a 4TB dataset extracted from GitHub commits spanning 350 programming languages, created through a multi-step process involving BigQuery SQL queries to extract commit metadata from GitHub action data, followed by scraping actual file changes from GitHub repositories. CommitPackFT is a filtered version of CommitPack designed specifically for instruction tuning, containing high-quality commit messages that resemble natural language instructions. The creation of CommitPackFT involves running filtering scripts to identify commits with instruction-like messages suitable for fine-tuning language models.
The repository includes two instruction-tuned models built on top of existing code LLMs. OctoCoder is a 16-billion parameter model based on StarCoder that has been instruction-tuned on CommitPackFT combined with OASST data. OctoGeeX is a smaller 6-billion parameter variant based on CodeGeeX2, also instruction-tuned on the same datasets. Both models represent different points on the model size and capability spectrum for code understanding and generation tasks.
For evaluation purposes, OctoPack introduces HumanEvalPack, an extension of OpenAI's HumanEval benchmark that covers three distinct scenarios across six programming languages. The evaluation framework supports multiple task types including HumanEvalFix for fixing code based on test cases, HumanEvalExplain for generating code explanations and then synthesizing code from those explanations, and HumanEvalSynthesize for direct code generation. The evaluation harness is integrated with the bigcode-evaluation-harness repository and supports running tasks with various prompting formats specific to different model architectures.
The repository provides detailed documentation on recreating all components from scratch. The CommitPack creation process involves seven distinct steps including BigQuery operations, data export, GitHub scraping with multi-threading and multi-processing, optional sharding, and language-specific filtering. Scripts are provided for each step, though users are expected to adapt paths and configurations to their specific environments. The evaluation section includes specific instructions for running benchmarks with different models and configurations, including notes on batch sizes, Python versions, and handling of randomness in evaluation results.
GitGenius activity data shows median issue and pull request response latency of 2.4 hours with a mean of 3.5 hours across tracked items, indicating active maintenance. Niklas Muennighoff is identified as the most active contributor tracked by GitGenius. The repository maintains connections with related bigcode-project repositories including bigcode-evaluation-harness, starcoder, and integrations with huggingface/datasets, reflecting its position within a broader ecosystem of code-focused machine learning research and development tools.
