Databricks' Dolly is an instruction-following large language model built on the Databricks machine learning platform and licensed for commercial use. The model is derived from EleutherAI's Pythia-12b and fine-tuned on approximately 15,000 instruction-response pairs generated by Databricks employees. The training data covers capability domains from the InstructGPT paper, including brainstorming, classification, closed question answering, generation, information extraction, open question answering, and summarization. The dataset, called databricks-dolly-15k, is released under a permissive CC-BY-SA license and is publicly available on Hugging Face alongside the model weights.
The Dolly model family represents Databricks' effort to democratize access to large language models and ensure that organizations and individuals can benefit from AI technology. The 12 billion parameter version, dolly-v2-12b, demonstrates surprisingly high quality instruction-following behavior despite not being designed as a state-of-the-art model. The repository provides comprehensive documentation for both inference and training, with specific guidance for running the model on different GPU hardware configurations including A100, A10, and V100 GPUs. For inference, the model is available directly on Hugging Face and can be used with the transformers library, with instructions provided for optimizing performance on various instance types through techniques like 8-bit weight loading and float16 precision.
The repository includes detailed training instructions for users who want to fine-tune or train the model themselves. Training is facilitated through Databricks notebooks and requires GPU clusters, with the documentation specifying that A100 instances with 8 GPUs are preferred for training the full 12B parameter model. For users with access to smaller GPU configurations, the repository provides alternative configurations and memory optimization strategies, though these are noted as suboptimal compared to A100 training. The training process uses DeepSpeed for distributed training optimization, with separate configuration files tailored to different GPU families.
The documentation explicitly acknowledges significant limitations in the Dolly model's capabilities. The model struggles with syntactically complex prompts, programming problems, mathematical operations, factual accuracy, date and time handling, open-ended question answering, hallucination, list enumeration, stylistic mimicry, and humor generation. Additionally, some capabilities present in the original Pythia model, such as well-formatted letter writing, are absent in the fine-tuned version. The repository also documents dataset limitations, noting that the model reflects biases and content from both The Pile pretraining corpus and the Wikipedia passages included in the instruction tuning dataset. The training data represents the interests and semantic choices of Databricks employees during March and April 2023, a demographic not representative of the global population.
According to GitGenius activity tracking, the repository shows relatively low engagement with a median issue and pull request response latency of approximately 15,527 hours and a mean latency of 15,030 hours. The most active contributors tracked include dolphnkim, huzhengdong, and iwannabewater, each with single recorded events. The repository shares overlapping contributors with major projects including Microsoft's VSCode and TypeScript repositories as well as the Rust language repository, indicating some cross-pollination with broader open-source communities.
