The StableLM repository contains Stability AI's development of the StableLM series of language models, with continuous updates and new model checkpoints. The repository is primarily composed of Jupyter Notebooks and serves as a central hub for releasing and documenting language models of varying sizes and capabilities.
The repository's most recent major release is StableLM-3B-4E1T, a 3 billion parameter model released in September 2023 under the CC BY-SA-4.0 license. This model was trained on 4 trillion tokens across 4 epochs using a multi-epoch training regime to study the impact of repeated tokens on downstream performance. The model incorporates a decoder-only transformer architecture similar to LLaMA with specific modifications including rotary position embeddings applied to the first 25 percent of head embedding dimensions, LayerNorm normalization with learned bias terms, and the GPT-NeoX tokenizer. The training data comprises a filtered mixture of open-source datasets including Falcon RefinedWeb, RedPajama-Data, The Pile, and StarCoder. According to the repository's evaluation results, StableLM-3B-4E1T achieves state-of-the-art performance at the 3B parameter scale for open-source models as of September 2023, with an average score of 66.93 across multiple benchmarks and competitive performance against many 7B parameter models.
Earlier releases documented in the repository include StableLM-Alpha v2 models with 3B and 7B parameters released in August 2023, which incorporated architectural improvements such as SwiGLU and higher-quality data sources. These models use a context length of 4096 tokens and were trained on 1.1 trillion tokens using a multi-stage approach to context length extension. The repository also documents StableVicuna-13B from April 2023, an RLHF fine-tune of Vicuna-13B, and the initial StableLM-Alpha models with 3B and 7B parameters released in April 2023, which included a tuned variant available for interactive chat on Hugging Face Spaces.
According to GitGenius activity tracking, the repository shows relatively low issue and pull request response latency with a median of 0.0 hours and mean of 50.1 hours across tracked items. The most active contributors tracked include ucalyptus2 with 2 events, afyffe2015 with 1 event, and maxim-saplin with 1 event. The repository maintains overlapping contributors with pytorch-lightning, PyTorch, and the Hugging Face diffusers repository, indicating integration within the broader machine learning ecosystem.
The repository is classified across multiple domains including large-scale models, model training, ethical AI, language modeling, NLP tools, model stability, deployment, neural networks, and research tools. The codebase includes detailed YAML configuration files for hyperparameter settings, such as stablelm-3b-4e1t.yml and stablelm-base-alpha-3b-v2-4k-extension.yml, and maintains an evals directory containing full evaluation results in JSON format. The repository emphasizes that base models are recommended for fine-tuning on downstream tasks given the large amount of web data used in pre-training.
