nanoVLM is a framework for training and finetuning small-sized vision-language models with a lightweight pure PyTorch implementation.
The project addresses the need for an accessible, understandable codebase for vision-language model development. Rather than providing a production-grade system, it prioritizes simplicity and readability by implementing core components—vision backbone, language decoder, modality projection, and training loop—in approximately 750 lines of code. This approach lets developers quickly understand how VLMs work and experiment with different architectural choices and training configurations without navigating complex abstractions.
The tool suits researchers and practitioners who want to train or finetune small VLMs on custom datasets, particularly those with access to high-end hardware like H100 GPUs. It works well for educational purposes and for exploring the capabilities of compact models, as demonstrated by achieving reasonable performance on standard benchmarks with modest training time. The project explicitly positions itself as an educational effort rather than a state-of-the-art model, drawing inspiration from similar simplified implementations in the language model space. It provides a starting point for those who want to tinker with VLM architectures and training approaches without the overhead of larger frameworks.
The codebase has undergone significant refactoring to support image splitting and multi-node training, with a newer 450M model variant reflecting these changes while maintaining an older 222M model for backward compatibility. The project includes a Colab notebook for immediate experimentation and documentation through a dedicated tutorial. Breaking changes have been introduced to enable smarter packing strategies, indicating active development focused on improving training efficiency and model capabilities.
