FastVLM is a vision language model that prioritizes efficient vision encoding for processing high-resolution images with minimal latency.
The project addresses the computational bottleneck of vision encoding in vision language models by introducing FastViTHD, a hybrid vision encoder that reduces the number of output tokens and significantly decreases encoding time. Rather than processing full-resolution images with traditional encoders, the approach uses a specialized architecture designed to maintain accuracy while substantially lowering the computational cost of the vision component.
Developers should consider FastVLM if they need vision language capabilities on resource-constrained environments, particularly mobile devices or edge hardware. The tool suits projects requiring fast inference on high-resolution images where Time-to-First-Token latency is critical. The repository provides multiple model variants ranging from 0.5B to 7B parameters, allowing selection based on accuracy and speed requirements. A demo iOS application is included to show real-world performance on mobile devices. The project builds on the LLaVA codebase, so those familiar with that framework will find the training and finetuning process straightforward.
The project maintains pretrained checkpoints for multiple model sizes and training stages, with detailed evaluation results referenced in the accompanying paper. Inference code is provided for standard PyTorch execution, and separate tooling in the model_export subfolder enables conversion to formats suitable for Apple Silicon devices. The repository includes practical usage examples and setup instructions to facilitate adoption.
