Ferret is a multimodal large language model that accepts any form of referring input and grounds anything in response at any granularity.
The project addresses the challenge of enabling vision-language models to understand spatial references and perform fine-grained object grounding. Ferret combines a hybrid region representation approach with a spatial-aware visual sampler to support both referring and grounding tasks. The model accepts diverse input formats for specifying regions of interest and can output grounded responses that identify and localize objects in images. The approach is grounded in the GRIT dataset, a large-scale instruction tuning collection designed for hierarchical and robust ground-and-refer tasks.
Ferret suits research projects requiring precise spatial understanding and object localization in multimodal contexts. The tool is particularly relevant for applications involving user interface understanding, detailed image analysis, and tasks that demand fine-grained visual reasoning. The project includes Ferret-Bench, a multimodal evaluation benchmark that jointly assesses referring, grounding, semantic understanding, knowledge, and reasoning capabilities. Training requires significant computational resources, with the baseline implementation using eight A100 GPUs with 80GB memory each, though the configuration can be adjusted by modifying batch size and gradient accumulation settings while maintaining the global batch size.
The project maintains active development with multiple model variants released, including Ferret-UI focused on user interface tasks and Ferret-v2 with enhanced capabilities. The codebase includes comprehensive training infrastructure with documented hyperparameters for both 7B and 13B model sizes, evaluation frameworks, and demonstration capabilities. The project is restricted to research and non-commercial use under its licensing terms.
