The ml-ferret repository contains the implementation of Ferret, a multimodal large language model designed to accept any form of referring expressions and ground objects in images at varying levels of granularity. The project accompanies a research paper and represents work accepted to ICLR 2024 as a Spotlight presentation, with a subsequent version, Ferret-v2, accepted to COLM 2024. The repository also includes Ferret-UI, released in October 2024, which extends the model's capabilities to UI-centric tasks involving referring, grounding, and reasoning.

The core technical contribution of Ferret involves a hybrid region representation approach combined with a spatial-aware visual sampler that enables fine-grained and open-vocabulary referring and grounding capabilities within a multimodal language model framework. The model builds upon existing architectures, specifically using Vicuna as the base language model and incorporating LLaVA's pre-trained projector weights. Two model sizes are provided: a 7-billion parameter version and a 13-billion parameter version, both trained on 8 A100 GPUs with 80GB memory.

The repository includes the GRIT dataset, a large-scale hierarchical dataset containing approximately 1.1 million instruction-tuning examples designed to support robust ground-and-refer tasks. Additionally, Ferret-Bench is provided as a multimodal evaluation benchmark that jointly assesses referring and grounding capabilities alongside semantic understanding, knowledge, and reasoning tasks. The training hyperparameters follow a similar configuration to LLaVA, using a global batch size of 128, a learning rate of 2e-5, and training for 3 epochs with a maximum sequence length of 2048 tokens.

The codebase is written in Python and is classified across multiple domains including natural language processing, machine learning, query understanding, document classification, text processing, information retrieval, and semantic analysis. The repository is structured to support training, evaluation, and interactive demonstration of the model through a Gradio-based web interface. Model checkpoints are distributed as weight differentials relative to the base Vicuna model, allowing users to reconstruct the full model by applying the provided offsets to the Vicuna weights.

The project is licensed for research use only under the CC BY NC 4.0 license, with restrictions tied to the underlying licenses of LLaMA, Vicuna, and GPT-4. The repository includes comprehensive documentation covering installation procedures, training scripts for both model sizes, evaluation protocols detailed in a separate document, and instructions for running an interactive demo locally. The work represents a collaborative effort involving researchers from multiple institutions and demonstrates active development with regular releases and updates throughout 2023 and 2024.