CoTracker is a transformer-based model for tracking any point in a video. It addresses the problem of following arbitrary pixels across video frames by combining the benefits of optical flow with point tracking capabilities. The model uses a transformer architecture to process video sequences and can track individual pixels, dense sets of pixels, or manually selected points across frames, whether sampled from a single frame or specified by the user.
Developers should adopt this tool if they need reliable point tracking without the constraints of traditional methods that require pre-defined objects or sparse feature matching. The project suits applications ranging from video analysis and motion understanding to 3D reconstruction pipelines. The tool distinguishes itself through its ability to handle quasi-dense tracking of many points simultaneously and its lightweight architecture that achieves strong performance despite being trained on significantly less data than competing approaches.
The project maintains active development with multiple model versions available. Recent work has focused on improving efficiency and performance, with the latest iteration introducing a state-of-the-art lightweight architecture and releasing associated training datasets. The team has also extended the core tracking capability into downstream applications like structure-from-motion frameworks. Interactive demos are provided through both Colab notebooks and Hugging Face spaces, making the tool accessible for experimentation without local setup. The codebase includes implementations of baseline models and the pseudo-labeling pipeline used for training, supporting reproducibility and further research.
