Marigold is a computer vision method that adapts pretrained diffusion-based image generators for dense image analysis tasks including monocular depth estimation, surface normal prediction, and intrinsic decomposition.
The project addresses the challenge of extracting dense per-pixel predictions from images without task-specific training data. It works by fine-tuning existing latent diffusion models like Stable Diffusion to perform these analysis tasks. Rather than training from scratch, the approach leverages the rich visual knowledge already present in generative models, requiring only minimal architectural modifications and training on small synthetic datasets over a few days on a single GPU.
It is particularly valuable for applications requiring high-resolution depth maps or other image characteristics from arbitrary photographs. The method demonstrates state-of-the-art generalization to unseen data without task-specific fine-tuning on target domains.
The project maintains active development with implementations of multiple dense prediction modalities beyond the original depth estimation work. The codebase is well-documented with references to peer-reviewed publications and includes a project homepage with additional resources. Development appears focused on extending the core adaptation protocol to new analysis tasks while maintaining the efficiency characteristics that allow training on modest computational resources.
