NVIDIA Cosmos is an open platform designed to enable developers to build Physical AI applications for robots, autonomous vehicles, smart infrastructure, and related domains. The repository serves as the primary hub for the Cosmos world models ecosystem, providing access to models, datasets, and tools necessary for developing AI systems that understand and generate physical world dynamics.
The repository is written primarily in Jupyter Notebook format and is classified across multiple infrastructure and machine learning domains including AI infrastructure, distributed computing, GPU orchestration, resource scheduling, and scalable deployment. This classification reflects the platform's focus on supporting large-scale model deployment and training workflows across data center operations and cloud automation environments.
Cosmos 3 represents the newest model family within the platform. It is built on a unified Mixture-of-Transformers architecture that combines an autoregressive transformer for reasoning with a diffusion transformer for multimodal generation. The architecture employs a unified 3D multi-dimensional rotary position embedding representation to encode spatial and temporal structure across modalities. Cosmos 3 exposes two distinct runtime surfaces: a Reasoner that accepts text and vision inputs to produce text outputs for world understanding, physical reasoning, task planning, and autonomous system decision making, and a Generator that accepts text, vision, sound, and action inputs to produce vision, sound, and action outputs for world generation, simulation, and synthetic data generation.
The model family includes five specific variants. Cosmos3-Nano is a 16-billion parameter compact omnimodal world model, while Cosmos3-Super is a 64-billion parameter frontier-scale variant. Specialized 64-billion parameter models include Cosmos3-Super-Text2Image for high-fidelity image generation and Cosmos3-Super-Image2Video for temporally coherent video generation. A robotics-specific variant, Cosmos3-Nano-Policy-DROID, provides a 16-billion parameter vision-language robot policy for manipulation and control tasks.
The platform supports flexible generation configurations including multiple resolution tiers from 256p to 720p, aspect ratios ranging from 16:9 to 9:16, frame rates from 10 to 30 FPS, and frame counts from 5 to 300 frames. Input types encompass text, images in JPG, PNG, JPEG, or WEBP formats, MP4 videos, and JSON action arrays. Action conditioning supports multiple embodiment types including camera motion, autonomous vehicles, egocentric motion, single-arm robots, dual-arm robots, and humanoid robots with varying dimensional specifications for each.
The platform provides multiple integration paths for both research and production use cases, including Diffusers and Transformers for Python-first development, vLLM-Omni and vLLM for OpenAI-compatible serving, and support for NVIDIA NIM deployment. The repository includes comprehensive troubleshooting documentation, inference benchmarks, fine-tuning capabilities, and detailed examples demonstrating generator workflows for text-to-image, text-to-video, image-to-video, video-to-video, forward dynamics, and action policy generation, as well as reasoner workflows for multimodal understanding tasks.
