NVIDIA Open-Sources Cosmos 3 Models for Streaming World Simulation

NVIDIA has released four open models from the Cosmos 3 family. They generate what a car's or robot's sensors are supposed to see: synchronized video from several cameras, LiDAR data, and the image for the control neural network. Some of the models work in streaming mode: they output frames as the simulation proceeds and react to the agent's actions.

All four checkpoints were fine-tuned from Cosmos 3 Nano, each with 16 billion trainable parameters. They are distributed under the OpenMDW1.1 license, which on its terms permits commercial and non-commercial use. Along with the weights, NVIDIA published a technical report, demos, a recording of the Cosmos Labs stream, and post-training recipes. The announcement appeared on NVIDIA's blog on Hugging Face on November 5, 2026.

Four models for four tasks

• Cosmos3-Nano-Transfer-Auto generates driving data in offline mode. The model outputs synchronized video from up to 11 cameras, including fisheye lenses, and can generate a color image together with LiDAR data. The viewpoints of all cameras are synchronized with one another.
• Cosmos3-Nano-Sim-Auto is the streaming version of Transfer-Auto. It was distilled so that each next chunk relies only on those already generated. It generates camera and LiDAR in a few denoising steps, which makes it suitable for interactive simulation: the car acts, the scene responds.
• Cosmos3-Nano-Sim-Transfer takes spatial control signals from the simulator (depth, segmentation, edges, a blurred scene layout) and, together with a text prompt, turns them into photorealistic color video. That video is then fed to the control model of the robot or car.
• Cosmos3-Nano-Sim-Bimanual interactively simulates the bimanual robots YAM and AgiBot. They can be controlled by a trained policy or by a live operator with joysticks. In the mixed-action example, 59 control values are fed in, and during training they are padded to 64.

For a technical artist assembling scenes for simulation, Sim-Transfer is the most interesting. Physics, contacts, and scene state are still computed by the simulator, while the look of the frame is set by the text prompt. Presumably, with this approach part of the visual polish of the environment with materials and lighting can be offloaded to the model.

How the models are built and what you need to run them

Cosmos 3 is based on the Mixture-of-Transformers architecture: text processing and image generation are handled by separate networks linked by an attention mechanism. After the starting image, each step in the model's internal representation corresponds to four RGB frames. Training of the streaming variants proceeds in four stages: first the model is adapted to bidirectional generation, then it is taught to continue the sequence, then it is distilled, and finally it is post-trained on its own generations. At the distillation stage, the student model generates each video chunk in four denoising steps. The recipes start from Cosmos3-Nano weights or compatible base weights and cover all four stages, so the models can be adapted to your own camera set or your own robot.

To run them you need Linux, an NVIDIA graphics card, Docker, and the NVIDIA Container Toolkit. Inference instructions and ready-made notebooks are in the cosmos-framework repository: for Transfer-Auto, Sim-Auto, and Sim-Transfer, while for Sim-Bimanual the data format for post-training is documented. NVIDIA does not state minimum video memory requirements. The published configuration for Sim-Bimanual was tested on a single GB200 server accelerator, and the company explicitly notes that this is not a minimum specification. How much memory is enough on a workstation is not stated in the materials.

NVIDIA itself frames the release as a common starting point for researchers and developers:

"Ahead lies the work of measuring where the models reliably support policy training and evaluation, of understanding their limitations, and of improving them where they fall short."

That is how the announcement states the Cosmos team's next steps. There are no figures on the quality and speed of the streaming variants in the announcement; details about the family's architecture are laid out in the Cosmos 3 technical report.

Where this fits in

Cosmos 3 came out in May 2026, and in July NVIDIA added a lightweight 4-billion-parameter version, Cosmos3-Edge, to run directly on the device: Jetson AGX Orin, Thor, and RTX Pro 6000. In NVIDIA's pipeline for autonomous vehicles, real drives are first reconstructed in 3D via Omniverse NuRec based on Gaussian libraries. Then the Cosmos and Cosmos-Dreams models generate scene variations with different weather, lighting, and rare situations, and validation runs in a closed loop in the open simulator AlpaSim. According to NVIDIA, Oxa, Plus, and Uber use the Cosmos models for synthesizing driving data.

Judging by the makeup of the release, NVIDIA is shifting generative models from the role of an offline dataset generator closer to the role of the simulator itself: Sim-Auto and Sim-Bimanual respond to actions interactively, and Sim-Transfer can take on the photorealistic image. For those building scenes to train robots and autopilots, this potentially lowers the demands on visual polish of the environment, but adds a dependency on NVIDIA hardware and the question of how physically accurate the generated frame is. The company itself calls evaluating that future work. For artists outside robotics and the automotive industry there is probably little practical benefit: the models are tailored to sensors, multi-camera rigs, and robot control signals.

Why is NVIDIA opening up Cosmos 3 for commercial use: promoting its own hardware, the race for a simulation standard, a shortage of real-world data?