NVIDIA Cosmos 3 ships open-weights text-to-image for physical AI

Most text-to-image models optimize for pretty pixels. Robotics and AV teams need pixels that obey physics: plausible lighting, object contact, motion cues, and spatial relationships that do not fall apart when you feed the frame into a simulator.

NVIDIA launched Cosmos 3 as an open frontier foundation model for physical AI. The headline release is a full omnimodal stack (text, image, video, audio, action). For builders who only need still frames today, the specialized checkpoint Cosmos3-Super-Text2Image is the one to watch: 64B parameters, open weights under OpenMDW 1.1, and integrations that already ship in Hugging Face Diffusers and vLLM-Omni.

What makes Cosmos 3 different from a generic T2I model

Cosmos 3 is not a standalone image API wearing a robot sticker. NVIDIA describes a mixture-of-transformers design: a reasoning transformer understands object interactions, motion, and spatial-temporal structure; a separate generation transformer produces the pixels (or video frames, or actions) from that understanding.

That split matters when you are generating training data for rare edge cases. A warehouse robot that has never seen a specific box stack at dusk should still get images where shadows, occlusion, and support surfaces look consistent enough to fine-tune a policy or stress-test perception.

NVIDIA's technical blog positions Cosmos 3 Super as the quality leader on open text-to-image and image-to-video leaderboards (per Artificial Analysis rankings cited in their launch materials). Nano is the path if you need something closer to on-robot or edge deployment.

How to run Cosmos3-Super-Text2Image today

You have three realistic entry points, depending on whether you want notebooks, a serving stack, or NVIDIA-hosted APIs.

Hugging Face Diffusers (fastest to try)

Diffusers exposes Cosmos3OmniPipeline, which is the integration NVIDIA tested for release:

from diffusers import Cosmos3OmniPipeline
import torch

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Super-Text2Image",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    enable_safety_checker=True,
)

image = pipe(
    prompt="a forklift navigating a narrow aisle with stacked pallets at golden hour",
    num_inference_steps=28,
    guidance_scale=7.0,
).images[0]
image.save("warehouse_forklift.png")

This is the path I would use to validate prompt style and safety settings before committing GPU hours to a batch job.

vLLM-Omni (OpenAI-compatible serving)

For production-ish serving, NVIDIA documents an 8x H100 profile with CFG parallelism and HSDP. The container tag vllm/vllm-omni:cosmos3 is the tested starting point. Once the server is up, you hit /v1/images/generations like any other image API.

Hardware honesty: this is not a "run on your laptop" model. Plan for a multi-GPU node or a cloud burst when you need Super quality.

build.nvidia.com and NIM microservices

If you want to benchmark before you operate GPUs, NVIDIA exposes try-it flows on build.nvidia.com and packages deployment as NIM microservices. That is useful when your robotics team wants samples this week and your infra team is still sizing a cluster.

Agentic upsampling: prompts are JSON, not vibes

One detail that is easy to miss on the Hugging Face card: Cosmos 3 Super Text2Image expects JSON-upsampled prompts for best quality. The model repo ships an agentic_upsampling/ package and an AGENTIC_UPSAMPLING.md guide.

In practice that means your pipeline should look like:

1. User or planner emits a short intent ("rainy loading dock, worker in hi-vis vest").
2. An LLM expands it into structured JSON with scene, lighting, camera, and object constraints.
3. Cosmos generates from the enriched prompt.

If you are already building agents for ops or simulation, this is a natural handoff: the same agent that plans a test matrix can also author generation prompts with explicit physical constraints.

Where this fits in a physical AI workflow

Cosmos 3 is deliberately multimodal even when you start with stills:

For applied AI teams, the near-term win is often synthetic image diversity for classifiers and detectors that must not overfit to a single warehouse lighting setup. Longer term, the same weights family moves you toward video and action-conditioned generation without swapping vendors.

What I would test first

Before betting a sprint on Cosmos-generated data:

1. Pick 20 real failure images from your domain (glare, motion blur, odd SKUs).
2. Generate matched synthetic frames with JSON-upsampled prompts that spell out lighting and materials.
3. Measure detector mAP or classifier recall with and without the synthetic mix.
4. Track inference cost on your target GPU shape (Nano vs Super is a real budget fork).

If synthetic frames do not move the metric, the bottleneck is probably labels or policy, not rendering.

Licensing and safety

Weights ship under OpenMDW 1.1. Diffusers integration supports enable_safety_checker=True on the pipeline shown above. Treat that as a baseline, not a compliance program, if you are generating human-facing marketing assets.

Cosmos 3 is the kind of open release that rewards teams already thinking in simulation loops: structured prompts, measurable physics plausibility, and a path from stills to video without changing foundation vendors.

If you are wiring synthetic data or agentic media pipelines into production and want a second pair of eyes on GPU sizing and eval design, book a free discovery call.