NVIDIA's TwoTower model writes text in parallel and keeps 98.7% of AR quality

Most language models generate text one token at a time. NVIDIA just shipped an open-weight alternative that fills whole blocks of tokens in parallel and reports 2.42x wall-clock throughput while retaining 98.7% of the autoregressive baseline's benchmark quality.

The model is Nemotron-Labs-TwoTower-30B-A3B-Base-BF16. The trick is architectural, not magical: take one pretrained 30B Nemotron-3-Nano backbone, split it into two towers, and let each tower do one job well.

The two-tower design

Traditional diffusion language models use a single network for two roles: representing clean context tokens and iteratively denoising corrupted ones. That forces one model to compromise on both tasks.

TwoTower decouples them:

Tower	Role	Training
Context tower (AR)	Causally processes clean prompt and committed tokens	Frozen
Denoiser tower (diffusion)	Refines noisy token blocks via bidirectional attention + cross-attention to context	Trained on ~2.1T tokens

Both towers start as copies of the same Nemotron-3-Nano-30B-A3B checkpoint (a hybrid Mamba-2, attention, and MoE architecture with ~3B active parameters per token). Only the denoiser is trained. The context tower preserves the backbone's autoregressive representations.

Nemotron-Labs-TwoTower on Hugging Face

How block-wise generation works

Generation is block-wise autoregressive at the macro level, parallel inside each block:

The context tower encodes the prompt once.
The denoiser initializes a block of 16 masked token slots.
For up to 16 denoising steps, it predicts all masked positions in parallel.
Positions above a confidence threshold (gamma = 0.8) get committed. The rest stay masked.
The context tower processes the committed block and updates KV and Mamba caches.
Repeat until max_new_tokens or EOS.

Default settings: block size 16, 16 denoising steps per block, confidence threshold 0.8. Hardware for full diffusion mode: 2x H100 or A100 GPUs at roughly 59GB per GPU in BF16.

TwoTower architecture with frozen context tower and parallel diffusion denoiser generating token blocks

Three inference modes in one checkpoint

The same weights support three decoding paths:

Mode	API call	Tokens per step	GPUs
Mask diffusion	`generate_mask_diffusion()`	Up to block_size (16)	2
Mock-AR	`generate_mock_ar()`	1	2
AR-only	`generate_ar()`	1	1

Mask diffusion is the default for throughput. AR-only runs on a single 80GB GPU if you do not need the speed boost.

outputs = model.generate_mask_diffusion(
    inputs["input_ids"],
    max_new_tokens=128,
    block_size=16,
    steps_per_block=16,
    mask_token_id=3,
    temperature=0.1,
    confidence_threshold=0.8,
    eos_token_id=tokenizer.eos_token_id,
)

Requires trust_remote_code=True when loading from Hugging Face.

Why this matters for inference economics

Throughput is the bottleneck most teams feel before raw quality. If you can get 2.4x more tokens per second at 98.7% quality, your serving cost per request drops without retraining from scratch.

The training budget is modest relative to pretraining: ~2.1T tokens for the denoiser versus 25T for the backbone. That suggests parallel decoding might be achievable as a post-training conversion on other strong AR models, not just Nemotron.

Nemotron-TwoTower paper on arXiv

Throughput comparison chart showing 2.42x speedup versus standard autoregressive decoding

What I would test before shipping it

Benchmark retention at 98.7% is an aggregate number. Before routing production traffic:

Run your own eval set, especially for code and tool-calling if you build agents.
Measure latency variance across block boundaries. Parallel blocks can introduce different failure modes than token-by-token decoding.
Check license fit. NVIDIA Open Model License allows commercial use, but read the terms for your deployment context.

If you are optimizing inference cost for an agent or RAG stack and want help benchmarking parallel decoding against your current AR setup, book a free discovery call.