DiffusionGemma hits ~1,500 TPS on one H100: when a diffusion LLM beats autoregressive serving

One H100. One user. About 1,500 output tokens per second.

That number comes from Google's DiffusionGemma technical report (arXiv:2608.00146), not from a synthetic micro-benchmark on empty prompts. The model is an experimental open-weight text diffusion variant of Gemma 4, and it targets a serving regime most API stacks ignore: low concurrency, memory-bound, latency-first.

If you run local agents, voice loops, or copilots where batch size stays near 1, this is the first open model I would actually benchmark against your current AR route.

What DiffusionGemma is

DiffusionGemma is a discrete diffusion language model (dLLM). Instead of sampling one token at a time left to right, it generates text by iteratively denoising fixed 256-token canvases in parallel.

Think of each canvas as a scratch pad filled with random tokens. The model runs a bidirectional decoder pass over all 256 positions, accepts the tokens it is confident about, re-randomizes the rest, and repeats until the block converges. Finished blocks get committed to the KV cache, and the next canvas starts.

Google did not train this from scratch. They fine-tuned the post-trained Gemma 4 26B A4B MoE checkpoint with a two-stage pipeline:

1. Supervised fine-tuning (SFT) to teach bidirectional denoising across 256-token blocks
2. Sampler distillation plus reinforcement learning (SD·RL) to improve quality and compress the average number of denoising steps

Total training used less than 10% of the original AR model's token budget. That is a pragmatic path: reuse Gemma 4's reasoning, multimodal inputs, long context, and thinking mode, then swap the decoding algorithm.

The specs that matter for serving

The headline speed number assumes batch size 1. That is not a footnote. It is the whole point.

Autoregressive serving at low concurrency is memory-bound. You spend more time moving KV cache and expert weights than doing math. DiffusionGemma trades heavier per-step compute (256 tokens per pass) for far fewer total forward passes. On an H100 with 4096 input tokens and 1024 output tokens, Google measures about 13.6 ms per denoising step versus roughly 3.2x the per-step cost of single-token AR decoding, but with ~20 tokens per forward pass instead of 1.

Net result: roughly 1,479 TPS in diffusion mode versus 303 TPS for Gemma 4 AR with multi-token prediction, on the same hardware setup in the paper.

Why low concurrency is the sweet spot

Most cloud LLM economics assume high batch utilization. You pack dozens of requests, amortize memory transfers, and accept per-user latency.

Local and edge stacks often look different:

• A coding agent waiting on the next chunk
• A voice pipeline that needs the first sentence fast
• A desktop copilot with one active session
• An internal tool where privacy rules out multi-tenant batching

In those cases, spare GPU compute sits idle while memory bandwidth chokes AR decoding.

Google's Figure 12 in the technical report makes the trade explicit. DiffusionGemma wins on per-user throughput at low batch sizes. Autoregressive models with speculative decoding only pull ahead around 32 concurrent requests on an H100 in their PG-19 benchmark setup.

I would not rip out your production AR stack on headline TPS alone. I would A/B it on the paths where one user waits on one GPU.

Quality versus speed: read the fine print

Diffusion mode is faster. It is not free.

Across Google's full eval suite, DiffusionGemma in text-diffusion mode trades benchmark score for throughput compared to the Gemma 4 AR baseline it started from. The SD·RL stage fixes early SFT failure modes (repetitive token loops that triggered adaptive stopping too early), but you should still expect a capability gap on hard reasoning versus the AR parent.

The dual-mode design is the interesting part. The same weights can run:

• Diffusion decoding for ultra-low latency
• Standard AR decoding with partial recovery of the quality gap

That opens hybrid routing: fast diffusion for drafts, summaries, and UI copy; AR mode (or a frontier model) when the eval harness says the task failed your threshold.

The model also inherits Gemma 4 features that most open diffusion baselines lack: thinking mode, multimodal inputs, and long context. Mercury and Gemini Diffusion stay API-only. LLaDA and Nemotron diffusion models ship open weights but sit on a different speed-to-intelligence curve in Google's Pareto plots.

How the sampler actually works

The default entropy-bound sampler (Algorithm 1 in the paper) is worth understanding before you tune flags.

Each denoising step:

1. Runs the decoder in bidirectional mode over the full 256-token canvas
2. Samples a candidate token at every position
3. Accepts positions from most confident to least until an entropy budget (0.1 by default) is exhausted
4. Re-randomizes rejected positions for the next step
5. Stops when the argmax canvas stabilizes and mean entropy drops below 0.005, or when it hits the step cap

Recommended hyperparameters from Table 2 in the report:

Average effective steps land around 12 across the eval suite, which is how you get ~20 tokens per forward pass and the ~1,500 TPS headline.

Self-conditioning feeds the previous step's softmax back through a small gated MLP so re-randomized positions are not starting cold. vLLM implements this inside its diffusion-specific ModelState and DiffusionSampler, reusing the speculative decoding data path with a clever twist: during denoising, zero tokens are committed to the KV cache until a block is accepted.

Serve it with vLLM

Google collaborated with the vLLM team on day-zero support. The DiffusionGemma developer guide and the vLLM integration post are the references I would keep open while tuning.

Minimum viable single-GPU server (matches Google's published recipe):

vllm serve google/diffusiongemma-26B-A4B-it \
  --trust-remote-code \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.85 \
  --attention-backend TRITON_ATTN \
  --generation-config vllm \
  --hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
  --diffusion-config '{"canvas_length": 256}' \
  --enable-chunked-prefill \
  --host 0.0.0.0 \
  --port 8000

A few flags deserve explicit attention:

Quick smoke test once the server is up:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/diffusiongemma-26B-A4B-it",
    "messages": [{"role": "user", "content": "Explain discrete diffusion for text in three sentences."}],
    "max_tokens": 512,
    "temperature": 0.7
  }'

For thinking mode, add --reasoning-parser gemma4 at serve time and pass "chat_template_kwargs": {"enable_thinking": true} in the request body.

Docker users can start from the vllm/vllm-openai:gemma image with the same flags. Quantized FP8 and NVFP4 checkpoints from Red Hat AI are available if you want to reproduce the ~1,008 TPS H100 numbers from the vLLM benchmarks.

Where I would actually deploy this

Strong fits:

• Local coding assistants where time-to-first-token dominates UX
• Voice or realtime copilots on a dedicated GPU with one active session
• High-volume structured generation (JSON blocks, form fills, short summaries) where parallel canvas refinement helps
• Research and fine-tuning on diffusion mechanics (Hackable Diffusion adapter in the paper)

Weak fits:

• Multi-tenant SaaS at 32+ concurrent requests per GPU (AR batching wins)
• Tasks where you already fail Gemma 4 quality bars in AR mode
• Tool-heavy agent loops that need frontier-level reliability without a fallback route

Community adoption moved fast even in the paper's short post-release window: speech recognition finetunes, radiology report drafting, Sudoku format-following with 8M-parameter LoRA. The Apache 2.0 license removes friction for commercial experiments.

What I am benchmarking next

My checklist for any client stack flirting with local models:

1. Log time-to-first-token and tokens per second at batch size 1 on your real prompts
2. Compare diffusion mode against Gemma 4 AR and your current default route on the same eval harness
3. Test hybrid routing: diffusion draft, AR or frontier verification on failed checks
4. Watch VRAM with --max-num-seqs 4 before you crank concurrency

DiffusionGemma does not replace frontier models for hard reasoning. It redefines the latency floor for open-weight local serving when one user owns the GPU.

If you want help scoping a local inference stack (model choice, vLLM flags, eval harness, hybrid routing), book a free discovery call. I would rather measure TPS on your prompts than debate leaderboard averages.