DiffusionGemma: Google Brings Diffusion Models into Text Generation

Google DeepMind has released DiffusionGemma, an experimental new branch in the Gemma family. It does not keep pushing along the traditional autoregressive path of predicting one token at a time. Instead, it brings the idea of diffusion models into text generation: first create a noisy text canvas, then gradually refine the whole segment through multiple denoising steps.

Google’s positioning is clear: this is an experimental open model for researchers and developers exploring low-latency, local, interactive text generation workflows. It is not a production-quality replacement for standard Gemma 4.

Key facts

Two details matter most. First, this is not a small model; it is a 26B MoE. Second, only 3.8B parameters are active during inference, and the design tries to move the bottleneck away from memory bandwidth and toward compute.

How it differs from ordinary LLMs

A traditional autoregressive LLM works like a typewriter: it generates left to right, one token after another. This method is mature and reliable, and it is well suited to high-quality long-form output. But for single-user local inference, it has a practical problem: the GPU is often not fully fed, and the bottleneck is repeated weight loading plus token-by-token decoding.

DiffusionGemma takes a different route. It first creates a random 256-token canvas, then performs multiple rounds of parallel refinement. Within each round, the tokens on the canvas can see one another. The model does not only look backward; it can use bidirectional attention inside the block.

This leads to three direct consequences:

• Generation is not strictly left to right; the whole block converges together.
• The model can revise earlier positions during generation.
• It is more natural for code infilling, inline editing, bracket and tag closure, Sudoku-like tasks, and other nonlinear constraints.

In other words, DiffusionGemma is not chasing “a larger model on the same path.” It is testing another path for text generation: treating text as a canvas that can be repeatedly refined.

Why it can be faster

The key point Google emphasizes is that DiffusionGemma tries to shift the bottleneck from memory bandwidth to compute.

An autoregressive model repeatedly accesses model weights for every generated token. In single-user, local, low-batch inference, GPU compute may not be fully utilized. DiffusionGemma processes a 256-token canvas at once, giving the GPU a larger parallel workload and making it easier to keep tensor cores busy.

Google’s reported numbers include:

• More than 1000 tokens/s on a single NVIDIA H100.
• More than 700 tokens/s on an NVIDIA GeForce RTX 5090.
• Up to about 4x faster text generation on dedicated GPUs.

But the speed claim has boundaries. Google also notes that DiffusionGemma’s advantage mainly applies to local, low-concurrency, single-accelerator, low-to-medium batch inference. In high-QPS cloud services, autoregressive models can use large batches to keep hardware saturated, so DiffusionGemma’s parallel decoding advantage may shrink and may even raise serving cost.

That point matters: this is more like a new route for local real-time interaction than a universal accelerator for every deployment.

How the architecture works

The developer guide gives a more specific explanation. DiffusionGemma generation can be split into two phases:

1. Prefill / Incremental Prefill

   It uses causal attention to read the prompt and write context into the KV cache. For long text, after each 256-token block is finished, the model commits the result into the KV cache before processing the next block.

2. Denoising

   It uses bidirectional attention to iteratively denoise the current canvas. Query tokens in the current block can see other tokens on the canvas and also use historical context already written into the KV cache.

This design is called block autoregressive denoising. It does not completely abandon ordering. Instead, it keeps block-to-block ordering for long-text stability while allowing parallel generation inside each block.

That tradeoff makes sense. Fully parallel generation makes long-text consistency hard; fully autoregressive generation returns to the token-by-token bottleneck. DiffusionGemma chooses “ordered between blocks, diffusion inside blocks.”

Best-fit scenarios

DiffusionGemma is not primarily aimed at ordinary chat. It is best suited to interactive scenarios that need low latency, fast rewriting, local completion, and global constraints.

Typical directions include:

• Inline editing: the user changes one sentence, and the model quickly fills in a local replacement.
• Code infilling: not writing from the start of a file to the end, but filling a gap in the middle.
• Format closure for Markdown / JSON / XML: the model can see the whole output block and more easily fix brackets, tags, and list structure.
• Nonlinear text structures: graphs, tables, Sudoku, amino acid sequences, mathematical graph structures, and similar tasks.
• Local real-time tools: developer tools, editor plugins, and desktop AI assistants that need updates while the user types.

The official developer guide also includes a Sudoku fine-tuning example. The base model is not specifically trained to solve Sudoku and starts with a success rate near zero. After a simple JAX SFT recipe, Sudoku accuracy rises to 80%, while the number of inference steps drops. The point is not that it is “for Sudoku,” but that bidirectional denoising is better suited to strongly constrained, multi-variable tasks that require global consistency.

Poor-fit scenarios

DiffusionGemma is still experimental, so speed is not the only factor.

Google states plainly that because it prioritizes speed and parallel layout generation, its overall output quality is lower than standard Gemma 4. For applications that need the highest quality, standard Gemma 4 is still recommended.

It may also be a poor fit for:

• High-quality long-form writing.
• High-concurrency cloud API serving.
• Production tasks that require high output stability and factual accuracy.
• Local inference mainly relying on Apple Silicon unified memory.

The last point also comes from Google’s explanation: DiffusionGemma’s acceleration depends on high arithmetic intensity on accelerators. Apple Silicon-style unified memory architectures are often more constrained by memory bandwidth during inference, so they may not see the same relative speedup over autoregressive models.

Deployment and tooling

DiffusionGemma weights are available from Hugging Face, and the model can also be accessed through Kaggle and Google Cloud Model Garden. The official developer guide provides a vLLM local OpenAI-compatible server example:

1
2
3
4
5
6
7
8
9

vllm serve google/diffusiongemma-26B-A4B-it \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.85 \
  --attention-backend TRITON_ATTN \
  --generation-config vllm \
  --hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
  --diffusion-config '{"canvas_length": 256}' \
  --enable-chunked-prefill

Google also mentions support across ecosystems including:

• vLLM
• Hugging Face Transformers
• SGLang
• MLX
• Hackable Diffusion
• Unsloth
• NVIDIA NeMo
• NVIDIA NIM

Google says llama.cpp support is coming soon. For local model users, that is an important signal, but until support actually lands, the practical toolchain should be verified by what runs today.

Relationship to Gemma 4

DiffusionGemma is not a replacement for Gemma 4. It is more like an experimental branch of the Gemma 4 family.

One way to frame it:

• Standard Gemma 4: better for quality-first production output.
• DiffusionGemma: better for speed-first, low-latency, local interaction, and nonlinear generation experiments.

It builds on the Gemma 4 backbone and Gemini Diffusion research, but its goal is not simply to raise benchmark scores. It is testing whether text diffusion can change developer workflows, especially interactions that autoregressive generation has historically struggled with: real-time editors, code infilling, and instant repair of structured content.

Why it is worth watching

DiffusionGemma is worth watching not because it instantly becomes the strongest text model, but because it shifts a basic assumption in text generation.

For the past few years, text models have almost defaulted to autoregressive generation. That path is mature, but it also makes output a linear process: write the beginning first, then the rest, and errors written early are hard to revise. Diffusion-style text generation offers another possibility: sketch the whole thing first, then repeatedly fix local parts until the full block becomes clear.

This is especially interesting for developer tools. Real editing rarely starts from a blank document and proceeds straight downward. It involves insertion, deletion, completion, formatting, local repair, and filling gaps in the middle. DiffusionGemma’s structure is closer to that “local editing + global constraints” workflow.

Summary

DiffusionGemma is an experimental open model whose core idea is to replace traditional token-by-token generation with text diffusion and block-level parallel denoising. On dedicated GPUs, local low-concurrency, real-time interaction scenarios may see meaningful speedups. It is also better suited to inline editing, code completion, structured text, and multi-variable constraint tasks.

But it is not a production-quality replacement for Gemma 4. At this stage, it is better suited to research, tool prototypes, and developer experiments. For model selection, put it in the “low-latency interaction model” column, not the “best general-purpose large model” column.

References:

• DiffusionGemma - Google DeepMind
• DiffusionGemma: 4x faster text generation
• DiffusionGemma: The Developer Guide