Deploying DiffusionGemma Locally: Running Google’s Text Diffusion Model with vLLM

The previous article explained why DiffusionGemma is worth watching: it is not traditional token-by-token autoregressive generation, but uses text diffusion and parallel denoising over a 256-token canvas, making it better suited to low-latency local interaction, inline editing, and code completion.

This article focuses on the practical question: how to deploy it and run it from the command line.

The main official path today is vLLM. DiffusionGemma can be started through vLLM’s OpenAI-compatible local server, then queried with an interface similar to OpenAI Chat Completions.

Before you start

First decide whether DiffusionGemma is worth trying on your machine.

DiffusionGemma is a 26B total MoE model, with 3.8B active parameters during inference. It is not a small model. MoE, quantization, and parallel generation only bring the local deployment threshold down into the range that high-end consumer GPUs can explore.

If you only want stable long-form writing, knowledge Q&A, or production APIs, standard Gemma 4 is still safer. DiffusionGemma is more suitable for trying low-latency editors, code infilling, and instant structured-text repair.

Option 1: start directly with vLLM

The core command from the official developer guide is:

1
2
3
4
5
6
7
8
9

vllm serve google/diffusiongemma-26B-A4B-it \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.85 \
  --attention-backend TRITON_ATTN \
  --generation-config vllm \
  --hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
  --diffusion-config '{"canvas_length": 256}' \
  --enable-chunked-prefill

This command pulls google/diffusiongemma-26B-A4B-it from Hugging Face and starts a local OpenAI-compatible server. By default, the service usually listens on http://localhost:8000.

If your Hugging Face environment needs authentication, run:

1

huggingface-cli login

If vLLM is not installed, you can start with a Python virtual environment:

1
2
3
4

python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -U vllm

Whether pip install -U vllm is enough depends on whether the current vLLM release already includes DiffusionGemma support. DiffusionGemma is a new architecture. If you see unknown model structures, unrecognized parameters, or attention backend errors, check the latest vLLM release, the Google developer guide, and the model card first.

Option 2: run vLLM with Docker

If you do not want to modify the local Python environment, you can use a vLLM Docker image. vLLM recipes have used commands similar to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

docker run -itd --name diffusiongemma \
  --ipc=host \
  --network host \
  --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:gemma \
    --model google/diffusiongemma-26B-A4B-it \
    --max-model-len 262144 \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.85 \
    --generation-config vllm \
    --enable-chunked-prefill \
    --host 0.0.0.0 \
    --port 8000

This keeps the environment cleaner and is useful on servers, workstations, or temporary test machines. Two things matter:

• The host must already have the NVIDIA driver and NVIDIA Container Toolkit installed.
• If the vLLM version inside the image does not include DiffusionGemma support, startup will still fail. Use a newer image or the appropriate branch.

The cache mount -v ~/.cache/huggingface:/root/.cache/huggingface is useful if you want to reuse the host Hugging Face cache and avoid downloading the model every time.

Test the service with curl

After the service starts, check the model list:

1

curl http://localhost:8000/v1/models

If the response includes google/diffusiongemma-26B-A4B-it, the service is basically up.

Then test Chat Completions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/diffusiongemma-26B-A4B-it",
    "messages": [
      {
        "role": "user",
        "content": "Explain the difference between DiffusionGemma and ordinary autoregressive LLMs in three sentences."
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

If you prefer the OpenAI SDK, point base_url to the local service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="google/diffusiongemma-26B-A4B-it",
    messages=[
        {
            "role": "user",
            "content": "Write a Python function that converts a Markdown table to CSV.",
        }
    ],
    max_tokens=512,
)

print(response.choices[0].message.content)

Understanding the parameters

Several parameters in the official command deserve a closer look.

--max-model-len 262144

This sets the maximum context length. DiffusionGemma / Gemma 4 supports very long context, but that does not mean you should always open the limit.

Longer context increases VRAM and scheduling pressure.

For a first local test, you can keep the official value. If VRAM is tight, reduce it and check whether your actual task is affected.

--max-num-seqs 4

This limits the number of sequences processed at the same time. DiffusionGemma is better suited to local, low-concurrency interaction. Higher concurrency is not necessarily faster and may increase VRAM pressure.

For a single-user local tool, try values between 1 and 4. Multi-user serving needs more serious benchmarking.

--gpu-memory-utilization 0.85

This tells vLLM how much of GPU memory it may use. 0.85 is a common conservative value.

If startup OOMs, try:

1

--gpu-memory-utilization 0.75

If VRAM is plentiful, you can raise it a little, but do not max it out at the beginning. Leave room for the system and other processes.

--attention-backend TRITON_ATTN

This selects the attention backend. The official command uses TRITON_ATTN, which is related to DiffusionGemma’s special attention and denoising path.

If the backend is unsupported, the issue is often a mismatch among vLLM, CUDA, Triton, and GPU architecture. Do not randomly change model parameters first; check the software stack.

--hf-overrides

This part of the official command is important:

1

--hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}'

It overrides diffusion sampler settings in the Hugging Face config. entropy_bound can be understood as a strategy controlling denoising or sampling behavior, used together with DiffusionGemma’s iterative generation.

This is not a normal LLM parameter. Start with the official value, confirm it runs, then experiment.

--diffusion-config

The official command uses:

1

--diffusion-config '{"canvas_length": 256}'

canvas_length corresponds to DiffusionGemma’s 256-token canvas. The model does not generate one token at a time linearly; it denoises in parallel inside a block. This value is directly tied to the block diffusion generation mechanism.

Do not change it casually at first. Use the official value to verify speed, quality, and VRAM usage, then test according to later vLLM documentation.

--enable-chunked-prefill

This enables chunked prefill. DiffusionGemma’s long-sequence processing coordinates prefill and denoising, and chunked prefill can help scheduling in long-context scenarios.

If you only test short prompts, you may not feel much difference. It matters more for long context.

A more conservative local test command

If you only want to see whether it can start, lower concurrency and VRAM pressure:

1
2
3
4
5
6
7
8
9

vllm serve google/diffusiongemma-26B-A4B-it \
  --max-model-len 65536 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.75 \
  --attention-backend TRITON_ATTN \
  --generation-config vllm \
  --hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
  --diffusion-config '{"canvas_length": 256}' \
  --enable-chunked-prefill

This command is not necessarily optimal for performance, but it is better for first-time debugging. Get the model running first, then gradually increase context length and concurrency.

Good demos to try

DiffusionGemma should not be tested only with ordinary chat questions. Its real value is in nonlinear generation and real-time local repair.

Try prompts like these:

1
2
3
4

Complete the missing logic in this Python function. Output only the full function:

def markdown_table_to_csv(markdown: str) -> str:
    ...

1
2
3

Fix the following JSON so it becomes valid JSON, while preserving the original field meanings:

{"name":"demo","items":[{"id":1,"tags":["a","b",],},]}

1
2
3
4

Complete the following Markdown table to 5 rows and make sure every row has the same number of columns:

| Parameter | Purpose | Recommendation |
| --- | --- | --- |

1
2
3

You are an inline completion model inside an editor. Rewrite only the text inside brackets and keep the surrounding context coherent:

DiffusionGemma is suitable for [write a short phrase about low-latency interaction here], but not for quality-first long-form writing.

These tasks reveal its bidirectional attention, block-level self-repair, and structured output capabilities better than “tell me a story.”

Common issues

The model is unsupported at startup

First check the vLLM version. DiffusionGemma is new, and older vLLM versions may not include the implementation.

Check:

1

vllm --version

Then compare against the official developer guide, vLLM release notes, or the DiffusionGemma model card.

Hugging Face download fails

Check network and login status:

1

huggingface-cli whoami

Log in again if needed:

1

huggingface-cli login

On a server, consider pre-downloading the model or mounting the Hugging Face cache into the container.

OOM

Reduce pressure in this order:

1

--max-num-seqs 1

1

--gpu-memory-utilization 0.75

1

--max-model-len 65536

If it still OOMs, check whether quantized weights are being used, whether vLLM correctly loads the quantization format, and whether the GPU meets the model requirements.

Speed is not as high as expected

First confirm that your scenario is actually in DiffusionGemma’s advantage zone. Its speedup mainly targets local, low-concurrency, dedicated GPU, low-to-medium batch workloads.

In high-concurrency cloud serving, autoregressive models can use batching to saturate hardware, reducing DiffusionGemma’s advantage. Apple Silicon-style unified memory may also not show the same speedup.

Output quality is worse than Gemma 4

That is expected. Google explicitly states that because DiffusionGemma prioritizes speed and parallel layout generation, overall output quality is lower than standard Gemma 4. Quality-first production applications should still use standard Gemma 4.

Minimal validation flow

You can follow this sequence:

1. Log in to Hugging Face.

1

huggingface-cli login

2. Start the vLLM service.

1
2
3
4
5
6
7
8
9

vllm serve google/diffusiongemma-26B-A4B-it \
  --max-model-len 65536 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.75 \
  --attention-backend TRITON_ATTN \
  --generation-config vllm \
  --hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
  --diffusion-config '{"canvas_length": 256}' \
  --enable-chunked-prefill

3. Check the model list.

1

curl http://localhost:8000/v1/models

4. Send one request.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/diffusiongemma-26B-A4B-it",
    "messages": [
      {
        "role": "user",
        "content": "Complete a Python function: input a Markdown table string and output a CSV string."
      }
    ],
    "max_tokens": 512,
    "temperature": 0.4
  }'

5. Then test structured repair or code infilling.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/diffusiongemma-26B-A4B-it",
    "messages": [
      {
        "role": "user",
        "content": "Fix this JSON and output only valid JSON: {\"name\":\"demo\",\"items\":[{\"id\":1,\"tags\":[\"a\",\"b\",],},]}"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.2
  }'

Once these five steps work, tune context length, concurrency, and GPU memory utilization upward.

How DiffusionGemma Works and Where It Fits

Google DeepMind has released DiffusionGemma, an experimental new branch in the Gemma family. It does not keep pushing along the traditional autoregressive path of predicting one token at a time. Instead, it brings the idea of diffusion models into text generation: first create a noisy text canvas, then gradually refine the whole segment through multiple denoising steps.

Google’s positioning is clear: this is an experimental open model for researchers and developers exploring low-latency, local, interactive text generation workflows. It is not a production-quality replacement for standard Gemma 4.

Key facts

Two details matter most. First, this is not a small model; it is a 26B MoE. Second, only 3.8B parameters are active during inference, and the design tries to move the bottleneck away from memory bandwidth and toward compute.

How it differs from ordinary LLMs

A traditional autoregressive LLM works like a typewriter: it generates left to right, one token after another. This method is mature and reliable, and it is well suited to high-quality long-form output. But for single-user local inference, it has a practical problem: the GPU is often not fully fed, and the bottleneck is repeated weight loading plus token-by-token decoding.

DiffusionGemma takes a different route. It first creates a random 256-token canvas, then performs multiple rounds of parallel refinement. Within each round, the tokens on the canvas can see one another. The model does not only look backward; it can use bidirectional attention inside the block.

This leads to three direct consequences:

• Generation is not strictly left to right; the whole block converges together.
• The model can revise earlier positions during generation.
• It is more natural for code infilling, inline editing, bracket and tag closure, Sudoku-like tasks, and other nonlinear constraints.

In other words, DiffusionGemma is not chasing “a larger model on the same path.” It is testing another path for text generation: treating text as a canvas that can be repeatedly refined.

Why it can be faster

The key point Google emphasizes is that DiffusionGemma tries to shift the bottleneck from memory bandwidth to compute.

An autoregressive model repeatedly accesses model weights for every generated token. In single-user, local, low-batch inference, GPU compute may not be fully utilized. DiffusionGemma processes a 256-token canvas at once, giving the GPU a larger parallel workload and making it easier to keep tensor cores busy.

Google’s reported numbers include:

• More than 1000 tokens/s on a single NVIDIA H100.
• More than 700 tokens/s on an NVIDIA GeForce RTX 5090.
• Up to about 4x faster text generation on dedicated GPUs.

But the speed claim has boundaries. Google also notes that DiffusionGemma’s advantage mainly applies to local, low-concurrency, single-accelerator, low-to-medium batch inference. In high-QPS cloud services, autoregressive models can use large batches to keep hardware saturated, so DiffusionGemma’s parallel decoding advantage may shrink and may even raise serving cost.

That point matters: this is more like a new route for local real-time interaction than a universal accelerator for every deployment.

How the architecture works

The developer guide gives a more specific explanation. DiffusionGemma generation can be split into two phases:

1. Prefill / Incremental Prefill

   It uses causal attention to read the prompt and write context into the KV cache. For long text, after each 256-token block is finished, the model commits the result into the KV cache before processing the next block.

2. Denoising

   It uses bidirectional attention to iteratively denoise the current canvas. Query tokens in the current block can see other tokens on the canvas and also use historical context already written into the KV cache.

This design is called block autoregressive denoising. It does not completely abandon ordering. Instead, it keeps block-to-block ordering for long-text stability while allowing parallel generation inside each block.

That tradeoff makes sense. Fully parallel generation makes long-text consistency hard; fully autoregressive generation returns to the token-by-token bottleneck. DiffusionGemma chooses “ordered between blocks, diffusion inside blocks.”

Best-fit scenarios

DiffusionGemma is not primarily aimed at ordinary chat. It is best suited to interactive scenarios that need low latency, fast rewriting, local completion, and global constraints.

Typical directions include:

• Inline editing: the user changes one sentence, and the model quickly fills in a local replacement.
• Code infilling: not writing from the start of a file to the end, but filling a gap in the middle.
• Format closure for Markdown / JSON / XML: the model can see the whole output block and more easily fix brackets, tags, and list structure.
• Nonlinear text structures: graphs, tables, Sudoku, amino acid sequences, mathematical graph structures, and similar tasks.
• Local real-time tools: developer tools, editor plugins, and desktop AI assistants that need updates while the user types.

The official developer guide also includes a Sudoku fine-tuning example. The base model is not specifically trained to solve Sudoku and starts with a success rate near zero. After a simple JAX SFT recipe, Sudoku accuracy rises to 80%, while the number of inference steps drops. The point is not that it is “for Sudoku,” but that bidirectional denoising is better suited to strongly constrained, multi-variable tasks that require global consistency.

Poor-fit scenarios

DiffusionGemma is still experimental, so speed is not the only factor.

Google states plainly that because it prioritizes speed and parallel layout generation, its overall output quality is lower than standard Gemma 4. For applications that need the highest quality, standard Gemma 4 is still recommended.

It may also be a poor fit for:

• High-quality long-form writing.
• High-concurrency cloud API serving.
• Production tasks that require high output stability and factual accuracy.
• Local inference mainly relying on Apple Silicon unified memory.

The last point also comes from Google’s explanation: DiffusionGemma’s acceleration depends on high arithmetic intensity on accelerators. Apple Silicon-style unified memory architectures are often more constrained by memory bandwidth during inference, so they may not see the same relative speedup over autoregressive models.

Deployment and tooling

DiffusionGemma weights are available from Hugging Face, and the model can also be accessed through Kaggle and Google Cloud Model Garden. The official developer guide provides a vLLM local OpenAI-compatible server example:

1
2
3
4
5
6
7
8
9

vllm serve google/diffusiongemma-26B-A4B-it \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.85 \
  --attention-backend TRITON_ATTN \
  --generation-config vllm \
  --hf-overrides '{"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}' \
  --diffusion-config '{"canvas_length": 256}' \
  --enable-chunked-prefill

Google also mentions support across ecosystems including:

• vLLM
• Hugging Face Transformers
• SGLang
• MLX
• Hackable Diffusion
• Unsloth
• NVIDIA NeMo
• NVIDIA NIM

Google says llama.cpp support is coming soon. For local model users, that is an important signal, but until support actually lands, the practical toolchain should be verified by what runs today.

Relationship to Gemma 4

DiffusionGemma is not a replacement for Gemma 4. It is more like an experimental branch of the Gemma 4 family.

One way to frame it:

• Standard Gemma 4: better for quality-first production output.
• DiffusionGemma: better for speed-first, low-latency, local interaction, and nonlinear generation experiments.

It builds on the Gemma 4 backbone and Gemini Diffusion research, but its goal is not simply to raise benchmark scores. It is testing whether text diffusion can change developer workflows, especially interactions that autoregressive generation has historically struggled with: real-time editors, code infilling, and instant repair of structured content.

Why it is worth watching

DiffusionGemma is worth watching not because it instantly becomes the strongest text model, but because it shifts a basic assumption in text generation.

For the past few years, text models have almost defaulted to autoregressive generation. That path is mature, but it also makes output a linear process: write the beginning first, then the rest, and errors written early are hard to revise. Diffusion-style text generation offers another possibility: sketch the whole thing first, then repeatedly fix local parts until the full block becomes clear.

This is especially interesting for developer tools. Real editing rarely starts from a blank document and proceeds straight downward. It involves insertion, deletion, completion, formatting, local repair, and filling gaps in the middle. DiffusionGemma’s structure is closer to that “local editing + global constraints” workflow.

Summary

Deploying DiffusionGemma is not about finding a generic chat-model replacement. It is about validating a new local interaction route: start an OpenAI-compatible service with vLLM, then experiment around inline editing, code infilling, structured text repair, and low-latency output.

For a first deployment, use conservative parameters: --max-num-seqs 1, --max-model-len 65536, and --gpu-memory-utilization 0.75. After it runs, return to the official configuration and gradually test speed, VRAM, and output quality.

References:

• DiffusionGemma - Google DeepMind
• DiffusionGemma: 4x faster text generation
• DiffusionGemma: The Developer Guide
• Gemma 4 Usage Guide - vLLM Recipes