Gemma 4 Local VRAM Guide: Choosing E2B, E4B, 12B, 26B, or 31B

Gemma 4 local deployment discussions should not list only E2B, E4B, 26B, and 31B. Google’s official material also includes a 12B model. Names containing E or A describe embedding or active-parameter scales; they are not enough to infer weight size.

This guide keeps three numbers separate: Google’s approximate inference memory, the size of a particular community GGUF file, and peak VRAM measured with a specific backend. Only the third answers how much memory your machine actually uses.

Model positioning

Check the matching official model card for architecture, modalities, and context limits.

Why a VRAM table is only a starting point

Google’s approximate memory table describes weight-loading scale and does not include every runtime cost. Local GGUF use also depends on:

• the effective bit rate of Q4_K_M, Q5_K_M, Q6_K, or Q8_0;
• KV-cache type and context length;
• concurrency and batch size;
• GPU-offload layer count;
• a vision encoder or multimodal projection file;
• the llama.cpp or Ollama version.

A 9 GB GGUF file does not prove that 9 GB of VRAM is sufficient.

Consumer-GPU selection baseline

These are conservative test directions for a quantized model, short context, and one concurrent request—not guarantees:

Do not claim deployability from arithmetic alone when no compatible quantized file or backend exists.

Avoid downloading the wrong model

1. Enter the model card from Google’s official Gemma page.
2. Distinguish base from it (instruction tuned).
3. Trace a GGUF repository back to the upstream weights.
4. Save the complete filename, quantization, shards, and SHA-256.
5. Confirm the Gemma license permits your use and redistribution.

1

Get-FileHash .\gemma-model.gguf -Algorithm SHA256

llama.cpp measurement procedure

After confirming that your llama.cpp build supports the target architecture, start with a 4K context:

1
2
3
4
5
6
7

.\llama-server.exe `
  -m .\gemma-model.gguf `
  -ngl 999 `
  -c 4096 `
  -np 1 `
  --host 127.0.0.1 `
  --port 8080

Monitor the GPU separately:

1

nvidia-smi -l 1

Record the model buffer, KV cache, compute buffer, and offloaded layers. Repeat at 8K context without changing the model or quantization.

Interpreting the result

• OOM during startup: reduce context and concurrency, then try a smaller quantization.
• Loads but runs slowly: check whether substantial weights are on CPU.
• Broken formatting: verify tokenizer and chat template.
• Image input fails: confirm model, projection file, and backend multimodal support.
• Different use for the same quantization: compare backend version, KV type, and offload settings.

Inventory four resources before deployment

GPU capacity alone misses RAM, storage, and CPU limits:

1
2
3
4
5
6
7
8

Get-CimInstance Win32_VideoController |
  Select-Object Name, AdapterRAM, DriverVersion

Get-CimInstance Win32_ComputerSystem |
  Select-Object TotalPhysicalMemory

Get-PSDrive -PSProvider FileSystem |
  Select-Object Name, Free, Used

Some Windows drivers report AdapterRAM incorrectly, so confirm it with nvidia-smi. RAM must cover non-offloaded weights, runtime buffers, and the OS. Storage must hold download shards and every quantization being compared. CPU and memory-channel bandwidth matter whenever offload is used.

How each model fits real hardware

E2B: validate compatibility first

Use E2B to confirm that the backend recognizes Gemma 4, applies the right chat template, and exposes a working local API. Its main benefit is low resource use, not replacing a larger reasoning model.

If even E2B stays on CPU, fix the GPU backend or driver before testing a larger model.

E4B: a practical 8–12 GB starting point

E4B is suitable for chat, summaries, classification, and light coding. Start with Q4 or Q5 at 4K context, then decide whether Q6/Q8 produces a meaningful quality gain.

Vision tests need a separate record for projection-file and image-token memory.

12B: reserve KV-cache capacity

A 12B quantization can approach the comfortable limit of a 12 GB GPU. Reserve roughly 1–2 GB for the desktop and runtime, and begin with 2K/4K context.

If Q4 fits fully on GPU while Q5 requires CPU offload, Q5 may be slower enough to lose the practical advantage of its higher precision.

26B A4B: active parameters are not load size

A4B does not mean the model consumes the memory of a 4B model. Consumer GPUs usually require lower quantization or CPU offload. Confirm explicit MoE support and correct expert loading in the log.

31B: define a speed threshold

A 24 GB card usually needs quantization and limited context. Define acceptable first-token latency and generation speed before testing; otherwise “it loads” may describe a configuration too slow for daily use.

Read the important parts of a GGUF filename

1
2

gemma-4-12b-it-q4_k_m.gguf
gemma-4-26b-a4b-it-q5_k_m-00001-of-00003.gguf

Confirm base versus instruction tuning, parameter scale, full quantization name, all shards, upstream repository, and plausible total size. An unusually small “26B” or “31B” download may be only a projection, LoRA, index, or first shard.

Confirm identity from metadata

Use GGUF metadata or loader logs to check the architecture, tokenizer, chat template, requested context, and all shards. Stop when you see unknown tensor or unsupported architecture.

Do not compensate for a broken template by repeatedly changing the system prompt.

Test context in steps

Restart between rounds so the previous cache is released. Record KV-cache type when the backend makes it configurable.

Tune partial GPU offload

If -ngl 999 OOMs, lower the GPU layer count gradually:

1
2
3
4
5
6
7

.\llama-server.exe `
  -m .\gemma-model.gguf `
  -ngl 40 `
  -c 4096 `
  -np 1 `
  --host 127.0.0.1 `
  --port 8080

For each step record GPU/RAM peak, prompt-processing speed, generation speed, first-token latency, and paging. Heavy paging means it is time for a smaller model or lower quantization—not a larger page file.

Multi-GPU memory is not simple addition

Backend tensor splitting, unequal GPUs, and PCIe topology all matter. Monitor each card:

1

nvidia-smi --query-gpu=index,name,memory.total,memory.used,utilization.gpu,power.draw --format=csv -l 1

If cross-GPU transfers reduce performance, one GPU plus modest CPU offload may be simpler. Judge end-to-end latency, not merely nonzero utilization on both cards.

Validate the local API, not only a web UI

1

curl.exe http://127.0.0.1:8080/v1/models

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$body = @{
  model = 'local-gemma'
  messages = @(
    @{ role = 'user'; content = '把 17×23 的计算过程写成两步。' }
  )
  temperature = 0
} | ConvertTo-Json -Depth 5

Invoke-RestMethod `
  -Uri 'http://127.0.0.1:8080/v1/chat/completions' `
  -Method Post `
  -ContentType 'application/json' `
  -Body $body

Check the HTTP status, model field, answer, and server log. A working web UI does not prove that API clients use the correct endpoint and request shape.

Common selection questions

E4B or 12B on a 12 GB GPU?

Choose E4B for speed, longer context, or running other software. Test 12B Q4 when answer quality matters more and short context is acceptable.

Can a 24 GB GPU go straight to 31B?

Q4 with short context is a reasonable experiment, not a guarantee. Check the actual file, loader log, and peak VRAM.

Why do two Q4 files have different sizes?

Q4 is a family. Mixed-precision tensors, grouping, and metadata change the effective bit rate; always record the full quantization name.

Why does VRAM grow after loading?

Tokens accumulate in KV cache, while concurrency and image input add buffers. Record both idle-after-load and peak-after-longest-request memory.

Save a reproducible Gemma test record

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

GPU / 显存：
驱动：
后端与版本：
Gemma 型号（base/it）：
GGUF 文件与 SHA256：
量化：
上下文 / 并发：
GPU offload：
空载与峰值显存：
首 token 延迟 / tokens/s：

Google and inference-backend references

• Google Gemma getting started
• Gemma core models and memory guidance
• llama.cpp official repository

Read Names Such as Gemma-4-31B-it

Parse the name in parts. Gemma-4 identifies the family and generation. 31B means roughly 31 billion total parameters; it does not mean exactly 31GB at runtime or that every parameter is active for every token. it normally means instruction-tuned: the pretrained model has been adapted for instructions, question answering, and conversation. Choose -it for chat, summarization, coding assistance, and agents; base checkpoints are mainly for further training or research.

Download names can also include precision, quantization, and format suffixes such as Q4_K_M, Q8_0, BF16, or GGUF. Those suffixes directly affect file size, backend support, and memory. If a name does not match the official model table, inspect the model card, repository owner, and metadata rather than assuming the filename represents an official release.

Laptop Configuration and Acceptance Test

A laptop is constrained by shared memory, cooling, and sustained power as well as VRAM. On a 16GB machine without a discrete GPU, start with a low-bit E2B or E4B and a 4K context. An 8GB GPU is a reasonable E4B Q4 validation target; try 12B Q4 cautiously with 12–16GB VRAM. Apple Silicon uses unified memory, so leave several gigabytes for macOS and applications instead of treating all installed memory as model capacity.

Close GPU-heavy browser tabs and media applications, connect power, and watch temperature during the first run. Do not begin with a 32K or 128K context: KV cache grows with context, so a model that loads successfully can still run out of memory on a long prompt.

1
2

ollama ps
nvidia-smi

Record the exact model tag, quantization, context, backend, first-token latency, generation speed, and peak memory. A CPU-only response proves compatibility, not interactive performance. If the laptop throttles, starts swapping, or misses your tokens-per-second target, reduce model size, quantization cost, or context. Keep the previous model or Modelfile until the replacement passes the same test.