Qwen3.6 Local VRAM Guide: Measuring 27B and 35B-A3B Quantizations

The two common open weight versions of Qwen3.6 for local deployment are Qwen3.6-27B and Qwen3.6-35B-A3B. The former is a 27B dense model, and the latter is a MoE model with about 35B total parameters and each token activates about 3B parameters.

MoE has fewer activation parameters, which can usually reduce the amount of calculation per token, but it will not reduce the weight that needs to be loaded to 3B. When choosing a graphics card, you should first look at the total weight and quantization file, and then calculate the KV cache and running margin.

Do not mix three different memory figures

• Theoretical weight volume: calculated using parameter quantity and bit width, only used for preliminary screening;
• GGUF file size: The size of the specific conversion file on disk, including quantitative metadata;
• Runtime memory: the sum of weights, KV cache, calculation buffer, multi-modal projection and backend overhead.

Writing “Q4 requires 18GB of VRAM” online without explaining the model file, context and backend cannot be regarded as a conclusion from actual measurements.

Theoretical bare-weight estimate

Formula:

1

权重 GiB ≈ 参数量 × 位宽 ÷ 8 ÷ 1024³

The actual GGUF is usually larger than the corresponding bare weight estimate. The effective average bitwidth varies between Q4/Q5 plans, so the table is not a proxy for the actual file size of the download page.

Initial selection by available VRAM

The following is the deployment starting point, not a guaranteed value. The default is single concurrency, about 4K context and allows necessary CPU offload:

If you load the vision mmproj at the same time, increase the context to hundreds of thousands of tokens, or increase concurrency, the memory requirements will increase significantly. Do not treat the model card’s maximum context as the default startup value.

Record the complete GGUF filename

Document at least:

1
2
3
4
5
6

基础模型：Qwen/Qwen3.6-27B 或 Qwen/Qwen3.6-35B-A3B
转换仓库与 commit：
GGUF 文件名：
量化：Q4_K_M / Q5_K_M / 其他
SHA256：
llama.cpp 版本：

Third-party GGUFs should be traceable to official base weights and describe conversion tools and licenses. Only the warehouse names are similar, which cannot prove that the documents are from the official source.

Measure with llama.cpp

Starting with 4K context, single concurrency:

1
2
3
4
5
6
7

.\llama-server.exe `
  -m .\qwen-model-q4_k_m.gguf `
  -ngl 999 `
  -c 4096 `
  -np 1 `
  --host 127.0.0.1 `
  --port 8080

Open another terminal record:

The

1

nvidia-smi --query-gpu=name,memory.total,memory.used,driver_version --format=csv

startup log also stores model buffer, KV cache, offload layer number and backend information. Then run the same prompt word in 4K and 8K context respectively, and record the peak memory, first token delay and generated tokens/s.

Measure with Ollama

The specific label of

1

ollama run qwen3.6:27b

is subject to the current page of the Ollama model library. Execute after running:

PROCESSOR of

1
2

ollama ps
nvidia-smi

ollama ps shows CPU/GPU allocation. If the model label does not specify the quantification type, the results cannot be directly compared to Q4_K_M of a GGUF.

Deployment differences between dense models and MoE

Each layer of the 27B dense model participates in the calculation, and the structure is relatively straightforward; 35B-A3B will route to some experts based on the token. When deploying MoE, in addition to weight capacity, you also need to pay attention to whether the backend correctly implements expert routing.

Possible differences include:

• Under the same weight volume, the calculation amount of each token is different;
• MoE is more sensitive to memory access and expert scheduling;
• Different backends have different GPU/CPU allocation strategies for experts;
• Some old versions of the backend can read the configuration, but cannot generate it correctly;
• Expert cross-device transfer may become a bottleneck when multiple cards are allocated.

Therefore, the 35B-A3B cannot be compared only to the 3B model for speed, nor can it be compared only to the 35B dense model for VRAM.

Calculate disk and system memory before downloading

A local deployment needs download cache, GGUF files, and runtime memory mapping. Testing several quantizations can exhaust disk space first.

Windows can check:

1
2
3
4
5

Get-PSDrive -PSProvider FileSystem |
  Select-Object Name, Used, Free

Get-CimInstance Win32_ComputerSystem |
  Select-Object TotalPhysicalMemory

system memory should not only equal unoffloaded weights. If the model part is placed on the CPU, space should also be reserved for the operating system, file cache, KV cache and back-end buffer. After the system starts to change pages frequently, the generation speed will drop significantly.

Create independent startup configurations for two models

Don’t modify the same long command repeatedly. Creating separate scripts can avoid misuse of 35B context or files when testing 27B.

27B Start command

1
2
3
4
5
6
7

.\llama-server.exe `
  -m .\qwen3.6-27b-q4_k_m.gguf `
  -ngl 999 `
  -c 4096 `
  -np 1 `
  --host 127.0.0.1 `
  --port 8081

35B-A3B start command

After the

1
2
3
4
5
6
7

.\llama-server.exe `
  -m .\qwen3.6-35b-a3b-q4_k_m.gguf `
  -ngl 999 `
  -c 4096 `
  -np 1 `
  --host 127.0.0.1 `
  --port 8082

ports are separated, they can be started and compared one by one; do not run two services at the same time when the VRAM is insufficient.

Verify the chat template and reasoning output

Different versions of Qwen may have different chat templates, thinking modes or special tokens. When the following phenomena occur, first check the model card and backend support:

• Output and print character labels such as system, user as they are;
• Answers are repeated;
• The boundary between the thinking content and the final answer is abnormal;
• Mix in additional tags before JSON output;
• The tool call parameter is not valid JSON.

Don’t “fix” by removing random special tokens. Template errors can lead to completely different results between benchmark tests and real-world applications.

API connectivity and model identity check

After starting both services, check the model list separately:

1
2

curl.exe http://127.0.0.1:8081/v1/models
curl.exe http://127.0.0.1:8082/v1/models

uses the same request again to compare:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$payload = @{
  model = 'local-qwen'
  messages = @(
    @{ role = 'user'; content = '用 JSON 返回三个 Linux 日志排查步骤。' }
  )
  temperature = 0
} | ConvertTo-Json -Depth 5

Invoke-RestMethod `
  -Uri 'http://127.0.0.1:8081/v1/chat/completions' `
  -Method Post `
  -ContentType 'application/json' `
  -Body $payload

Change the port to 8082 and repeat the test. Check the actual model file in the service log to avoid the model field passed in by the client masking the file loaded by the backend.

KV cache should be measured separately

The measurement sequence can be:

1. The service has just been started and has not been requested;
2. Enter 1K token;
3. Enter 4K token;
4. Enter 8K token;
5. Keep 8K and add a second concurrent request.

Record the memory peak and response time at each step. If you only record the VRAM after startup, you will miss the long context and concurrent cache.

The test text should come from a local fixed file to avoid different token numbers each time. You can first use client tokenizer statistics, or check the actual input token from the API usage field.

Multi-card deployment first confirms the allocation result

Two 24GB graphics cards don’t necessarily automatically result in the ideal 48GB of space. Different backends may be split by layer, tensor, or expert, and a card may also host additional output layers and caches.

Observe each card:

1

nvidia-smi --query-gpu=index,name,memory.total,memory.used,utilization.gpu --format=csv -l 1

If one card is close to OOM and the other card still has a lot of idle time, you should check the tensor split and device selection parameters of the current llama.cpp. The parameters will change with the version. Before use, refer to the help output of the corresponding release:

1

.\llama-server.exe --help

Do not copy the multi-card parameters of the old version and use them directly for production.

Visual input requires three matchings

Multi-modal deployment needs to meet both:

• The Qwen model natively supports visual input;
• mmproj matches the specific base model;
• The current llama.cpp build supports this multimodal architecture.

Visual projection files cannot be mixed between 27B and 35B-A3B. When testing, first use a small picture and a clear question, and then check whether the log actually loads the visual encoder. Just receiving a text answer does not mean the image has been processed.

Also keep track of image size and quantity, as visual tokens also expand context and cache.

Establish your own task baseline

It is recommended to include at least five types of tasks:

Each task is run at least three times with fixed temperature, seed, and maximum output length. MoE and dense model fluctuations, speed and accuracy should be recorded together.

Separate prompt and generation speed

• Prompt processing: The speed of reading input, long text and RAG pay more attention to it;
• Token generation: The speed of generating answers, the chat experience pays more attention to it.Publishing only one tokens/s can be confusing. If 35B-A3B is fast to generate but slow to process long input, it may not be suitable for your document workflow.

The first token delay must also be listed separately. Users are often more sensitive to the time they wait for the first word than the final throughput.

Decide whether a higher quantization is worthwhile

Before moving from Q4_K_M to Q5_K_M, answer:

• Whether the error rate of critical tasks has been significantly reduced;
• Whether structured output is more stable;
• Whether the VRAM still leaves room for KV cache;
• Is the slowdown in generation speed acceptable?
• Whether to change from full GPU to partial CPU offload.

If upgrading quantization results in a large amount of offload, a small improvement in quality may not be worth the loss in speed.

FAQ

Why 35B-A3B is larger but potentially faster than 27B

It requires storing more total weights, but each token only activates some experts. Actual speeds depend on expert routing, memory bandwidth, and backend optimizations and cannot be extrapolated from activation parameters alone.

Which 24GB graphics card should I choose?

Test 27B Q4/Q5 first, then test 35B-A3B Q4. Fixed context and task, compare full GPU residency, speed and accuracy, don’t directly regard MoE as a sure win.

Why does changing to lower quantization still cause OOM?

It could be that there are no changes in context, concurrency, visual projection, or other program occupancy. Look at the boot log and nvidia-smi, don’t just compare the GGUF file size.

Is it possible to directly use the maximum context of the model card?

Theoretical support does not mean that the native configuration can bear it. Start with 4K/8K first, increase and record the KV cache according to actual needs.

Recovery sequence after OOM

1. Turn off concurrency and keep -np 1;
2. Reduce the context from 8192 to 4096 or 2048;
3. Change to smaller quantization;
4. Reduce GPU offload and use system memory to take over part of the weight;
5. Stop other processes occupying VRAM;
6. Restart the service and confirm that the configuration takes effect from the startup log.

CPU offload turns “runs” into “runs slowly,” so the speed must be recorded at the same time.

27B or 35B-A3B

• Requires simple deployment and stable back-end compatibility: priority comparison is 27B;
• Focus on the inference efficiency of MoE: measured 35B-A3B on the same hardware;
• Only 12GB of VRAM: give priority to smaller models, and do not force a large amount of offload for the model;
• Doing multi-modality: first check the official model card and the visual support of the backend, and then include the mmproj overhead.

Qwen model cards and backend references

• Qwen3.6-27B official model card
• Qwen3.6-35B-A3B official model card
• llama.cpp official warehouse