AI Tools

Deploy Qwen3.6-35B-A3B Locally: GGUF, llama.cpp, VRAM, and API Checks

Using the official Qwen3.6-35B-A3B weights as the baseline, this guide covers GGUF quantization, llama.cpp startup options, VRAM checks, vision projection files, and verification through an OpenAI-compatible API.

Qwen3.6-35B-A3B is Qwen’s official multi-modal MoE model with open weights. 35B in the name is the total parameter scale, and A3B means that each token activates approximately 3B parameters. Its calculation amount can be lower than that of a dense model of the same size, but it still has to accommodate all expert weights when loading, and it cannot be regarded as a 3B small model estimation memory.

This article only discusses the official base model and its GGUF conversion. The weights marked by third parties as Uncensored, Aggressive or “jailbroken version” are not officially released by Qwen. The training data, alignment modifications and capability statements need to be independently proven by the publisher, so they are not used as deployment benchmarks. Confirm three things before deploying

Model source

The official model ID is:

1

Qwen/Qwen3.6-35B-A3B

When downloading GGUF, you must also check whether the conversion warehouse is clearly marked:

  • Basic model commit;
  • llama.cpp converted version;
  • Quantitative methods and fragmentation list;
  • Whether to provide mmproj that matches the model;
  • File checksum or Hugging Face LFS metadata.

Only the warehouse name contains Qwen3.6, which is not enough to prove that it is consistent with the official weight. Available memory

The GGUF file size is close to the lower limit of weight loading and is not equal to the final VRAM usage. Actual operation also requires space for KV cache, calculation buffer, visual projection and runtime.

For a single 24GB card, Q4 quantization is usually a more reasonable starting point; 16GB or 12GB of VRAM often requires lower quantization, CPU/RAM tiered loading, or shorter contexts. This is an engineering estimate, not the actual measurement conclusion of this site on each type of graphics card. Inference backend

Record the version first to avoid “the same command behaves inconsistently in different versions” but cannot be located:

1
2

.\llama-server.exe --version
nvidia-smi

The baselines that need to be saved include llama.cpp commit, GPU model and driver, system memory, GGUF file name, context length and KV cache type. How to choose quantization

Goal Try first Main cost
Verify whether it can be loaded first IQ2 / IQ3 The output quality and stability of long tasks are reduced
Quality and capacity trade-off Q4_K_M or equivalent dynamic quantization Requires more VRAM or partial CPU offload
More emphasis on quality Q5 / Q6 Files, VRAM and loading times continue to increase
Comparison accuracy Q8 / FP8 / BF16 Usually beyond the comfort range of ordinary single cards

A3B mainly reduces the computational burden of each token and will not compress the 35B total weight into 3B. The choice of quantization should be based on total file size and runtime reporting. Start llama-server on Windows

Start with text mode, short context, and native listening only. PowerShell’s line continuation character is a backtick, not CMD’s ^:

1
2
3
4
5
6
7
8

.\llama-server.exe `
  -m "D:\models\Qwen3.6-35B-A3B-Q4_K_M.gguf" `
  --host 127.0.0.1 `
  --port 8080 `
  -c 8192 `
  -n 2048 `
  -ngl 999 `
  --jinja

parameter function:

  • -m: Main model GGUF; shard model should point to the first shard.
  • -c 8192: Use 8K context verification first to avoid initially pulling the KV cache to 128K.
  • -ngl 999: Request to put as much as possible into the GPU; whether all the data will eventually be put into the GPU depends on the log.
  • --jinja: Use model chat templates to avoid splicing messages directly.
  • --host 127.0.0.1: Only accept local access to prevent unauthenticated services from being exposed to the network.

If the startup fails, first lower -ngl to confirm whether there is insufficient VRAM; if it still fails, check the model file, backend and driver instead of continuing to lower the quantization and perform a blind test. Determine whether it is successful from the startup log

Keep the startup terminal output and check three types of information:

  1. Whether the model metadata and architecture are recognized as Qwen3.6 MoE;
  2. How many layers or tensors go into the GPU;
  3. Whether the KV cache and calculation buffer occupancy is within expectations.

If the process exits directly and memory allocation fails, reduce -c or GPU offload first. Only when the small context cannot be loaded does it mean that the weight itself has exceeded the capacity of the current combination. Verify local API

After the service starts, first verify the health status:

1

Invoke-RestMethod http://127.0.0.1:8080/health

Check the model endpoint again:

1
2

Invoke-RestMethod http://127.0.0.1:8080/v1/models |
  ConvertTo-Json -Depth 6

finally issued the minimum chat request:

The acceptance criteria for

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$body = @{
  model = "Qwen3.6-35B-A3B"
  messages = @(
    @{ role = "user"; content = "只回复 READY" }
  )
  temperature = 0
  max_tokens = 16
} | ConvertTo-Json -Depth 6

Invoke-RestMethod `
  -Uri http://127.0.0.1:8080/v1/chat/completions `
  -Method Post `
  -ContentType 'application/json' `
  -Body $body

are not that READY must be returned verbatim, but that the HTTP request is successful, there is an assistant message in the return structure, there is no template in the terminal, or there is a parsing error. Load visual projection file

After confirming that the text request is normal, add mmproj matching the same model:

When the

1
2
3
4
5
6
7
8

.\llama-server.exe `
  -m "D:\models\Qwen3.6-35B-A3B-Q4_K_M.gguf" `
  --mmproj "D:\models\mmproj-Qwen3.6-35B-A3B.gguf" `
  --host 127.0.0.1 `
  --port 8080 `
  -c 8192 `
  -ngl 999 `
  --jinja

main model and mmproj do not match, common results are that dimension errors are reported at startup, the image request fails, or the model ignores the image completely. Visual functions should be accepted separately. Don’t assume that multimodality has taken effect just because text chat is normal. How to record VRAM and speed

During inference, open another terminal and observe once every second:

1
2

nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu `
  --format=csv -l 1

records at least two sets of data:

  • VRAM when the model has just been loaded and has not yet been requested;
  • Peak memory and speed when generating fixed prompts and fixed output lengths.

A comparable record should be written as:

1
2
3
4
5
6
7
8
9

GPU：具体型号与显存
后端：llama.cpp commit
模型：完整 GGUF 文件名
上下文：8192
KV cache：默认或具体量化
GPU offload：日志中的实际结果
提示词：固定测试文本
输出：固定上限
结果：加载显存、峰值显存、prompt tok/s、generation tok/s

Without these conditions, it cannot be reproduced by just writing “24GB can run” or “very fast”. Common failures and recovery

Startup is OOM

Reduce context, concurrency, and GPU offload in that order; close other GPU-intensive programs. If it still fails, change to smaller quantization, or allow more weights to enter system memory. Duplicate output and confusing roles

Make sure to use the latest version of llama.cpp, retain --jinja, and check that the GGUF contains the correct chat template. Do not duplicate templates on the client and server at the same time. 404 after connecting to the client

llama-server provides an OpenAI-compatible Chat Completions interface. If the client only supports Responses API, it cannot only modify base_url; a protocol conversion layer is required. This issue has nothing to do with whether the model loaded successfully. Exit and rollback

Press Ctrl+C in the service terminal to stop the process. Do not register as a startup service in the early stages of deployment; first save a baseline command that can start stably, and then add context, vision, and remote access one by one. Security Boundary

  • By default, only 127.0.0.1 is monitored.
  • Do not expose unauthenticated inference ports directly to the public network.
  • Agent uses least privileges and manual confirmation when accessing files, shells or browsers.
  • Third-party fine-tuning of weights requires separate verification of sources, licenses, data and behavioral differences.

- Local running only addresses part of the data transfer path and does not automatically guarantee correct output or system security. Conclusion

Local deployment of Qwen3.6-35B-A3B should start with official model identity, short context text service and API verification, and then gradually add GPU offload, visual projection and Agent access. MoE can reduce the amount of calculation, but the complete weight, KV cache and runtime overhead still determine whether the machine can run stably. References