Best Local LLMs for an RTX 3060 12GB: Quantization, Context, and Benchmarks

This recommendation is based on Desktop RTX 3060 12GB. NVIDIA also has an 8GB RTX 3060 model, and the power consumption, heat dissipation, and VRAM of the notebook version are also different; if your card is not 12GB, the conclusions of this article cannot be directly applied.

The most comfortable range for 12GB of VRAM is usually the Q4/Q5 quantization of the 7B–9B models. You can try Q4 for 12B-level models, but you need to control the context; larger 20B and 32B models often require CPU offload, and being able to load does not mean a good experience.

Confirm the hardware first

The

1

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

output should clearly show the GPU name, total VRAM, and driver. If it is the 8GB version, give priority to 7B/8B Q4 and shorten the context.

Recommended model tiers

Model versions are updated quickly, and the specific name should be confirmed from the developer’s official organization or model card. Don’t download “optimized versions” with similar names but unknown sources.

How to choose quantization

• Q4_K_M: Balanced quality, speed and capacity, suitable as a first test;
• Q5_K_M: The file is larger, the quality is usually more stable, 8B level can often be tried on 12GB;
• Q6_K / Q8_0: takes up more VRAM and is suitable for smaller models or short contexts;
• Q2 / Q3: The capacity pressure is small, but the quality loss must be verified by your own tasks.

The GGUF file size is just the starting point for weight occupancy. KV cache, run buffers, and desktop programs also use VRAM, so don’t download a file approaching 12GB and expect full GPU usage.

Budget for 12GB of VRAM

Desktop, browser, video playback and development tools will first occupy part of the VRAM. The available space of the model should be calculated based on the free VRAM before startup, not the nominal 12GB of the graphics card.

1

nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

can divide the budget into:

If you only have 10GB left before booting, don’t press the “12GB Graphics Card Recommendation Table” to select a model file close to 12GB.

Validate the driver and CUDA backend first

Before testing the model, first confirm that the driver can report the graphics card normally:

The

1

nvidia-smi

llama.cpp boot log should show the CUDA backend and report GPU offload; Ollama can check PROCESSOR via ollama ps. If the 3B small model all shows 100% CPU, solve the driver or backend problem first, and then discuss the 8B and 12B performance.

Don’t assume that the application must call the GPU just because the complete CUDA Toolkit is installed. The precompiled program may come with its own runtime library, but the real basis for judgment is still logs, processes, and VRAM changes.

Select by task, not by list

Chinese writing and knowledge work

Test the Qwen 7B/8B level command model first. Focus on checking Chinese expression, factual fidelity, long summary and format compliance. If Q5 can still fully host the GPU, it can be compared with Q4 on the same question.

Local code assistant

Prepare a small real repository and test reading multiple files, generating patches, fixing unit tests and adhering to project constraints. Only letting the model write one function cannot reflect multi-file code tasks.Code scenes also check the context: 8B Q5 may have better quality, but if the VRAM is not enough to retain the required code context, the actual effect may not be as good as Q4.

RAG and Documentation Q&A

RAG requires both embedding models, vector libraries, and generative models. Do not default to all 12GB available to LLM. You can put the embedding on the CPU, or choose a smaller generative model to leave room for retrieval results and KV cache.

During the test, the number of input document tokens, the number of recalled paragraphs, and whether the answer citations are correct are recorded.

Image understanding

Confirm that the model, visual projection, and backend match. When RTX 3060 processes multiple large images, the visual token and projection buffer will increase the VRAM. Start with a single zoomed image instead of bulk uploading the original photos.

Automation Agent

Agent pays more attention to tool calling format, low latency and stability, and does not necessarily require the maximum parameters. If the 3B–8B model can stably output legal JSON, it may be more suitable than the 12B/20B model that requires CPU offload.

How to manage model files

Different tools may download the same model repeatedly. It is recommended to create a clear directory and save the sources:

1
2
3
4
5
6
7
8
9

D:\Models\
  qwen\
    8b\
      model-q4_k_m.gguf
      SHA256.txt
      source-url.txt
  gemma\
    12b\
      model-q4_k_m.gguf

save hash:

1
2

Get-FileHash 'D:\Models\qwen\8b\model-q4_k_m.gguf' -Algorithm SHA256 |
  Format-List

Don’t just name multiple files with model.gguf, otherwise you will quickly become confused about the model, quantification, and source.

Create an everyday 8B configuration

llama.cpp can start with conservative parameters:

1
2
3
4
5
6
7

.\llama-server.exe `
  -m 'D:\Models\qwen\8b\model-q5_k_m.gguf' `
  -ngl 999 `
  -c 4096 `
  -np 1 `
  --host 127.0.0.1 `
  --port 8080

acceptance model list:

1

curl.exe http://127.0.0.1:8080/v1/models

If the service is only used locally, keep 127.0.0.1. Do not directly monitor public network addresses for mobile phone access; firewall restrictions and authentication proxies should also be added to LAN sharing.

Do a baseline before testing 12B

Complete the baseline of 8B Q5 first, then switch to 12B Q4. Fixed:

• Backend version;
• 4K context;
• Single concurrency;
• The same test set;
• Same sampling parameters;
• Same drive and power mode.

This will answer the question “Is the 12B quality improvement worth the memory and speed costs?” If you upgrade the backend and change the context at the same time, the conclusions will not be comparable.

Use PowerShell to record VRAM time series

Data can be written to a file during continuous observation:

After

1
2
3
4
5

nvidia-smi `
  --query-gpu=timestamp,name,memory.used,utilization.gpu,temperature.gpu,power.draw `
  --format=csv `
  -l 1 `
  -f .\rtx3060-llm-test.csv

completes the test, press Ctrl+C to stop. CSV can see model loading peaks, build stage utilization, temperature and power consumption, rather than just a snapshot of the moment.

Temperature, power consumption and sustained speed

The RTX 3060 is very fast in the short term, but that doesn’t mean it’s still stable after running for an hour. The test lasts at least 15–30 minutes, observing:

• Whether the temperature continues to rise;
• Whether the GPU clock drops due to temperature or power consumption;
• Is the fan noise acceptable?
• Whether the power supply is stable;
• Whether tokens/s decay over time.

Don’t run the graphics card at abnormal temperatures for a long time just for a small amount of speed. Prioritize improving chassis air ducts, cleaning dust, and using reasonable power settings; understand the hardware risks before modifying voltage or power consumption limits.

How system memory affects CPU offload

When some weight goes to the CPU, system memory capacity and bandwidth directly affect speed. It is recommended to use at least dual-channel memory and ensure that the model is run without heavy page file activity.

Check memory pressure:

1

Get-Counter '\Memory\Available MBytes','\Paging File(_Total)\% Usage'

If the available memory is nearly exhausted and the page file continues to grow, you should stop testing and use a smaller model or low quantization. Increasing the page file may avoid crashes, but will not restore inference to normal speeds.

How to design the evaluation form

Fill in for each candidate model:

Finally, filter based on task pass rate, instead of directly setting the model with the highest tokens/s as the default.

When should you upgrade hardware?

Consider changing the graphics card or increasing the memory in the following situations:

• Real missions must use Class 20B/32B models;
• Contexts above 8K are fixed requirements;
• Requires multiple concurrent users;
• The visual model and the text model must reside at the same time; -CPU offload has become the main source of latency;
• Production services require larger stability margins.If you only need a larger model occasionally, it may be cheaper to use the official API on a pay-per-use basis than to purchase the hardware. Graphics cards, power supplies, cooling, idle time and maintenance costs should be compared.

Privacy boundaries for local services

Not running locally automatically means that the data will never be sent out. The model manager, frontend, plug-in, or telemetry component may still be online. When sensitive data needs to be processed:

• Check download and update sources;
• Check the network requests of front-end and plug-ins;
• The local API only listens to the loopback address;
• Do not synchronize chat logs to unknown cloud;
• Regularly update the backend to fix security issues;
• Conduct review of personal information in model output.

When offline requirements are strict, a complete test should be completed in a disconnected environment instead of just closing the browser.

Ollama Quick Test

Take the 8B tag that actually exists in the model library as an example:

Check after running

1

ollama run qwen3:8b

:

Among

1
2

ollama ps
nvidia-smi

ollama ps, PROCESSOR is close to 100% GPU, indicating that the model is mainly on the graphics card; the presence of CPU/GPU hybrid indicates that partial offload has occurred. Possible changes in model labels and their quantification should be confirmed on the model library details page, rather than just looking at the parameter amounts in the labels.

llama.cpp controlled test

Using a GGUF from a trusted source, start with the 4K context in PowerShell:

1
2
3
4
5
6
7

.\llama-server.exe `
  -m .\model-q4_k_m.gguf `
  -ngl 999 `
  -c 4096 `
  -np 1 `
  --host 127.0.0.1 `
  --port 8080

Open another terminal and continue to observe:

1

nvidia-smi -l 1

logs should show the CUDA backend, offload layers, model buffer and KV cache. Record the first token delay and tokens/s generated in the first round, and then use the same prompt word to compare other quantifications.

A fair set of comparison conditions

Fixed when comparing models:

• Same RTX 3060, driver and system power modes;
• The same backend version;
• same context (first 4096, then 8192);
• Single concurrency;
• Same temperature, seed and maximum output length;
• Each model uses its own official chat template;
• Turn off games, video enhancements, and other reasoning programs that consume VRAM.

It is recommended to test the tasks you can actually do: Chinese summarization, code modification, structured JSON, long text retrieval and tool invocation. The public list can only filter candidates and cannot replace the native workflow.

12GB VRAM common failures and recovery

CUDA OOM

Lower the context, concurrency, and quantization size in that order. If it still fails, reduce GPU offload and let some weight go into system memory.

Can run but very slowly

Check the ollama ps or llama.cpp log. If a lot of weight is placed on the CPU, the bottleneck is usually system memory bandwidth and PCIe transfers. Instead of forcing a 32B run, I’d rather switch to a better quality 8B/12B model.

The VRAM is enough but the output is abnormal.

Check whether the model is the command version, whether the chat template is correct, and whether the backend supports the architecture. Enough VRAM does not mean format compatibility.

Crash after long conversation

Context growth will expand the KV cache. Limit the maximum context, restart the service release model, and retest peak memory.

Qwen3 quantization choices for RTX 3060

Let’s look at the conclusion first: Which one should I choose, 3060 12GB?

If you only want to download one version and don’t want to go through repeated troubles, select Qwen3-8B Q6_K.

Why not just go to 14B or 30B-A3B?

Qwen3-8B has about 8.2B parameters, the official native context is 32K, and can be extended to longer contexts through YaRN. For RTX 3060 12GB, the key advantage of the 8B model is not “the strongest”, but the ability to put the model, runtime overhead and part of the KV cache into the video memory.

Qwen3-14B Q4_K_M has higher quality potential, but the quantized file itself already squeezes 12GB noticeably. Even if the model loads, long prompts, thinking modes, long output, or larger contexts are more likely to strain video memory. It’s more suitable for people who are willing to sacrifice context and speed for single-round answer quality.

Qwen3-30B-A3B is a MoE model with fewer parameters per activation, but full weights still need to be loaded. MoE can reduce some of the computing pressure and cannot turn dozens of GB model files into 12GB video memory models. On the 3060, you can use CPU memory with some GPU offloading to conduct experiments, but the speed, memory usage, and parameter adjustment complexity will increase.

Therefore, the “best quantized version” is not the version with the largest file size, nor the version with the most parameters, but the version that can run stably under your common context length, has sufficient output quality, and is less prone to OOM.

How to choose between Q6_K, Q5_K_M and Q8_0

Three common options can be understood as:

Quantization selection cannot be understood only in terms of “the higher the bit width, the better”. For local inference, the memory margin will directly affect the context length, batch processing, first token delay and operational stability. Q8_0 If you are forced to lower the context very low, the actual experience may not be better than Q6_K.

It is recommended to test in the following order first:

1. With Q6_K, the context is set to 8192.
2. Observe the memory usage, generation speed and stability.
3. When often dealing with long codes and long documents, change to Q5_K_M and compare again.
4. If you only want to do a short Q&A and there is still memory left, try Q8_0 again.

llama.cpp recommended configuration

Qwen officially recommends using the newer llama.cpp for full Qwen3 support. Here’s a practical starting point for the RTX 3060 12GB:

1
2
3
4
5
6
7
8
9

./llama-cli \
  -hf Qwen/Qwen3-8B-GGUF:Q6_K \
  --jinja \
  -ngl 99 \
  -c 8192 \
  -n 1024 \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95

The meaning of several parameters:

• -ngl 99: Try to put the offloadable layer on the GPU. If the video memory is insufficient or startup fails, gradually reduce it.
• -c 8192: Start with 8K context first, don’t set 32K at the beginning.
• -n 1024: Limit the length of a single generation to prevent long output from continuously occupying resources.
• --jinja: Organize input according to model chat template. Qwen3 does not recommend handwriting a random format.

When you want to do services, you can use:

1
2
3
4
5
6

./llama-server \
  -hf Qwen/Qwen3-8B-GGUF:Q6_K \
  --jinja \
  -ngl 99 \
  -c 8192 \
  --port 8080

After starting, look at nvidia-smi first. If the video memory is nearly full, the system response becomes slow, or an error is reported when the prompt is long for the first time, first reduce the context or switch to Q5_K_M, and do not blindly continue to add layers.

How do Ollama users choose?

Ollama can be run directly:

1

ollama run qwen3:8b

It is more suitable for people who want to use it quickly and don’t want to deal with GGUF files. But there are two points to note:

1. The actual corresponding quantified version behind the label may be updated with the warehouse, and you cannot infer which GGUF it must be just based on qwen3:8b.
2. Ollama’s default context settings may not be suitable for your task. When long context is required, num_ctx should be adjusted explicitly and attention should be paid to video memory changes.

If you want precise control over Q5_K_M, Q6_K, or Q8_0, it’s often more intuitive to import GGUFs in llama.cpp, LM Studio, or manually.

How to downgrade RTX 3060 Laptop 6GB/8GB

The video memory of notebook RTX 3060 is usually 6GB or 8GB, and the 12GB conclusion cannot be copied.

Notebooks also need to consider power consumption and heat dissipation. Even if the video memory is the same, the continuous generation speed may be significantly lower than that of a desktop card; first run with a short prompt for 10 to 20 minutes, and then judge whether the configuration is really suitable for daily use.

Don’t ignore KV cache and thinking patterns

Just because the model file can be put into the video memory does not mean that the real task will run smoothly. Qwen3’s context, historical conversations and generated content will all form a KV cache; the longer the context, the higher the graphics memory usage.

Especially for the following types of tasks, it is recommended to give priority to Q5_K_M or reduce -c:

• Post multiple source codes, logs or long documents at one time;
• long continuous conversations;
• Enable think mode and allow very long output;
• The local API serves multiple requests simultaneously.

The 3060 12GB is more suitable for single-user, short to medium context local assistants. If the target is more than 32K contexts, multi-person concurrency or large-scale RAG, it is usually more time-saving to upgrade the graphics memory or switch to cloud inference than to continue compression.

Record these four items during the actual measurement

Don’t just look at tokens/s. Test with the same prompt word and record:

Q6_K that can stably complete your common tasks is usually more worthy of long-term retention than Q8_0 that occasionally has slightly higher quality but frequently bursts the video memory.

Selection conclusion

• For stable daily use: 8B Q4_K_M/Q5_K_M;
• Want to improve quality: test 12B Q4 and keep 4K–8K context;
• Want to increase speed: choose 3B–4B model or shorter context;
• Want to run 20B/32B: accept the obvious CPU offload and speed drop first, and then decide whether it is worth it;
• Production services are required: in addition to VRAM, concurrency, long-term stability and fault recovery must also be tested.

Benchmark record template

1
2
3
4
5
6
7
8
9

GPU：RTX 3060 12GB / 驱动版本
后端：Ollama 或 llama.cpp 版本
模型与来源：
量化与文件 SHA256：
上下文 / 并发：
GPU offload：
峰值显存：
首 token 延迟 / tokens/s：
任务准确性：

References

• NVIDIA RTX 3060 official specifications page
• Ollama GPU support
• llama.cpp official warehouse
• Qwen Official Hugging Face