GPU VRAM on KnightLi Blog

How to calculate the cost of running Agent on a consumer-grade graphics card: electricity bill, depreciation and cost per task

Sat, 11 Jul 2026 10:30:00 +0800

If you want to know whether a consumer-grade graphics card is worth running a local agent, you can’t just look at “how much the graphics card cost”, nor can you just compare the unit price per million tokens of the API. Agents will repeatedly call models, tools, and browsers; long contexts, failed retries, and idle standby will all change the actual cost.

The most practical way is to split the cost into four items: electricity, hardware depreciation, supporting equipment and manual maintenance, and then calculate it according to “each successful task” and “per million output tokens” at the same time.

First determine which costs you want to calculate

caliber	Questions suitable to be answered	Items that are easily missed
incremental cost	I already have a game console. How much does it cost to run Agent in my spare time?	The graphics card is already there, is it included in the depreciation?
full cost	Is it cost-effective to purchase a machine specifically for local agents?	Host, memory, SSD, cooling and depreciation
Cost per task	How much does it cost to automate a document and a code task?	Failure, retry, and manual review
throughput cost	What is the cost per million tokens to build a self-built inference service?	Enter token, KV cache and idle time

If the graphics card was originally used for gaming or other work, it would be more honest to calculate the incremental cost first; only when the entire machine is purchased for Agent, the depreciation of the entire machine is fully included.

Total monthly cost formula

Let’s use monthly caliber first, the data is easiest to obtain:

月度总成本 = 电费 + 硬件月折旧 + 配套服务费 + 人工维护成本

电费 = 实测整机平均功耗(kW) × 实际运行小时数 × 电价(元/kWh)
硬件月折旧 = (购入总价 - 预估残值) ÷ 使用月数
单次成功任务成本 = 月度总成本 ÷ 月成功任务数
每百万输出 token 成本 = 月度总成本 ÷ 月输出 token 数 × 1,000,000

The “average power consumption of the entire machine” should come from a socket power meter or UPS as much as possible, rather than just looking at the graphics card’s nominal TBP. The Agent also runs with CPU, memory, SSD, fans, and monitors; using only GPU power consumption will underestimate the electricity bill. Conversely, periods when the machine is on standby but not processing tasks should not be mixed into pure inference costs and is best recorded separately.

An example of replaceable parameters

Assume that a machine has a measured average power consumption of 280W under the actual Agent workload, runs 6 hours a day, and 30 days a month; the local electricity price is calculated as 0.8 yuan/kWh. The hardware purchased specifically for this purpose is amortized over 7,000 yuan, 36 months, and the residual value is ignored.

1
2
3

电费 = 0.28 × 6 × 30 × 0.8 = 40.32 元/月
硬件折旧 = 7,000 ÷ 36 = 194.44 元/月
月度基础成本 = 234.76 元/月

If 600 accepted tasks are completed this month, the basic cost will be approximately 0.39 yuan/task. The “completion” here cannot only be counted as model return; it should be based on business results such as successful script execution, work order closing, and manual inspection passing.

This is just an example calculation, not a universal price. It only makes sense to replace power consumption, usage time, electricity price, purchase amount and task volume with your own data.

Why is Agent more difficult to estimate than regular chat?

Chat models are often billed by token, and the cost of Agent is also affected by the execution path:

A task may include planning, retrieval, calling tools, reading results, and multiple rounds of corrections.
The longer the context, the more KV cache is occupied; when there is insufficient video memory, it will slow down, unload the CPU/RAM, and even trigger retries.
Browser automation, code testing, and file processing can stretch task times even if the GPU is not constantly fully loaded.
Multi-Agent parallelization will increase throughput, but may also increase memory contention, queuing, and failure rates.
In order to reduce the first token delay, resident model will increase idle power consumption.

Therefore, do not use the tokens/s of a short question and answer to represent the Agent cost. Select at least a set of real tasks, run them continuously for a week, and record input/output tokens, success rate, number of retries, wall clock time, and overall machine power consumption.

Two units that must be seen at the same time

every successful task

This is closest to business decisions. It is suitable for comparing fixed processes such as “local agent automatically processes PR”, “batch data collection”, “customer service draft generation”, etc.

`1`	`每个成功任务成本 = (本周期全部成本) ÷ 成功且验收通过的任务数`

Failed tasks cannot be deleted from the numerator. The power, inference time, and manual troubleshooting they consume are the true operating costs of local systems.

per million tokens

This is suitable for comparing on-premises inference services and cloud APIs, but distinguishing between inputs and outputs. The input of many agents includes long tool logs and historical context, and the input tokens are much more than the output tokens; if divided by the output tokens only, the number will be high. A more reliable approach is to record input, output and total tokens at the same time, and maintain the same statistical caliber.

How to collect your own data

There is no need to build a complex monitoring system from the beginning. First create a table and record the following fields for each run:

Field	Recording method
start and end time	Agent log or task queue
Whole machine power consumption	Smart outlet, power meter or UPS
input/output token	Inference service logs or client statistics
Success, failure, retry	Task status and error log
Manual review time	Work order, PR or random inspection record
Model and quantified version	Avoid mixing different models in the same set of data

Start with 20 to 50 representative tasks as a baseline. Then, tasks with short contexts, long contexts, many tool calls, and many retries are tested separately to see the impact of memory and model selection on costs. Video memory planning can be judged in conjunction with Local Model Quantification and Video Memory Selection.

How to calculate depreciation so as not to mislead yourself

There is no one answer to depreciation, but the assumptions must be written down first.

Existing graphics card: You can only calculate the electricity bill and additional maintenance as incremental costs; you can also set an opportunity cost for the graphics card as a more conservative full cost.
New equipment purchased: Graphics card, host, memory, SSD, and radiator should be included together; don’t just spread the price of the graphics card.
Lifespan: 24, 36 or 48 months will work, the key is comparing on-premises vs. cloud using the same assumptions.
Residual value: When the equipment is expected to be resold, the estimated residual value can be deducted from the purchase price; if it is uncertain, set it to 0, and the conclusion is more conservative.

NAS or small hosts usually consume less power, but memory, cooling, and expansion capabilities also limit models and concurrency. If you consider this route, you can first read Performance Judgment of NAS Deployment Ollama.

When comparing to cloud APIs, don’t just compare unit price

The cloud side can also be converted into cost per task:

`1`	`云端每任务成本 = 模型 API 费用 + 工具/API 费用 + 必要的存储与网络费用 + 人工复核成本`

Local is usually better suited for these situations:

There is a stable and predictable workload every day, and machine utilization is high.
Data cannot or is not expected to be sent to external services.
Already have a working GPU, the incremental cost is mainly electricity.
Self-maintainable models, drivers, service and crash recovery are acceptable.

The cloud is often better suited for these situations:

The load is sporadic and the machine will be idle most of the time.
Tasks require large models, very long contexts, or peak concurrency that far exceed local video memory.
The team is unwilling to take on the driver, model upgrades, monitoring and troubleshooting.
The task results are of high value, and the strongest model or multi-modal capabilities are required first.

Local deployment is not just a choice between two options. Stable, privacy-sensitive steps are placed locally; complex, low-frequency steps or steps that require stronger models are called to the cloud, which is often easier to control the total cost.

A practical decision line

Record the local monthly fixed cost as F, the variable power and maintenance costs per task as v, and the cloud per-task cost as c. When c > v, the approximate amount of tasks required to achieve local breakeven is:

`1`	`每月盈亏平衡任务数 = F ÷ (c - v)`

If c is less than or close to v, local will not be significantly cheaper due to scale; privacy, latency, controllability, and offline capabilities should be considered more at this time. This formula is only a financial filter and still requires failure rates and manual review time.

Common misunderstandings

Only look at the power consumption of the graphics card: The power, standby and cooling of the entire machine also need to be paid.
Think of nominal tokens/s as throughput: Real agents will be slowed down by context, tools, network, and test steps.
Ignore failed tasks: The process that is retried most frequently is often a cost black hole.
Only model charges are calculated: Browser, search, vector library, storage and human review charges may also apply.
Mixing data from different models: The quantization version, context length, and concurrency are different, and the results cannot be compared horizontally.

Summarize

The cost of running Agent on a consumer-grade graphics card should not just be answered by “a few kilowatt-hours of electricity per hour.” First use the measured power consumption of the entire machine to calculate the electricity cost, then add the hardware cost according to your own depreciation assumptions, and finally use the number of successful tasks and the amount of tokens to amortize it. After continuously recording a period of real workload, you can clearly judge whether to continue running locally, expand the video memory, or hand over some steps to the cloud API.

Whether the Ollama performance of NAS deployment is insufficient: how to judge the CPU, memory and graphics card

Sat, 11 Jul 2026 09:40:43 +0800

NAS can deploy Ollama, but “can be installed” and “quick enough to use” are two different things.

If the goal is to make occasional summaries, file Q&A, home automation and low-frequency API calls, x86 NAS can use CPU to run small models; if you want to have continuous conversations like chat software, run 7B/8B code models, or make multiple service calls, the key becomes video memory, memory and heat dissipation, not how many TB the hard drive has.

Let’s give the conclusion first:

**NAS without available GPU is suitable for small models and low-frequency tasks; only NVIDIA GPUs correctly recognized by Ollama are more suitable for using 7B/8B models as daily services. **

First confirm which category your NAS belongs to

Don’t just look at “AI chip”, “NPU” or “4K transcoding” on the product promotion page. Whether Ollama can accelerate depends on whether the system, driver, container permissions and hardware are recognized in the actual running environment.

NAS status	suitability	More realistic uses
ARM CPU, 8GB RAM, no available GPU	limited	Minimal model, simple classification, offline tasks
x86 CPU, 8GB RAM, no available GPU	Barely usable	1B–3B Quantitative model, low-frequency text processing
x86 CPU, 16GB–32GB memory, no available GPU	testable	3B–7B Low quantization model, but generally slower response
x86 NAS + 8GB NVIDIA video memory	Available every day	7B/8B quantitative chat, lightweight code assistance, single-user API
x86 NAS + 12GB or above NVIDIA video memory	more suitable	The 8B model is more comfortable, and some 14B quantitative models can be chosen based on context.

The table is a starting point for deployment judgment, not a performance promise. CPU model, memory channels, model format, context length, number of concurrencies, NAS system limitations, and thermal dissipation will vary actual results.

Let’s first look at “storage” and “reasoning” separately.

The large hard disk of NAS mainly solves the problem of model file storage: you can save many GGUF, Ollama model layers and document libraries. The inference speed is mainly determined by the following items:

CPU core performance and memory bandwidth;
Whether the system memory is sufficient to install the model and runtime;
Whether the GPU is available and whether the video memory can accommodate the model and context;
Whether PCIe lanes, drivers and containers are transparently transmitted correctly;
Context length, model quantization, concurrency and output length.

So the answer to “Can a NAS run a 32B model if it has a 40TB hard drive?” is usually: files can certainly be placed, but they may not be able to be inferred at an acceptable speed.

CPU-only NAS: When is it worth deploying?

It is possible to run Ollama without a GPU. The official Docker image supports pure CPU boot:

docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

CPU-only suitable for:

Nightly batch summarization, document classification, tag extraction;
low-frequency text tasks in home automation;
Validate API calls, RAG pipelines, and permissions logic;
Personal tools that are insensitive to initial token latency and generation speed.

Not very suitable for:

Multi-person chat service;
Long time code Agent;
Large models are generated continuously;
Real-time voice assistant or high-frequency web Q&A.

The correct strategy for CPU-only is to use a small model first, a short context, and limit concurrency. Don’t download the largest model first and then expect to get a cloud-like experience by “relying on the NAS and calculating slowly”.

How much memory should be reserved at least?

System memory is not only used for model weights, but also for the NAS itself, file cache, Docker, download tasks, photo index, containers and KV Cache.

A practical principle: **Don’t let Ollama eat up the NAS memory until it starts swapping a lot. ** Once swap intervenes frequently, the response time will significantly deteriorate and file sharing and other services will also be affected.

Memory	suggestion
8GB	Only small models and lightweight testing are recommended; be more conservative when the NAS is running multiple services.
16 GB	Can be used as a starting point for CPU-only local assistants, prioritizing small models or low-quantized versions
32GB	More suitable for the coexistence of 7B-level low-quantity models, RAG and multiple lightweight containers
64GB+	Good for CPU/hybrid offloading of larger models and long documents, but does not equate to fast enough generation

Model file size is only a lower limit. Model loading, context, and runtime will continue to take up memory, so don’t rely on “file 8GB, machine 8GB” to judge whether it can run.

The biggest difference in experience is seen with NVIDIA GPUs

If the NAS is x86 Linux and has a supported NVIDIA graphics card, Ollama can use GPU acceleration. Ollama officially lists the support range of NVIDIA compute capability 5.0+ and driver 531+, among which the RTX 30 series is also included in the support list.

The Docker solution requires the host to first configure the NVIDIA Container Toolkit, and then use the GPU parameters to start the container:

docker run -d \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

After the container is started, verify the GPU transparent transmission itself instead of directly downloading the large model:

`1`	`docker run --gpus all ubuntu nvidia-smi`

When this command does not display the graphics card properly, the Ollama container cannot use the GPU. At this time, you should first check whether the driver, Container Toolkit, Docker runtime, and NAS system support PCIe GPU transparent transmission.

Can the core graphics, NPU and transcoding capabilities of NAS be used directly?

This cannot be assumed by default.

The core graphics of many NAS are mainly used for video decoding, transcoding or graphics output; the NPU advertised by some devices may only be open to the manufacturer’s own applications. Whether they can be used by Ollama depends on the backend, operating system, driver and device permissions supported by Ollama.

Actual evidence should be seen before deployment:

Whether the host can see the GPU device;
Whether nvidia-smi or the corresponding manufacturer’s detection tool can be run in the container;
Do Ollama logs clearly show GPU usage;
Under the same prompt, is the speed difference between GPU and CPU reasonable?

When you don’t see this evidence, plan according to CPU-only expectations. Don’t purchase a model based on discrete graphics performance just because the device has a “transcoding engine”.

How to choose a model so as not to bring down the NAS

The core of NAS deployment is not to pursue the largest model, but to let common tasks be stably completed first.

scene	Model selection ideas
Family document summary, classification, simple questions and answers	Try 1B–4B models first
Chinese chat, lightweight scripting, single-user API	You can try the 7B/8B quantization model when you have 8GB of video memory.
Long code, complex Agent	Prioritize larger memory hosts or clouds. It is not recommended to rely solely on ordinary NAS CPUs.
Embedding/RAG	Embedding models are usually lighter and suitable for NAS; generated models are called on demand
Called by multiple people at the same time	Do queue and concurrency limits first, don’t let all requests directly load large models in parallel

Once the model is too large, the common symptoms of NAS are not immediate failure, but long loading, slow first token generation, low generation speed, and continuous increase in system memory, which ultimately affects the stability of file services and containers.

Recommended deployment structure

For most families or small teams, it is recommended to use NAS as a “data and light reasoning node”:

NAS
├─ Ollama：小模型、Embedding、低频 API
├─ 文档与向量库：私有文件、备份、RAG 数据
├─ 反向代理：局域网访问控制
└─ 监控：CPU、内存、GPU、容器日志

高显存主机或云端
└─ 大模型、长上下文、复杂 Agent、高并发推理

This division of labor is more stable than forcing all inference into the NAS: sensitive files remain local, and heavy tasks are routed to more appropriate devices.

Five-minute test before deployment

Don’t download dozens of GB models first. First judge according to this process:

Confirm NAS architecture, Docker support, and remaining memory.
Start the Ollama container and confirm that http://NAS地址:11434 is accessible.
Pull up a small model and ask 5 to 10 fixed questions in a row.
Observe CPU, memory, swap, disk IO and GPU usage.
Then switch to a model that is close to real requirements, and gradually increase context and concurrency.

You can use the following command to view container resources:

`1`	`docker stats ollama`

If you have an NVIDIA GPU, observe at the same time:

`1`	`nvidia-smi`

As long as file sharing, photo serving, or backup tasks are noticeably slower when running a model, the current model or concurrency is not suitable for NAS, and the model size should be reduced or the inference should be moved to other machines.

Don’t ignore network and security

Ollama’s default local service port is 11434. If you want other devices on the LAN to call it, first implement access control at the NAS firewall or reverse proxy layer, and do not directly expose the unauthenticated inference port to the public network.

Especially when there are family photos, backups, documents and private files in the NAS, the model service, file service and management backend should use different permission boundaries. For calls that can be completed on the LAN, there is no need to open the public network port.

Summarize

Whether Ollama is sufficient for NAS deployment can be judged as follows:

只做摘要、分类、Embedding
-> CPU-only NAS 可以尝试

想要 7B/8B 模型的日常交互体验
-> 需要可用 GPU，优先看显存

想做长上下文、复杂 Agent 或多人并发
-> 更适合高显存主机或云端推理

First use a small model to verify resources and stability, and then decide whether to add a graphics card, expand memory, or split inference nodes. NAS is great for hosting private data and lightweight model services, but it is not a natural large model server.

refer to:

What to do if vLLM KV Cache has insufficient memory: video memory, context and concurrency troubleshooting

Sat, 11 Jul 2026 09:36:02 +0800

When the following error occurs when vLLM is started, it is usually not because the model weight download is broken:

`1`	`The model's max seq len ... KV cache is needed, which is larger than the available KV cache memory`

It means that after the model weights and runtime overhead take up the video memory, the remaining space is not enough to prepare the KV Cache for the maximum context length you set. The most direct processing sequence is: **Reduce --max-model-len first, then check concurrency, and finally consider improving memory utilization, quantifying KV Cache, or expanding capacity. **

Don’t pull --gpu-memory-utilization to 1 right off the bat. That may allow vLLM to pass initialization, but be more prone to OOM when real requests, CUDA graph, or other processes compete for video memory.

First understand: why KV Cache fills up the video memory

The model weight determines “whether the model can be loaded”, and the KV Cache determines “how many tokens the model can remember at the same time and how many requests it serves.” Every time a token is generated or read, the attention layer needs to save the corresponding Key/Value state.

Therefore, KV Cache usage will increase with the following items:

max_model_len: The longer the maximum context allowed, the greater the cache requirements.
Number of concurrent requests: Each concurrent request will occupy its own context space.
Model structure: number of layers, number of KV heads, hidden size and data type all affect cache size.
Cache precision: The default usually follows the model data type; low-precision caches such as FP8 can save space, but have compatibility and quality boundaries.

The most overlooked thing is concurrency. Even if each request is only given 8K context, the total demand on KV Cache will still grow rapidly when multiple requests are running simultaneously.

First determine what kind of problem it is from the logs

Startup logs usually give three key pieces of information:

The maximum sequence length specified by the model or configured by you.
How many GiB does KV Cache require.
The actual space currently available for the KV Cache, and the maximum feasible length estimated by vLLM.

If the log says “Maximum length 32768 requires 10GiB KV Cache, but currently only 4GiB”, don’t worry about whether the model supports 32K first. Model Support does not mean that your card can serve 32K requests under the current configuration.

Start by taking note of the estimated maximum length provided by the log, and set your first test at around 60% to 80% of it, and then work your way up.

First priority: lower `--max-model-len`

This is the adjustment with the highest success rate and the most predictable impact. For example, the model natively supports 32K, and your actual business only handles ordinary conversations, short codes and small documents, you can start with 8K:

1
2

vllm serve Qwen/Qwen3-8B \
  --max-model-len 8192

If it’s still not enough, try again:

1
2

vllm serve Qwen/Qwen3-8B \
  --max-model-len 4096

Do not write the model card’s nominal 128K, 256K or longer context directly into the service parameters. For single-card deployments, the actual available length depends on weight, video memory, concurrency, and cache accuracy.

A practical starting point:

Video memory and usage	Context length to try first
12GB–16GB single-card, Class 8B model	4096 or 8192
24GB single card, 7B–14B models	8192 or 16384
24GB single card, 30B level quantized model	Start with 4096
Long Document/RAG/Multi-User Service	Make decisions based on log estimates and stress testing, don’t just follow the table

These values are a starting point for troubleshooting, not memory commitments. KV Cache size can vary significantly between models.

Second priority: Limit concurrency and batch processing

Just because the service can be started does not mean it will be stable under high concurrency conditions. First, control concurrency to a small value:

1
2
3

vllm serve Qwen/Qwen3-8B \
  --max-model-len 8192 \
  --max-num-seqs 4

--max-num-seqs limits the number of sequences that can be processed in one iteration. A larger number may result in higher throughput, but also greater KV Cache and scheduling pressure.

If requests often contain long prompts, you should also pay attention to --max-num-batched-tokens. It determines the maximum number of tokens that can be processed at one time; if it is too high, the prefill phase may occupy more resources instantly. Be conservative when troubleshooting, and then gradually expand after confirming stability.

It is recommended to perform stress testing in this order:

Single request, 4K context, confirmed service stability.
Single request, 8K context, observe first token latency and video memory.
Two concurrent requests confirm that there will be no OOM.
Then gradually increase max_num_seqs or the number of batch tokens.

Do not adjust context, concurrency, batch and model quantization at the same time, otherwise it will be difficult to locate which parameter is causing the video memory to be exhausted.

Third priority: Properly set `--gpu-memory-utilization`

--gpu-memory-utilization specifies the proportion of GPU memory that can be used by the current vLLM instance, with a value between 0 and 1. vLLM will plan the weight, runtime and KV Cache based on this part of the space.

For example:

vllm serve Qwen/Qwen3-8B \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 4

When the error message clearly says “Too few available KV Cache” and there are no other processes on the GPU, you can improve it in small steps, such as trying from 0.90 to 0.92 or 0.94. Add just a little at a time and test with real requests.

It is not recommended to deadlift high in the following situations:

Also running on the same card is a desktop program, another inference service, or a training task.
Available video memory will fluctuate after startup.
Spikes are prone to occur when using CUDA graphs, vision models, or high-concurrency prefills.
You’ve encountered a CUDA OOM on the fly, not just a failed initialization check.

If multiple instances share the same GPU, you should have an explicit budget for each instance rather than setting them all to 0.9.

FP8 KV Cache: Confirm the model and version before saving graphics memory

vLLM supports changing the cache data type via --kv-cache-dtype. CUDA 11.8+ environments are available with FP8 related options, such as:

1
2
3

vllm serve Qwen/Qwen3-8B \
  --kv-cache-dtype fp8 \
  --max-model-len 16384

FP8 KV Cache can significantly reduce cache usage, but don’t think of it as a completely free switch:

First confirm that the current vLLM, CUDA and hardware support this data type.
Whether the model checkpoint provides an appropriate KV scale will affect the results; its absence needs to be evaluated with caution.
Long contexts, complex reasoning, tool calls, and structured output should all be tested against the default cache accuracy.
If you just want to fit an unsuitable large model into the graphics card, it is usually more stable to lower the context first or change to a more suitable model.

Therefore, the more recommended order is: first use the default precision to run 4K/8K stably, and then test whether FP8 can really allow the business to obtain the required context or concurrency.

Know priorities when manually specifying KV Cache size

Newer vLLMs provide kv_cache_memory_bytes to specify the KV Cache size precisely in bytes per GPU. It is suitable for scenarios where multiple services share a card and require a fixed graphics memory budget.

Note: Explicitly setting the number of cache bytes will override the automatic inference of the cache size via gpu_memory_utilization. Don’t think of both as additive gains; first decide whether to use a “proportional budget” or a “fixed cache budget”.

Fixed budget is suitable for servers with clear operation and maintenance constraints. It is more intuitive to use --gpu-memory-utilization and --max-model-len for single-machine troubleshooting.

What can CPU offload solve and what can’t it solve?

The new version of vLLM supports offloading part of the KV Cache to the CPU, or combining it with back-end processing cache layers such as LMCache. This can scale capacity when the GPU cache is insufficient, but at the cost of PCIe/memory transfers and latency.

It is more suitable for:

Occasionally very long context requests;
The business can accept higher initial token delay;
There is sufficient system memory and real stress testing has been done;
Want to make a degradation path for long prompts instead of pursuing the highest throughput.

It is not suitable to cover up the problem of “the model weight cannot be put down”, nor can it replace the management of concurrency and context upper limits. If all requests are swapped out and in frequently, throughput will often drop significantly.

A replicable troubleshooting template

Start with a conservative configuration:

vllm serve Qwen/Qwen3-8B \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096 \
  --max-num-seqs 2

After confirming that the order request can be processed stably, adjust in the following order:

4096 上下文稳定
-> 8192 上下文稳定
-> max_num_seqs 从 2 调到 4
-> 调整 max_num_batched_tokens
-> 再测试 FP8 KV Cache 或 CPU offload

Four pieces of data are recorded in each round: number of available KV Cache tokens, first token delay, generated tokens/s, and peak video memory. This way you can tell whether an optimization is improving throughput or simply deferring OOMs to higher concurrency.

Common misunderstandings

Misunderstanding 1: The model supports long context, so the service must be fully enabled

The upper limit of model capabilities and the upper limit of serviceability of your graphics card are two different things. The server’s max_model_len should be set according to the hardware and business upper limit.

Misunderstanding 2: Set `gpu_memory_utilization` to 1 to solve the problem

This reduces the safety margin and does not reduce the need for the KV Cache itself. Runtime spikes are more likely to trigger CUDA OOM.

Misunderstanding 3: Only reduce weight quantization, regardless of concurrency

Weight quantization can free up video memory, but concurrency and context will still cause the KV Cache to expand rapidly. Service configurations must limit both length and number of requests.

Myth 4: CPU offloading is definitely better than rejecting long requests

If latency is sensitive, frequent offloading may be worse than explicitly routing long requests to nodes with larger memory. First define the delay and cost boundaries of the business.

Summarize

vLLM KV Cache has insufficient memory. The most reliable processing priority is:

降低 max_model_len
-> 限制 max_num_seqs 与 batch
-> 小步调整 gpu_memory_utilization
-> 验证 FP8 KV Cache
-> 最后考虑固定缓存预算或 CPU offload

Let a short context and low concurrency configuration work stably first, and then gradually expand it according to business data. As long as the model weight, KV Cache, concurrency and real context length are looked at separately, most “out of memory” problems can be located faster.

refer to:

How to configure Ollama multi-model switching: resident, video memory and Modelfile tutorials

Sat, 11 Jul 2026 09:32:06 +0800

After Ollama has installed several models, the first question many people have is: How to quickly switch between chat, code, translation and Embedding models? The second question is often more practical: Why does the previous one disappear from the video memory just after switching to another model?

Let me start with the conclusion: Ollama does not need to serve each model separately. Use ollama run <模型名> for daily switching; use ollama ps to see which models are currently in memory; use ollama stop <模型名> to release unnecessary models. Whether multiple models can be resident at the same time depends on whether they can fit into the available video memory or memory, rather than how many models are downloaded to the local disk.

Most commonly used switching commands

First list the models that have been downloaded by this machine:

`1`	`ollama ls`

Start a chat or code model:

`1`	`ollama run qwen3:8b`

When you need to change to another model, run another name directly:

`1`	`ollama run qwen3:4b`

Or switch to the Embedding model for testing:

`1`	`ollama run embeddinggemma "测试一段文本"`

Model files remain local and will not be re-downloaded when switching. The weights need to be put into video memory or system memory when loading the model for the first time. If the model is still retained in memory, it will be faster to call it again.

`ollama ps`: First see who occupies the video memory

When the switching is not smooth, first execute:

`1`	`ollama ps`

It lists models that are running or still resident in memory. The most interesting things to look at here are the model name, footprint, processor location, and expiration time.

If you only have a medium-sized graphics card, and you launch two large models one after another, Ollama may uninstall the first model to make room for the second one. This is normal resource scheduling, not that the model is lost. The model is still on disk and will be reloaded on the next call.

If you don’t want to wait for it to expire naturally, you can stop it proactively:

`1`	`ollama stop qwen3:8b`

Then use ollama ps to confirm that the video memory has been released.

Default dwell time: 5 minutes

Ollama defaults to retaining a model for approximately 5 minutes after it was last used. This design is suitable for continuous questions: the first loading is slightly slower, and subsequent requests do not need to repeatedly move the same model back to the video memory.

If you use multiple models in turn on a machine with small video memory, 5 minutes may actually cause the feeling of “just finished running one, and the video memory has not been returned yet”. There are three control methods at this time.

Method 1: Uninstall immediately after one call

Set keep_alive to 0 via API:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "用一句话解释 KV cache",
  "keep_alive": 0
}'

This is suitable for situations where the model is large, video memory is tight, and each task is independent.

Method 2: Keep the model resident for a long time

If you are using the same code model all day, you can set keep_alive to a negative number:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "keep_alive": -1
}'

This way the model will remain in memory until manually ollama stop or the service is restarted. Don’t set this up for multiple large models when you don’t have enough video memory.

Method 3: Globally modify the default resident time

OLLAMA_KEEP_ALIVE can be set for Ollama services. For example, if you want all models to be retained for 30 seconds by default:

`1`	`OLLAMA_KEEP_ALIVE=30s`

Under Windows, Ollama inherits user or system environment variables. Once setup is complete, you need to exit Ollama from the tray and restart from the Start menu. If Ollama is managed by systemd on Linux, set the service environment variable and restart the service.

keep_alive in the API request will override the global OLLAMA_KEEP_ALIVE, so it is more suitable to set different policies for different tasks.

The key to multi-model persistence: `OLLAMA_MAX_LOADED_MODELS`

OLLAMA_MAX_LOADED_MODELS is used to limit the number of models that can be loaded simultaneously. For example, you only want the service to retain at most one model:

`1`	`OLLAMA_MAX_LOADED_MODELS=1`

The purpose of this setting is to avoid filling up the video memory for a long time when the model is rotated, but it is not a method of “forcing a large model to fit into the graphics card”. During GPU inference, a new model can reside concurrently with other models only if it can be completely fit into the available video memory. Otherwise Ollama will unload the old model, or put the model to a slower memory path.

For a single 8GB, 12GB or 16GB graphics card, the more stable strategy is usually:

scene	suggestion
Only use one chat model every day	`OLLAMA_MAX_LOADED_MODELS=1`, keep for 5 minutes or less
Small model chat + Embedding	First check the actual occupancy of both. If you can put them down at the same time, then increase the quantity.
Code model and general model are used alternately	Do not insist on dual resident, switch according to tasks and take the initiative `ollama stop`
Server multi-user call	Combine the model size, video memory, and request volume, and then set the concurrency and queue

Don’t ignore concurrency: contexts eat memory too

Multi-model problems are not just about model weights. Each parallel request increases the resource consumption of the context and KV cache.

Ollama also provides two related environment variables:

1
2

OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_QUEUE=512

OLLAMA_NUM_PARALLEL controls the number of requests that can be processed in parallel by the same model. When the number of concurrency increases, the required resources will increase with the context length. When using a single card locally, it is often easier to troubleshoot by leaving the default or explicitly setting it to 1; do not load multiple models while increasing concurrency.

OLLAMA_MAX_QUEUE is the number of requests that can be queued when busy. It only solves queuing and does not increase video memory.

Use Modelfile to create fixed aliases for different purposes

If you always set system prompt words, temperature, or context policies repeatedly for the same base model, you can use the Modelfile to create multiple local aliases. Instead of duplicating an entire set of weights, they define different configurations based on a model.

For example, create a code-biased configuration file Modelfile.code:

1
2
3

FROM qwen3:8b
SYSTEM 你是一个中文编程助手。先说明修改思路，再给出可运行的最小代码。
PARAMETER temperature 0.2

Create a model alias:

`1`	`ollama create qwen3-code -f Modelfile.code`

Then run directly:

`1`	`ollama run qwen3-code`

You can also make a writing-oriented version:

1
2
3

FROM qwen3:8b
SYSTEM 你是中文写作助手，回答前先给结论，再给必要的结构化说明。
PARAMETER temperature 0.7

1
2

ollama create qwen3-write -f Modelfile.write
ollama run qwen3-write

Note: qwen3-code and qwen3-write, although sharing the same base orientation, are still different model configurations at runtime. When video memory is tight, don’t assume they can be resident indefinitely at the same time.

Give scripts or APIs to switch models by task

In the API, the model name itself is the routing field. The script does not need to restart the service, just pass in different model according to the task:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3-code",
  "messages": [
    {"role": "user", "content": "解释这段 Python 的异常处理逻辑"}
  ],
  "stream": false,
  "keep_alive": "10m"
}'

A common division of labor is:

Small model: classification, rewriting, summary, simple question and answer;
Code model: interpret the warehouse, generate scripts, and fix errors;
Embedding model: vector retrieval, not responsible for chatting;
Larger models: complex problems, loaded on demand, released after use.

Writing “which model to choose” in the task routing of the code makes it easier to control speed and memory than cramming all the work on one large model.

Common pitfalls under Windows

The environment variable was changed but it didn’t take effect

After modifying OLLAMA_KEEP_ALIVE, OLLAMA_MAX_LOADED_MODELS, or OLLAMA_MODELS on Windows, you must exit the running Ollama tray program and restart. Simply reopening PowerShell is usually not enough.

Model files crowd the system disk

You can set OLLAMA_MODELS to move the model directory to another disk, for example:

`1`	`OLLAMA_MODELS=D:\OllamaModels`

Save and restart Ollama. Before migrating existing models, confirm the disk space and directory permissions. Do not delete the old directory just to change the volume.

I thought the model was deleted

After switching, you can’t see the old model with ollama ps, which just means that it has been unloaded from the memory; you can still see the downloaded model with ollama ls. Local model files will be deleted only by executing the following command:

`1`	`ollama rm qwen3:8b`

A set of default strategies suitable for a single card

If you only have one consumer-grade graphics card and frequently switch between multiple models, you can start with this strategy:

1
2
3

OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_KEEP_ALIVE=2m
OLLAMA_NUM_PARALLEL=1

Then call the qwen3-code, qwen3-write, or Embedding model on a per-task basis. When you need to run a large model, execute ollama stop first to stop unnecessary models; when encountering long contexts or large files, lower the number of concurrency and resident numbers.

The goal of this configuration is not to allow the graphics card to load as many models as possible at the same time, but to allow each switch to be predictable and the graphics memory will not be filled up with models that will be forgotten for a long time.

Summarize

There are only four core commands for Ollama multi-model switching:

ollama ls
ollama run <模型名>
ollama ps
ollama stop <模型名>

If you want to better manage models for different purposes, use Modelfile to create aliases; if you want to control loading and release, use keep_alive, OLLAMA_KEEP_ALIVE and OLLAMA_MAX_LOADED_MODELS. First determine the number of simultaneous residents according to the video memory, and then consider multi-model routing and concurrency. The configuration will be much more stable.

refer to:

GPU VRAM on KnightLi Blog

How to calculate the cost of running Agent on a consumer-grade graphics card: electricity bill, depreciation and cost per task

First determine which costs you want to calculate

Total monthly cost formula

An example of replaceable parameters

Why is Agent more difficult to estimate than regular chat?

Two units that must be seen at the same time

every successful task

per million tokens

How to collect your own data

How to calculate depreciation so as not to mislead yourself

When comparing to cloud APIs, don’t just compare unit price

A practical decision line

Common misunderstandings

Summarize

Whether the Ollama performance of NAS deployment is insufficient: how to judge the CPU, memory and graphics card

First confirm which category your NAS belongs to

Let’s first look at “storage” and “reasoning” separately.

CPU-only NAS: When is it worth deploying?

How much memory should be reserved at least?

The biggest difference in experience is seen with NVIDIA GPUs

Can the core graphics, NPU and transcoding capabilities of NAS be used directly?

How to choose a model so as not to bring down the NAS

Recommended deployment structure

Five-minute test before deployment

Don’t ignore network and security

Summarize

What to do if vLLM KV Cache has insufficient memory: video memory, context and concurrency troubleshooting

First understand: why KV Cache fills up the video memory

First determine what kind of problem it is from the logs

First priority: lower --max-model-len

Second priority: Limit concurrency and batch processing

Third priority: Properly set --gpu-memory-utilization

FP8 KV Cache: Confirm the model and version before saving graphics memory

Know priorities when manually specifying KV Cache size

What can CPU offload solve and what can’t it solve?

A replicable troubleshooting template

Common misunderstandings

Misunderstanding 1: The model supports long context, so the service must be fully enabled

Misunderstanding 2: Set gpu_memory_utilization to 1 to solve the problem

Misunderstanding 3: Only reduce weight quantization, regardless of concurrency

Myth 4: CPU offloading is definitely better than rejecting long requests

Summarize

How to configure Ollama multi-model switching: resident, video memory and Modelfile tutorials

Most commonly used switching commands

ollama ps: First see who occupies the video memory

Default dwell time: 5 minutes

Method 1: Uninstall immediately after one call

Method 2: Keep the model resident for a long time

Method 3: Globally modify the default resident time

The key to multi-model persistence: OLLAMA_MAX_LOADED_MODELS

Don’t ignore concurrency: contexts eat memory too

Use Modelfile to create fixed aliases for different purposes

Give scripts or APIs to switch models by task

Common pitfalls under Windows

The environment variable was changed but it didn’t take effect

Model files crowd the system disk

I thought the model was deleted

A set of default strategies suitable for a single card

Summarize

First priority: lower `--max-model-len`

Third priority: Properly set `--gpu-memory-utilization`

Misunderstanding 2: Set `gpu_memory_utilization` to 1 to solve the problem

`ollama ps`: First see who occupies the video memory

The key to multi-model persistence: `OLLAMA_MAX_LOADED_MODELS`