What to do if vLLM KV Cache has insufficient memory: video memory, context and concurrency troubleshooting

Sat, 11 Jul 2026 09:36:02 +0800

When the following error occurs when vLLM is started, it is usually not because the model weight download is broken:

`1`	`The model's max seq len ... KV cache is needed, which is larger than the available KV cache memory`

It means that after the model weights and runtime overhead take up the video memory, the remaining space is not enough to prepare the KV Cache for the maximum context length you set. The most direct processing sequence is: **Reduce --max-model-len first, then check concurrency, and finally consider improving memory utilization, quantifying KV Cache, or expanding capacity. **

Don’t pull --gpu-memory-utilization to 1 right off the bat. That may allow vLLM to pass initialization, but be more prone to OOM when real requests, CUDA graph, or other processes compete for video memory.

First understand: why KV Cache fills up the video memory

The model weight determines “whether the model can be loaded”, and the KV Cache determines “how many tokens the model can remember at the same time and how many requests it serves.” Every time a token is generated or read, the attention layer needs to save the corresponding Key/Value state.

Therefore, KV Cache usage will increase with the following items:

max_model_len: The longer the maximum context allowed, the greater the cache requirements.
Number of concurrent requests: Each concurrent request will occupy its own context space.
Model structure: number of layers, number of KV heads, hidden size and data type all affect cache size.
Cache precision: The default usually follows the model data type; low-precision caches such as FP8 can save space, but have compatibility and quality boundaries.

The most overlooked thing is concurrency. Even if each request is only given 8K context, the total demand on KV Cache will still grow rapidly when multiple requests are running simultaneously.

First determine what kind of problem it is from the logs

Startup logs usually give three key pieces of information:

The maximum sequence length specified by the model or configured by you.
How many GiB does KV Cache require.
The actual space currently available for the KV Cache, and the maximum feasible length estimated by vLLM.

If the log says “Maximum length 32768 requires 10GiB KV Cache, but currently only 4GiB”, don’t worry about whether the model supports 32K first. Model Support does not mean that your card can serve 32K requests under the current configuration.

Start by taking note of the estimated maximum length provided by the log, and set your first test at around 60% to 80% of it, and then work your way up.

First priority: lower `--max-model-len`

This is the adjustment with the highest success rate and the most predictable impact. For example, the model natively supports 32K, and your actual business only handles ordinary conversations, short codes and small documents, you can start with 8K:

1
2

vllm serve Qwen/Qwen3-8B \
  --max-model-len 8192

If it’s still not enough, try again:

1
2

vllm serve Qwen/Qwen3-8B \
  --max-model-len 4096

Do not write the model card’s nominal 128K, 256K or longer context directly into the service parameters. For single-card deployments, the actual available length depends on weight, video memory, concurrency, and cache accuracy.

A practical starting point:

Video memory and usage	Context length to try first
12GB–16GB single-card, Class 8B model	4096 or 8192
24GB single card, 7B–14B models	8192 or 16384
24GB single card, 30B level quantized model	Start with 4096
Long Document/RAG/Multi-User Service	Make decisions based on log estimates and stress testing, don’t just follow the table

These values are a starting point for troubleshooting, not memory commitments. KV Cache size can vary significantly between models.

Second priority: Limit concurrency and batch processing

Just because the service can be started does not mean it will be stable under high concurrency conditions. First, control concurrency to a small value:

1
2
3

vllm serve Qwen/Qwen3-8B \
  --max-model-len 8192 \
  --max-num-seqs 4

--max-num-seqs limits the number of sequences that can be processed in one iteration. A larger number may result in higher throughput, but also greater KV Cache and scheduling pressure.

If requests often contain long prompts, you should also pay attention to --max-num-batched-tokens. It determines the maximum number of tokens that can be processed at one time; if it is too high, the prefill phase may occupy more resources instantly. Be conservative when troubleshooting, and then gradually expand after confirming stability.

It is recommended to perform stress testing in this order:

Single request, 4K context, confirmed service stability.
Single request, 8K context, observe first token latency and video memory.
Two concurrent requests confirm that there will be no OOM.
Then gradually increase max_num_seqs or the number of batch tokens.

Do not adjust context, concurrency, batch and model quantization at the same time, otherwise it will be difficult to locate which parameter is causing the video memory to be exhausted.

Third priority: Properly set `--gpu-memory-utilization`

--gpu-memory-utilization specifies the proportion of GPU memory that can be used by the current vLLM instance, with a value between 0 and 1. vLLM will plan the weight, runtime and KV Cache based on this part of the space.

For example:

vllm serve Qwen/Qwen3-8B \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 4

When the error message clearly says “Too few available KV Cache” and there are no other processes on the GPU, you can improve it in small steps, such as trying from 0.90 to 0.92 or 0.94. Add just a little at a time and test with real requests.

It is not recommended to deadlift high in the following situations:

Also running on the same card is a desktop program, another inference service, or a training task.
Available video memory will fluctuate after startup.
Spikes are prone to occur when using CUDA graphs, vision models, or high-concurrency prefills.
You’ve encountered a CUDA OOM on the fly, not just a failed initialization check.

If multiple instances share the same GPU, you should have an explicit budget for each instance rather than setting them all to 0.9.

FP8 KV Cache: Confirm the model and version before saving graphics memory

vLLM supports changing the cache data type via --kv-cache-dtype. CUDA 11.8+ environments are available with FP8 related options, such as:

1
2
3

vllm serve Qwen/Qwen3-8B \
  --kv-cache-dtype fp8 \
  --max-model-len 16384

FP8 KV Cache can significantly reduce cache usage, but don’t think of it as a completely free switch:

First confirm that the current vLLM, CUDA and hardware support this data type.
Whether the model checkpoint provides an appropriate KV scale will affect the results; its absence needs to be evaluated with caution.
Long contexts, complex reasoning, tool calls, and structured output should all be tested against the default cache accuracy.
If you just want to fit an unsuitable large model into the graphics card, it is usually more stable to lower the context first or change to a more suitable model.

Therefore, the more recommended order is: first use the default precision to run 4K/8K stably, and then test whether FP8 can really allow the business to obtain the required context or concurrency.

Know priorities when manually specifying KV Cache size

Newer vLLMs provide kv_cache_memory_bytes to specify the KV Cache size precisely in bytes per GPU. It is suitable for scenarios where multiple services share a card and require a fixed graphics memory budget.

Note: Explicitly setting the number of cache bytes will override the automatic inference of the cache size via gpu_memory_utilization. Don’t think of both as additive gains; first decide whether to use a “proportional budget” or a “fixed cache budget”.

Fixed budget is suitable for servers with clear operation and maintenance constraints. It is more intuitive to use --gpu-memory-utilization and --max-model-len for single-machine troubleshooting.

What can CPU offload solve and what can’t it solve?

The new version of vLLM supports offloading part of the KV Cache to the CPU, or combining it with back-end processing cache layers such as LMCache. This can scale capacity when the GPU cache is insufficient, but at the cost of PCIe/memory transfers and latency.

It is more suitable for:

Occasionally very long context requests;
The business can accept higher initial token delay;
There is sufficient system memory and real stress testing has been done;
Want to make a degradation path for long prompts instead of pursuing the highest throughput.

It is not suitable to cover up the problem of “the model weight cannot be put down”, nor can it replace the management of concurrency and context upper limits. If all requests are swapped out and in frequently, throughput will often drop significantly.

A replicable troubleshooting template

Start with a conservative configuration:

vllm serve Qwen/Qwen3-8B \
  --gpu-memory-utilization 0.90 \
  --max-model-len 4096 \
  --max-num-seqs 2

After confirming that the order request can be processed stably, adjust in the following order:

4096 上下文稳定
-> 8192 上下文稳定
-> max_num_seqs 从 2 调到 4
-> 调整 max_num_batched_tokens
-> 再测试 FP8 KV Cache 或 CPU offload

Four pieces of data are recorded in each round: number of available KV Cache tokens, first token delay, generated tokens/s, and peak video memory. This way you can tell whether an optimization is improving throughput or simply deferring OOMs to higher concurrency.

Common misunderstandings

Misunderstanding 1: The model supports long context, so the service must be fully enabled

The upper limit of model capabilities and the upper limit of serviceability of your graphics card are two different things. The server’s max_model_len should be set according to the hardware and business upper limit.

Misunderstanding 2: Set `gpu_memory_utilization` to 1 to solve the problem

This reduces the safety margin and does not reduce the need for the KV Cache itself. Runtime spikes are more likely to trigger CUDA OOM.

Misunderstanding 3: Only reduce weight quantization, regardless of concurrency

Weight quantization can free up video memory, but concurrency and context will still cause the KV Cache to expand rapidly. Service configurations must limit both length and number of requests.

Myth 4: CPU offloading is definitely better than rejecting long requests

If latency is sensitive, frequent offloading may be worse than explicitly routing long requests to nodes with larger memory. First define the delay and cost boundaries of the business.

Summarize

vLLM KV Cache has insufficient memory. The most reliable processing priority is:

降低 max_model_len
-> 限制 max_num_seqs 与 batch
-> 小步调整 gpu_memory_utilization
-> 验证 FP8 KV Cache
-> 最后考虑固定缓存预算或 CPU offload

Let a short context and low concurrency configuration work stably first, and then gradually expand it according to business data. As long as the model weight, KV Cache, concurrency and real context length are looked at separately, most “out of memory” problems can be located faster.

refer to:

LLM推理 on KnightLi Blog