AI Tools

Ollama Using CPU Instead of GPU: Platform-Specific Troubleshooting and Recovery

Following the official Ollama documentation, use ollama ps, service logs, nvidia-smi, Docker GPU checks, and AMD device permissions to identify why inference falls back to the CPU, then verify the fix.

When running large models locally, the most confusing problem is: the machine obviously has a graphics card, but Ollama still mainly consumes CPU, and the speed is ridiculously slow.

Let me talk about the conclusion first. This type of problem usually does not have a single cause. The most common categories are:

  • Ollama No available GPU is recognized at all
  • The driver and ROCm or CUDA environment are not installed correctly.
  • Ollama The correct environment variables are not inherited when the service is started.
  • The model is too large and has actually been returned to CPU or CPU/GPU mixed loading
  • AMD platforms have additional compatibility issues, such as incorrect ROCm version, gfx codename, or incorrect device visibility settings

Listed below are the most time-saving order. 1. First confirm whether “GPU is really not used”

The most direct way is to look at:

1

ollama ps

Focus on the PROCESSOR column:

  • 100% GPU: all model weights are loaded on the GPU.
  • 100% CPU: the weights are in system memory.
  • A value such as 48%/52% CPU/GPU: part of the model is offloaded to the GPU and the rest remains in system memory.

These percentages describe where the model is loaded, not instantaneous GPU utilization. Utilization may fall between generations while the weights remain in VRAM.

If what you see is 100% CPU, you should focus on checking the environment and service configuration later.

If you see mixed loading, it’s not necessarily that “the graphics card is not working”, it’s more likely that there’s not enough VRAM. 2. First eliminate the most common misunderstanding: the model cannot be loaded into the VRAM

Many people think that as long as a GPU is installed, Ollama will be able to perform full graphics card inference. Actually not.

If the model is too large, the context is too long, or there are other models on the machine occupying VRAM, Ollama is likely to fall back to:

  • Part GPU + Part CPU
  • Direct 100% CPU

At this time, you can do the two simplest verifications first:

  1. Change to a smaller model for testing For example, use small models such as 4B and 7B first, instead of running a larger number of parameters right from the start.
  2. Unload other loaded models and try again First look at ollama ps to make sure there are no other models occupying VRAM.

If a small model can run on the GPU but a large model cannot, the problem is often not the driver but the VRAM capacity. 3. Check whether the graphics card driver and underlying environment are available

If even small models can only run CPU, the next step is to look at the underlying environment. NVIDIA Direction

First make sure the driver is normal and the system can see the graphics card. Common inspection methods include:

1

nvidia-smi

If errors are reported here, Ollama it is basically impossible to use the GPU normally. AMD / ROCm direction

If you are AMD GPU, especially in ROCm environment, the key points are:

1
2

rocminfo
rocm-smi

If these tools cannot list the device normally, it means that the problem still exists before Ollama. Do not continue to toss the application layer.

For AMD, the most common question is not “Are the drivers installed?” but:

  • ROCm version does not match the system version
  • Current GPU architecture support is incomplete

- Although the device exists, the operating environment is not correctly exposed to Ollama 4. Restart the Ollama service, do not just restart the terminal

This is a very high frequency pit.

After many people install the driver, change the environment variables, and complete ROCm, they just reopen a terminal and continue directly with ollama run. But if Ollama is running as a background service, it is likely that it is still using the old environment.

So a more stable approach is:

  • Completely restart Ollama service
  • Restart the system directly if necessary

If you are running as a service on Linux, you usually need to confirm that the service process has been restarted instead of using the old process. 5. Check whether the service environment variables are actually passed in

This step is especially important in the AMD ROCm environment.

Some machines have no problem manually executing commands in the terminal, but the Ollama service still only runs CPU. The reason is that the service process does not get the variables you set in the shell.

Common variables to watch include:

1
2

ROCR_VISIBLE_DEVICES
HSA_OVERRIDE_GFX_VERSION

Among them:

  • ROCR_VISIBLE_DEVICES is used to limit or specify which GPUs ROCm can see
  • HSA_OVERRIDE_GFX_VERSION Common in some AMD platform compatibility issues

If you only temporarily export variables in the current terminal, but Ollama is started by systemd, desktop background service or other daemon process, these variables may not take effect.

In other words, just because it “seems to have been set up” in the terminal does not mean that Ollama has actually been obtained. 6. AMD platform focuses on ROCm compatibility

Judging from the public page information, the video theme corresponding to this problem itself falls on the line AMD Max+ 395, strix halo, and AMD ROCm. In this kind of environment, Ollama does not use the GPU and often relies more on version matching than the NVIDIA platform.

You can prioritize the following items:

  1. Is version ROCm suitable for the current system and current graphics card?
  2. Whether the current GPU belongs to the architecture range that ROCm supports better
  3. Do you need to supplement HSA_OVERRIDE_GFX_VERSION?
  4. Is it the old version of Ollama or the old version of the underlying reasoning library that causes compatibility issues?

If you have confirmed that rocminfo is normal and the GPU can be recognized by the system, but Ollama still only runs CPU, then there is a high probability that you need to go back to the version combination and recheck instead of continuing to blindly adjust the model parameters. 7. Docker, WSL or remote environments need to additionally check device mapping

If you are not running directly on bare metal, but running in the following environments:

-Docker -WSL

  • Remote container -Virtualized environment

Then we need to look at one more layer: whether the GPU device is actually exposed.

Typical phenomena are:

  • The host can see the GPU
  • Ollama in the container can only run CPU

At this time, you must first confirm that the problem is not Ollama itself, but that the container or subsystem does not have access to the GPU at all. 8. Read the log at the end instead of guessing at the beginning

If you have checked everything before, the most effective way is not to continue to reinstall repeatedly, but to directly look at the Ollama startup log and running log.

Focus on two types of information:

  • Is the GPU recognized?
  • Are there any errors such as driver, library loading, device initialization failure, etc.

As long as something like “Compatible GPU not found” or “Initialization of ROCm/CUDA failed” clearly appears in the log, the troubleshooting direction will be much clearer immediately. Read real logs by platform

Windows

Completely exit Ollama from the tray before starting it in debug mode:

The

1
2

$env:OLLAMA_DEBUG="1"
& "ollama app.exe"

installation version log is usually located at:

1

%LOCALAPPDATA%\Ollama\server.log

can open the directory directly:

1

explorer $env:LOCALAPPDATA\Ollama

Linux systemd 

1
2

sudo systemctl restart ollama
sudo journalctl -u ollama --since "10 minutes ago" --no-pager

When NVIDIA device discovery fails, check again:

Typical CUDA errors listed in the

1
2
3

nvidia-smi
lsmod | grep nvidia_uvm
sudo dmesg | grep -iE 'nvrm|nvidia'

official troubleshooting document include not initialized, device unavailable, no device, and unknown errors. When these messages appear in the log, the direction is the driver, UVM module or container runtime, not the model prompt word. Docker

First prove that the GPU can be seen when the container is running:

1

docker run --rm --gpus all ubuntu nvidia-smi

When this command fails, the Ollama container cannot use the NVIDIA GPU. Repair the NVIDIA Container Toolkit or Docker configuration before restarting the Ollama container. Linux AMD

Check whether Ollama service users can access /dev/kfd and /dev/dri:

1
2

ls -lnd /dev/kfd /dev/dri /dev/dri/*
id ollama

When you need to debug the ROCm discovery process:

1

AMD_LOG_LEVEL=3 OLLAMA_DEBUG=1 ollama serve

If the log shows GPU discovery timeout and the system is still using ROCm 6.x or earlier driver, you should upgrade to a compatible ROCm 7 driver and restart according to Ollama’s current hardware documentation. Don’t use HSA_OVERRIDE_GFX_VERSION to mask a clear driver version error. Acceptance after repair

Complete at least one cold start verification after repair:

1
2
3

ollama stop <模型名>
ollama run <模型名> "只回复 READY"
ollama ps

Observe nvidia-smi, rocm-smi or the system GPU monitor simultaneously. The acceptance conclusion is recorded according to the PROCESSOR column of ollama ps:

  • 100% GPU: The model is completely loaded to the GPU;
  • 100% CPU: Still fully using system memory;
  • CPU/GPU: Mixed loading, usually you need to continue to judge whether it is due to insufficient VRAM capacity.

Finally, check the service log again to confirm that no new device discovery failed. Just seeing a momentary increase in GPU utilization is not enough to prove that the model is always on the GPU. Recovery and rollback

If Ollama cannot start after modifying the environment variables, first undo the variables added in this round and restart the service. Do not modify the driver, container mapping, ROCm version, and Ollama configuration at the same time, otherwise it will be impossible to determine which change will take effect.

If you just want to remember the shortest path, you can do it in this order:

  1. ollama ps Check whether it is GPU, CPU or a mixed load.
  2. Change to a smaller model to rule out insufficient VRAM.
  3. Use nvidia-smi, rocminfo, rocm-smi to first confirm that the underlying environment is normal
  4. Completely restart the Ollama service
  5. Check the service environment variables, especially AMD’s ROCR_VISIBLE_DEVICES and HSA_OVERRIDE_GFX_VERSION
  6. If it is Docker/WSL, check the device mapping again

7. Finally, look at the logs to locate the specific error report. Official information