CPU vs GPU for LLM: when a cheap CPU server is enough

CPU vs GPU for LLM work comes down to one question: is the CPU fast enough for the way you use the model? On our Hetzner CX33, a 4 vCPU cloud server for 10.27 € a month, Llama 3.2 3B generated 13.5 tokens/s with no GPU. That is faster than most people read. A GPU only pays off when you need bigger models, many users at once or near-instant answers.

This matters now because small open models got good enough for real tasks, and GPU rental is billed by the second, so "just rent a GPU" sounds cheap until it runs all month. Below are our CPU measurements translated into everyday terms, and the point where a GPU starts to make sense.

Tested in the Lab Hetzner CX23, CX33, CX43, CAX11, CAX21 · Nov 2026

This page sits in our self-hosted LLM section. For the raw benchmark detail per server, see Ollama CPU performance.

What we measured on CPU

We ran Ollama natively, CPU only, on five Hetzner Cloud servers in Helsinki: three x86 plans (CX23, CX33, CX43) and two ARM Ampere plans (CAX11, CAX21). Each model wrote a 256-token answer at temperature 0. Models were Llama 3.2 1B, Llama 3.2 3B and Llama 3.1 8B in the default 4-bit (q4) quantization from the Ollama library. Quantization shrinks the model weights so they fit in less RAM, at a small cost in quality.

Two speeds matter. Generation tokens/s is how fast the answer is written. A token is a piece of a word. Prompt tokens/s is how fast the model reads your input before it starts answering.

Measured by AI Hosting Lab on 2026-11-05 · 5 servers · method · raw data

Two gaps in the table are worth knowing. Llama 3.1 8B did not run on the 4 GB CAX11: Ollama stopped with "insufficient memory (attempted to allocate 4402.48 MB)". On the 4 GB CX23 it failed too: the kernel OOM killer stopped it after about 50 s, although Ollama's fit check had said it would fit.

What these numbers mean in practice

Reading speed: the chat test

An average adult reads English non-fiction silently at 238 words per minute, according to a 2019 meta-analysis by Marc Brysbaert. Google's token documentation gives a rule of thumb of 60 to 80 English words per 100 tokens. Llama's tokenizer is not identical, so treat this as approximate. Put together, a model that streams about 5 to 7 tokens/s keeps pace with a typical reader.

The word ranges are our arithmetic: tokens/s × 60 × 0.6 to 0.8.

Seconds per answer: the "waiting for it" test

Streaming hides some of the wait, but tools that need the full answer (an n8n workflow, an API call, a bot that replies in one message) wait for the last token. For our 256-token answers, divide 256 by the measured speed:

• Llama 3.2 3B on CX23 (5.8 tokens/s): about 44 seconds.
• Llama 3.2 3B on CX33 (13.5 tokens/s): about 19 seconds.
• Llama 3.2 3B on CX43 (21.2 tokens/s): about 12 seconds.
• Llama 3.1 8B on CX43 (9.7 tokens/s): about 26 seconds.
• Llama 3.1 8B on CAX21 (5.7 tokens/s): about 45 seconds.

That is fine for a background job and too slow for a support widget where a visitor expects a reply in a few seconds.

Long prompts: the part most CPU tests skip

Prompt reading is faster than writing, but it scales with input length. Our test prompt was short. If you paste a long document, or a retrieval step adds pages of context, the model reads all of it before the first word appears. At the measured prompt speed of 776.5 tokens/s for Llama 3.2 3B on the CX33, a 2,000-token input would take around 2.6 seconds; at 258.6 tokens/s for Llama 3.1 8B on the same server, around 7.7 seconds. We have not measured long prompts directly, and speed can drop as the context grows, so read these as best cases.

Batch jobs: where CPU shines

For work nobody waits for (tagging support tickets overnight, summarising RSS items, classifying rows), speed matters less than cost per day. Running nonstop at the measured 13.5 tokens/s, the CX33 would write about 1.17 million tokens a day with Llama 3.2 3B. That is a lot of short summaries for 10.27 € a month. We work out the cost per token on our cost per million tokens page.

x86 vs ARM on CPU inference

The ARM servers are close but not ahead. Llama 3.2 3B ran at 12.2 tokens/s on the CAX21 (4 vCPU ARM, 12.69 €) against 13.5 tokens/s on the CX33 (4 vCPU x86, 10.27 €). The 2 vCPU ARM CAX11 beat the 2 vCPU x86 CX23 with the 1B model (15 vs 10.3 tokens/s), but they were nearly level with the 3B (6.5 vs 5.8 tokens/s). More detail is in Ollama on ARM.

CPU or GPU: comparison

Option 1: CPU server

Best for: one user or a small team, 1B to 3B models, batch jobs, keeping data on a server you control. Main drawback: 8B models answer at around reading speed at best (9.7 tokens/s on the 8 vCPU CX43), and anything larger is out of reach on these sizes. Skip it if: you need replies in under five seconds or serve many people at the same time.

Option 2: rented GPU

Best for: models above 8B, many concurrent users, long-document work, short bursts of heavy batch work where per-second billing helps. Main drawback: leaving it on all month costs far more than a CPU server. Skip it if: your workload fits a 3B model and nobody is waiting on the answer.

GPU prices we checked today

These are list prices from RunPod's official pricing page, opened on 5 November 2026 (the page says "Updated November 4, 2026"). The Pods cards appear to show Secure Cloud prices; the FAQ on the same page gives lower "from" prices on Community Cloud for A100 and H100. Prices are in US dollars.

The last column is our arithmetic (hourly price × 24 × 30). Storage and network can add to the bill. We also tried to open Vultr's official pricing page for its cloud GPUs today, but it refused our request, so we left Vultr prices out rather than copy them from a third party. You can check them yourself on Vultr's pricing page.

The useful comparison: the cheapest GPU on that list, run 24/7, costs about 19 times the CX33's 10.27 € monthly price (different currencies, so this is a rough ratio). A GPU wins on speed. It only wins on cost when you either use it hard or switch it off between jobs. If you want to try one for a short batch run, RunPod bills per second.

When a GPU is worth it

• You need a model above 8B. Our 8 GB servers topped out at Llama 3.1 8B, which used 5,941 MiB of system RAM on the CX33. Bigger models need more memory than these plans have.
• Several people use it at once. Our numbers are for one request at a time. A CPU shares its cores between requests, so each user waits longer.
• Answers must arrive in seconds. The fastest 3B result we measured still took about 12 seconds for 256 tokens.
• You work with long documents. Prompt reading on CPU adds seconds per thousand tokens, as shown above.

Ollama itself supports NVIDIA GPUs with compute capability 5.0 or newer, AMD through ROCm, and Apple through Metal, so the same tool runs on both sides. Your setup stays the same when you move.

When CPU is enough

• A private assistant for yourself on a 1B or 3B model.
• Automations that call a model and can wait half a minute, such as n8n workflows that summarise or classify.
• Overnight batch jobs where cost per day matters more than seconds per answer.
• Keeping data in-house on a cheap server instead of sending it to an API.

Before you pick a size, check memory: a model that does not fit in RAM does not run at all, as the CAX11 showed with the 8B model.

Next step: enter your models and apps into our VPS RAM calculator to see which CPU server fits before you spend anything on a GPU.