I set out to run a large language model entirely on my own computer. The model loaded. It answered questions. Technically, the experiment worked. But “it runs” and “it is usable” turned out to be very different milestones.
Enough RAM does not mean good performance
A common assumption is that a computer can run a large model well as long as it has plenty of memory. My machine had about 62 GB of system RAM, more than enough to hold a 23 GB model. Ollama still started in CPU-only mode.
“If the model fits in memory, the machine should run it quickly.”
Memory determines whether the model fits. Compute and memory bandwidth determine how fast it runs.
A dedicated GPU combines specialized compute with fast, dedicated VRAM. An integrated GPU shares ordinary system memory. That shared pool can make a large model possible, but it does not turn an integrated GPU into a high-end discrete card.
GPU offloading also does not guarantee a speedup. The honest test is to benchmark CPU and GPU modes using the same model, prompt, and context size.
The GPU existed. Ollama still could not use it.
The computer had Intel Arc integrated graphics and a working Vulkan driver. Two software details kept Ollama on the CPU:
- Arch Linux packages Ollama's Vulkan backend separately.
- Ollama ignores integrated GPUs unless they are explicitly enabled.
# Install the missing backend sudo pacman -S ollama-vulkan # Systemd service override [Service] Environment="OLLAMA_IGPU_ENABLE=1" # Result library=Vulkan device=Intel(R) Arc(tm) Graphics (MTL) offloaded 42/42 layers to GPU
After the restart, Ollama reported 100% GPU placement and roughly 46.6 GB of shared GPU memory. That proved the configuration worked. It did not, by itself, prove that every workload would be faster.
The 20,000-token “hi”
I sent the word hi to the model twice. Directly through Ollama, the model received 11 input tokens. Through a full Hermes agent, it received 20,011.
Hermes was not only sending my greeting. It attached its operating instructions, definitions for 29 tools, a list of 68 skills, memory rules, browser controls, terminal schemas, delegation instructions, and other agent behavior.
System prompt 22,243 characters Tool definitions 61,775 bytes Skills index 6,950 characters Largest tool schemas: computer use 9,699 bytes cron jobs 7,314 bytes session search 6,457 bytes browser 6,341 bytes file tools 6,143 bytes
The model was not taking five minutes to understand “hi.” It was taking five minutes to read the technical manual attached to it.
Hosted models often process this overhead quickly enough that we barely notice it. Local models make the hidden cost visible. A capable agent harness is useful when the task requires tools; for casual chat, it can be expensive baggage.
Tokens per second can tell the wrong story
The Hermes request appeared to achieve 73 combined tokens per second. The direct Ollama request managed only about 12. That sounds like Hermes was faster. It was not.
| Request | Input | Output | Prompt rate | Generation rate |
|---|---|---|---|---|
| Full Hermes | 20,011 | 35 | 74.05 t/s | 9.74 t/s |
| Direct Ollama | 11 | 180 | 20.44 t/s | 11.32 t/s |
Reads many input tokens in parallel batches. Long prompts can keep the GPU busy and produce a high rate.
Produces one new token at a time. Each token depends on the token before it, so this stage is slower.
The Hermes average was dominated by 20,011 fast, batched input tokens. The direct request was dominated by 180 slower, sequential output tokens. Combining both stages into one number hid the part users actually experience.
Three numbers worth reporting
- Prompt throughput: how quickly the model reads the input.
- Generation throughput: how quickly it writes the response.
- Time to first token: how long the user waits before anything appears.
“73 tokens per second is faster than 12 tokens per second.”
Direct Ollama actually generated text faster: 11.32 versus 9.74 output tokens per second.
A short answer may still involve a long internal monologue
The direct hi request generated 180 tokens, even though the visible task was trivial. Qwen is a reasoning model, so some of that work can happen before the final answer appears.
Reasoning tokens still take time. For planning, coding, and difficult analysis, the extra work may be useful. For a greeting, it is waste. Matching reasoning effort to the task can improve perceived speed without changing hardware.
Input prompt 11 tokens
Output 180 tokens
Prompt processing 538 ms
Generation 15.90 s
Total request 16.66 s
Estimated first token 626 ms
GPU placement 100% Vulkan
A faster runtime cannot remove an oversized prompt
We considered llama.cpp, OpenVINO GenAI, and vLLM. A different runtime may improve raw inference, but it cannot remove instructions inserted by the agent layer.
Ollama already uses a llama.cpp-derived server. Running llama.cpp directly offers more tuning controls, while OpenVINO is an interesting Intel-specific experiment. Neither one changes the fact that a 20,000-token request contains 20,000 tokens.
System instructions, tools, skills, memory, conversation history, and the user's message.
GPU backend, quantization, batching, context allocation, caching, and token generation.
The highest-impact fix depends on the bottleneck. Reduce the harness when the prompt is bloated. Tune or replace the runtime when raw prompt and generation rates are poor.
Practical takeaways
- Separate capacity from speed.RAM may let the model load. Compute and bandwidth determine how well it responds.
- Verify the compute device.Do not assume a GPU is active. Check runtime logs, layer placement, and the serving process.
- Benchmark CPU and GPU fairly.Use the same model, quantization, context, and prompt before declaring a winner.
- Inspect the real input size.A one-word message can become a 20,000-token agent request.
- Match the harness to the job.Use a lightweight chat interface for conversation and a full agent only when its tools are needed.
- Report the right metrics.Track prompt rate, generation rate, first-token latency, model load time, and total time separately.
- Tune reasoning deliberately.Deep reasoning is valuable for hard work and unnecessary for a greeting.
- Treat local AI as a stack.The model, quantization, runtime, drivers, memory, context, and harness all affect the result.
“It runs locally” is the beginning of the story.
The model may fit but run slowly. The GPU may exist but remain unused. The GPU may be active without beating the CPU. A powerful agent may spend more time explaining its tools to the model than answering the user.
Instead of asking for one headline benchmark, ask a few more useful questions: