It cares where that memory is, how fast it can reach it, and whether the model fits. Here is how to choose between a conventional GPU, a DGX Spark, and a Mac with unified memory.
Imagine a workshop. RAM is the shelving around the room. VRAM is the workbench directly under the worker’s hands.
The shelves can hold tools, plans, open applications, and model files waiting to be used. The workbench holds the pieces the GPU is actively calculating with. Moving a piece from a shelf to the bench takes time. If the whole job fits on the bench, the worker can keep moving. If not, work becomes a cycle of fetching, waiting, and putting things back.
Holds the operating system, applications, model files, and CPU-side work. Usually easier and cheaper to expand.
Feeds thousands of GPU cores at once. Usually much faster, but fixed to the graphics card and expensive per gigabyte.
A model is mostly a very large collection of numbers called parameters. During inference, those weights need to be read again and again. A 70-billion-parameter model stored at 4 bits needs roughly 35 GB for weights alone. Real use needs more: metadata, runtime buffers, and a growing KV cache that remembers the conversation.
That sentence is deliberately simplified—compute, software kernels, prompt processing, and architecture also matter—but it is the right first mental model for buying hardware.
The GPU has its own high-bandwidth memory. This is ideal when your model fits in VRAM. Spill into system RAM and data must cross PCIe; performance can fall sharply.
CPU and GPU work from a shared memory system, avoiding a separate copy across a PCIe bridge. More of the installed capacity can be available to the model—but “unified” does not mean identical speed or software support.
The “best” machine changes when the model crosses a memory boundary. A smaller GPU can be faster—right up until the job no longer fits.
| Machine pattern | Memory story | Where it shines | Trade-off to accept |
|---|---|---|---|
| Discrete NVIDIA GPU | Dedicated GDDR VRAM. An RTX 5090, for example, has 32 GB. | Fastest route for models that fit. Mature CUDA ecosystem, broad tooling, strong training and image-generation support. | VRAM is a hard, costly ceiling. Multi-GPU can add capacity, but communication and software complexity enter the picture. |
| NVIDIA DGX Spark | 128 GB coherent unified LPDDR5x memory; NVIDIA lists 273 GB/s bandwidth. | Large CUDA models in a compact appliance. More model capacity than a single consumer GPU, the NVIDIA stack, and a compact system with a 240 W power supply. | Capacity over raw bandwidth. Its unified pool is much larger than 32 GB, but a high-end discrete GPU feeds its cores from much faster GDDR. |
| Apple silicon Mac | Unified memory shared by CPU and GPU. Current Mac Studio configurations range far beyond typical consumer VRAM; M3 Ultra can be configured up to 512 GB. | Exceptional single-box capacity, quiet operation, and a polished general-purpose workstation. Excellent with Metal-native tools such as MLX and llama.cpp. | Not CUDA. Some AI projects, kernels, training recipes, and extensions expect NVIDIA. Check your exact stack before buying. |
| CPU + lots of RAM | Commodity system RAM, often expandable to large capacities. | Cheapest way to make a large model run at all. Useful for experimentation, embeddings, or patient batch jobs. | Usually the slowest interactive experience. Capacity can rescue feasibility; it cannot manufacture GPU-class bandwidth. |
Weights + your runtime allowance. This is a planning estimate, not a guarantee.
Formula: parameters × bits ÷ 8, plus overhead. Quantization formats add metadata; context use varies by architecture, length, batch size, and runtime.
It can provide GPU-addressable capacity, but not the bandwidth, architecture, or software behavior of a hypothetical 128 GB discrete card.
Peak low-precision compute is only one figure. Memory bandwidth, kernels, quantization, context, and prompt-processing behavior all affect real speed.
Sometimes it is much slower. Offloading can be useful, but the PCIe bridge and CPU-side execution can become the bottleneck.
No. Leave headroom for the operating system, runtime, temporary buffers, KV cache, and any other applications.
Inference, fine-tuning, image generation, and training stress hardware differently. “Local AI” is not one workload.
Calculate weight size, then add context and runtime headroom. Do not buy from parameter count alone.
CUDA, ROCm, Metal, MLX, Vulkan, and CPU backends do not offer identical coverage. Verify your must-have projects and extensions.
Look for benchmark results using your model, quantization, context, batch size, and runtime—not a vendor’s peak compute number.
Plan for the OS and growing contexts, but avoid paying heavily for capacity that your software or workflow cannot exploit.