You found a great open model, but the file is bigger than your graphics card. Maybe bigger than your RAM. Here's what's possible, using llama.cpp's own docs.
The memory ladder
From fastest to slowest:
- VRAM on your GPU (or unified memory on a Mac or RTX Spark).
- System RAM.
- Your SSD.
The further down the model sits, the slower each token comes out. Our VRAM guide covers the "does it fit" maths.
Trick 1: split layers between GPU and RAM
llama.cpp's CLI docs list -ngl, --gpu-layers, --n-gpu-layers N: the max number of layers stored in VRAM, as an exact number, auto or all. The default is auto.
There's also --fit, on by default, which adjusts unset arguments to fit your device memory, keeping a margin (1024 MiB by default, set with --fit-target).
So usually, llama.cpp tries to fit what it can on the GPU and leaves the rest in RAM. To force it:
llama-cli -m model.gguf -ngl 20
Fewer GPU layers means less VRAM used and slower output.
Trick 2: MoE experts on the CPU
Mixture-of-Experts models only use some of their weights per token. The docs list:
--cpu-moe: keep all MoE expert weights in the CPU.--n-cpu-moe N: keep MoE weights of the first N layers in the CPU.
That puts the big but less-used expert weights in RAM, while attention and shared layers get the GPU.
Trick 3: mmap, when it's bigger than RAM
By default llama.cpp loads in auto mode, which means mmap unless a device doesn't support it.
The explanation, quoted in llama.cpp discussion #1876 from the project's docs:
"By default, models are mapped into memory, which allows the system to load only the necessary parts of the model as needed. However, if the model is larger than your total amount of RAM or if your system is low on available memory, using mmap might increase the risk of pageouts, negatively impacting performance."
And: "if the model is larger than the total amount of RAM, turning off mmap would prevent the model from loading at all."
So yes, a model bigger than your RAM can load. But your SSD becomes part of memory, and speed can fall off a cliff. That's also why your task manager can show much less memory used than the model's size: the OS hasn't paged it all in.
The current CLI also offers --load-mode mlock (force the model to stay in RAM) and mmap+mlock. Those help when you have enough RAM but the OS keeps swapping.
Don't forget the context
The KV cache grows with context length. -c, --ctx-size sets it (0 means use the model's value), and -ctk / -ctv let you store the cache at lower precision such as q8_0. A long context can push a model that "fit" over the edge.
What actually works best
- Smaller quantization first. A 4-bit version of a model often fits where 8-bit doesn't.
- Offload to RAM second. Fine for chat; slow for agents doing lots of steps.
- Disk last. Good for trying a model, not for daily use.
- Or rent. An hour on a big GPU may beat a day of paging. See what it costs to rent an H100 or B300.
Buying hardware? See unified memory: Mac Studio vs RTX Spark vs desktop GPU and the DDR5 buying guide.
