Using llama.cpp¶

llama.cpp is the heritage engine on this box. The default route through the slot proxy is Strata now, and llama-server is a manual-start container (restart=no) — it never comes up on its own. It stays in the rotation for two things the other engines don't both have: the host-RAM cache tier, and slot save/restore.

Speed¶

No recorded benchmark for this engine on this box. Historically it ran around 60–80 tok/s decode — well below NInfer, which is why NInfer took the day-to-day traffic. llama.cpp stays in the rotation for the cache tier and slot save/restore, not for speed.

Router mode¶

Router mode means the server holds a preset file and serves whatever model you load into it:

# models-preset.ini
[qwen3.6-27b]
model = /models/qwen3.6-27b.gguf
alias = qwen3.6-27b
ctx = 131072
cache_type_k = q4_0
cache_type_v = q4_0
mtp = /models/mtp-draft.gguf

Start the server with --models-preset models-preset.ini, then swap models through the API instead of restarting the container:

curl -X POST http://192.168.4.29:8080/models/load -d '{"model_name":"qwen3.6-27b"}'
curl -X POST http://192.168.4.29:8080/models/unload -d '{"model_name":"qwen3.6-27b"}'

A model swap costs VRAM, not a container rebuild, so a media job or a second engine can take the GPU and give it back without anyone restarting anything.

Slots, save and restore¶

The router keeps slots. Saving and restoring a slot is what makes a long conversation survive a GPU swap:

curl -X POST http://192.168.4.29:8080/slots/1 \
  -d '{"action":"save","model":"qwen3.6-27b"}'

Note the model field — it's not optional. In router mode the server routes by model name, so a save or restore without it has no idea which model the slot belongs to. This bit me once because the swap logic used to hardcode the --model-name flag, and after a ComfyUI round-trip it would reload the default model and bind the slot to the wrong thing.

Cache¶

--cache-ram gives llama.cpp a host-RAM checkpoint tier. Without it, the prefix cache is one window in VRAM and two agents working at once evict each other every turn. With it, a miss costs a read from RAM instead of a full re-prefill.

That's the measured reason llama.cpp stays in the rotation even though its decode is slower than NInfer's: 5.4% cache miss rate against 10.4% on an engine with no host-RAM tier, same workload. The measurement is in Strata.

The slot-load cache miss that motivated the custom build is fixed in my fork, not upstream. The Sunday maintenance cron checks whether upstream has picked it up.

The checkpoint sidecar patch¶

One bug I'd flag, and it's not in upstream. See the sidecar patch.