The llama slot proxy is the piece I'd least expect someone to build and the piece I'd least like to work without. It sits on port 8081 and fronts whatever engine currently owns the GPU, and it's the only thing the clients talk to.
A 125B MoE needs about 44 GiB of visible RAM to start. On the same boot, the VM went from 48.0 GiB to 15.9 GiB, and setup stopped with "smallest model needs 32 GB".
Three requests hit exactly 65,536 completion tokens. 240, 252 and 287 seconds each. Thirteen minutes — 12% of a 1.75 hour window — in three requests, none of which delivered an answer.
I ran this for a while with two agents on the same engine and both of them felt sluggish, and I couldn't work out why. The engine reported good numbers. The experience didn't match.
The answer is the prefix cache, and it's structural — not a tuning knob.
In this post I'm going to discuss project and user agents for each of the platforms as well as native and third party applications that allow remote access to your agents.