Skip to content

Three MoE quants, one 5090: what each one costs

The 5090 box now holds three mixture-of-experts models at once: Qwen3.8-Flash-Next on Basalt (my old production agent engine's model), GLM-5.3-Flash in Project Maya's Maya-M quant, and DeepSeek-V4.1-Flash in Maya's Maya-XS quant. All three are expert streamers - the model lives in host RAM, hot experts get promoted to VRAM, the rest streams per token - so they share the same physics and completely different numbers. I spent this week measuring them with the engines' own reported timings. The basalt-versus-strata story on the Qwen model already has its own post with the full A/B ledger; this one rounds out the picture with the two Maya quants, a context-length experiment that surprised me, and the quality data both quant makers publish.

The box and the ground rules

GPU RTX 5090 (32 GB)
CPU Ryzen 7 9800X3D
RAM 128 GB DDR5
Storage NVMe (DeepSeek's Engram tables read from here)
Engines Basalt 0.3.2; Project Maya v1.1.65 (maya-glm, maya-ds4)

One model runs at a time; the other containers are stopped, so every row below is a solo measurement. All decode and prefill figures are the engine's own emitted lines (Basalt's predicted_ms block, Maya's glm stat / ds4 serve lines), not client-side math, and requests were issued engine-direct. Decode rates are measured over 256-token windows, so they are short-window rates at the stated context.

One warning before the tables: all three engines use speculative decoding, and the draft acceptance rate is part of every decode number. Basalt's Qwen runs sat between roughly 50 and 80 percent acceptance; DeepSeek's 2-bit DSpark drafter kept 10 of 28 drafts (36 percent, 71 percent at the first position decaying to a third by the third). A decode figure from one of these engines is not directly comparable to another engine without its acceptance rate in hand.

Speed, measured

model / quant pass context prefill tok/s decode tok/s measurement type Source
Qwen3.8-Flash-Next UD-IQ3_XXS, Basalt 0.3.2 cold ~1k ≈1,340 (747 ms) 174.6 solo, engine-emitted measured, Nov 11 ledger (post)
Qwen3.8-Flash-Next UD-IQ3_XXS, Basalt 0.3.2 cold ~64k ≈11,800 (5,546 ms) 268.2 solo, engine-emitted measured, Nov 11 ledger (post)
Qwen3.8-Flash-Next UD-IQ3_XXS, Basalt 0.3.2 warm (cache hit) ~64k 8.4 ms resume 267.2 solo, engine-emitted measured, Nov 11 ledger (post)
GLM-5.3-Flash Maya-M, maya-glm short answer <1k — 31.1-33.1 (median 32.2) solo, engine-emitted, n=7 measured, Nov 17
GLM-5.3-Flash Maya-M, maya-glm cold ~4k 1,868 24.0 solo, engine-emitted measured, Nov 17
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 short answer, cold box <1k — 38.0-40.1 solo, engine-emitted, n=4 measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 short answer, warm box <1k — 50.3-51.0 solo, engine-emitted measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 cold 8k 2,242 48.4 solo, engine-emitted measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 cold 64k 4,581 51.0 solo, engine-emitted measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 cold 128k 4,716 54.5 solo, engine-emitted measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 cold 207k 4,480 56.7 solo, engine-emitted measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 cold 414k 3,933 57.9 solo, engine-emitted measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 cold 829k 3,081 53.9 solo, engine-emitted, 271 s prefill measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 warm follow-up (full prefix reuse) 128k 5.0 s TTFT 55.7 solo, engine-emitted measured, Nov 18
DeepSeek-V4.1-Flash Maya-XS, maya-ds4 warm follow-up (full prefix reuse) 829k 5.1 s TTFT 55.5 solo, engine-emitted measured, Nov 18

Three things stand out to me:

The Qwen lane is a different league on speed. Basalt moves 268 t/s at 64k with 50-80 percent draft acceptance while DeepSeek at the same context does 51-55 with 36 percent. That gap is mostly the active-params class (Qwen's is a smaller brain per token) plus speculative decoding efficiency, not a Maya deficiency.

DeepSeek's decode improves with context. 38 t/s short on a cold box, 54-58 t/s at 207k-414k on a warm one. The long shared prefixes of a session warm the expert cache and the router profile, and the same happens with GLM (Maya's docs date-stamp this as expert-cache warmth, not a long-context bonus). Treat any single decode number from these engines as a band, not a point.

The cold-prefill scaling is a ramp, not a cliff. 4,581 t/s at 64k, 4,480 at 207k, 3,933 at 414k, 3,081 at 829k. That is the attention/indexer cost growing with prompt length showing up late and gently - an 829k-token first paste costs 271 seconds, and after that the prefix cache owns your session: the follow-up to that same 829k conversation came back in 5.1 seconds.

The context experiment: 256K vs 512K vs 1M cost nothing

DeepSeek-V4.1-Flash reaches 1M positions natively (nothing scaled), and Project Maya's ladder offers 32768 through 1048576. The setup recommends 262144 for this machine's memory, and I have previously taken on faith that a big context setting taxes the expert cache through the KV. So I ran the identical battery - four short decodes, salted cold prefills (a unique prefix per run so the prefix cache cannot help), and a deliberate-reuse follow-up - at 262144, 524288 and 1048576, engine restarted between each:

task 262K 512K 1M
64k cold prefill (t/s) 4,581 4,596 4,579
128k cold prefill (t/s) 4,716 4,689 4,682
128k warm follow-up decode (t/s) 55.7 55.0 53.0

Every column matches within noise. The setting itself is free: the context's caches (about 1.1 KB per position, FP4 KV plus index keys) are allocated lazily, so the box only pays for context it actually uses. VRAM held the same ~31.5 GB of expert cache at load across the runs (31.4 GB in the 262K engine logs, 31.7 GB measured at 1M) with 93 GB of RAM resident at 1M. The near-ceiling rows in the speed table above (207k vs 414k vs 829k) degrade with prompt length at every setting, confirming it is the prompt, not the setting. My box now runs 1M permanently, and the 2.5-minute restart between settings is the only cost the experiment produced.

Cross-family percentages don't compare

DeepSeek's 1M reach and GLM's are both native, but their KV designs are not equivalent (MLA latent caches versus GLM's streamed int8 KV), so "1M was free here" does not transfer as a promise about any other model or engine. It is a measured property of this quant, this engine, this box.

Quality: what the quant makers publish

Both Maya quants are made by the same person (peasantsmith) with GPTQ-style error-feedback rounding and importance statistics taken from the original model itself, and both projects publish token-level fidelity against the original on held-out text: how often the quant emits the same top token the full model would, and the next-token loss. That is the best public answer to "how much model am I actually running".

Against GLM-5.3-Flash (FP8 original):

quant size same top token zero-shot accuracy kept Source
Maya-L 156.3 GB 90% 99.2% MAYA-M/L results, Project Maya repo
Maya-M 116 GB 86.2% (KL 0.329) 97.9% MAYA-M results, Project Maya repo
Maya-S 96.5 GB 83.3% 97.9% MAYA-S results, Project Maya repo
Maya-XS 73.4 GB no published numbers — —

Against DeepSeek-V4.1-Flash (FP4/FP8 original):

quant model in memory same top token next-token loss Source
Maya-S 178.2 GB 92.6% +3.2% model card, peasantsmith HF
Maya-XS 115.3 GB 87.5% +8.7% model card, peasantsmith HF

Calibration points from the DeepSeek card worth knowing: the original weights run through the Maya engine score 96.4% same-token, so about four points of every quant's distance is the engine's own rounding, not the quantization. And the 161 GB n-gram Engram tables, which my box reads from SSD at Q6_K, measured "97.5% the same top token - the measurement's own noise floor", meaning they cost no measurable quality. Both quants keep every routed expert; nothing is pruned.

What these percentages can and cannot tell you

Same-top-token is measured against each model's own original, on that family's own held-out text. 87.5 (DeepSeek XS) against 86.2 (GLM M) is not a claim that one model is better than the other - different base model, different tokenizer. What it does say: my two live quants sit one tier apart from their originals' best published quant, and I pay roughly 2-4 points of token fidelity to fit a 300-750 billion parameter model into 128 GB of RAM. Nobody publishes a cross-family bake-off or a zero-shot table for the DeepSeek quants; that is a gap, not a guarantee.

Where each one earns its slot

The proxy fronts all of these and I route by workload. The lane assignment after a week with all three measured:

  • Qwen3.8-Flash-Next on Basalt - voice, fast agents, anything where 250+ t/s and an 8 ms warm resume decide the experience. Fastest by roughly 5x.
  • DeepSeek-V4.1-Flash Maya-XS - the default chat and agent brain right now. Biggest model I can hold, 1M context that costs nothing to enable, vision for free between pictures (the tower borrows ~319 MiB from prompt buffers while a picture is read and holds zero VRAM otherwise), and the Engram memory gives it an unusually good verbatim-recall streak.
  • GLM-5.3-Flash Maya-M - tool-calling and front-end code. Its calibration text is deliberately weighted toward tool calls and HTML/CSS/JS work, it keeps 97.9 percent of the FP8 model's zero-shot accuracy, and on this box it is the slowest of the three (~32 t/s), which is why it is parked until its workload shows up.

For anything where being wrong is expensive, none of these percentages replace verification - quantization error compounds over long generations more than a 400-question multiple-choice table shows. But it is a good day when the honest answer to "how much model am I running?" is a published number instead of a guess.

Raw ledgers for the Nov 17-18 runs live on the box: bench-1165.json, bench-ds4.json and bench-ctx-{262k,512k,1m}.json in the Maya working directory, parsed from glm stat and ds4 serve engine lines.

Comments