Hardware comparison: Qwen3.8-Flash-Next and 27B¶
Two models, two dozen machines, one question: where does each model actually run fast. Every number on this page is a published community measurement with a link, or a number I measured on my own boxes. Where nobody has published a number, the cell says so instead of guessing.
The short version: for Qwen3.8-Flash-Next, decode speed is set by memory bandwidth and by which tensor your engine decides to page to SSD. That second part is not a footnote — it's a 14x swing, and it's the difference between a Mac that runs this model at 160 tok/s and the same class of Mac running it at 9.
Why this model behaves differently than everything else¶
Qwen3.8-Flash-Next is a 125B MoE with only 6B active per token, plus a 51B n-gram embedding table and a 4B MTP head (model card, architecture paper). Three facts drive every number below:
- Decode reads the active parameters, not the total. 6B active at ~4-bit is roughly 2.6-3.4 GB of weight traffic per token, so decode ceilings are high on any fast-memory box. A dense 70B at 4-bit reads ~40 GB per token and crawls on the same hardware.
- The n-gram table is a cache, and caches are happy to be slow. A token reads two rows of it (~5 KB). It can live on NVMe via mmap and cost a few percent. This is what Strata does on my 5090, what MTPLX does on every Mac since 2.12.0 (the table ships as a sidecar file), and what the Strix Halo recipe does with 47.7 GiB of table streamed from SSD.
- The experts must NOT be the thing you page. Each token routes through 10 experts x 48 layers, a different ten every token. Page those to SSD and decode collapses.
The 9 tok/s trap
A community oMLX benchmark on a 96 GB M5 Ultra shows 9.0 tok/s for this model. The settings JSON explains it: moe_expert_offload_enabled: true, resident_fraction: 0.25 — the 112 GB table-resident MLX build didn't fit, so the engine kept the n-gram table hot and paged three-quarters of the experts to SSD instead. GPU utilization averaged 19%. Same silicon, right split (Bare Speed pack, ~74 GB resident, table on SSD), and the expected speed is an order of magnitude higher. When a Flash-Next number looks absurd, check which tensor is being paged before believing the hardware verdict. The same trap, expert-pruning variant, is documented in this 128 GB Mac writeup.
Memory requirements by quant (Flash-Next)¶
| Build | Download | Resident in RAM | Notes |
|---|---|---|---|
| GGUF UD-IQ1_S | 72.5 GB | ~45-50 GB + table mmap | table quantized harder at 1-bit tiers (Unsloth) |
| GGUF UD-IQ3_XXS | 82 GB | ~60 GB + table mmap | fits 96 GB Macs / 64 GB RAM + GPU hybrid |
| GGUF UD-Q3_K_XL | 90 GB | ~65 GB + table mmap | |
| GGUF UD-IQ4_XS | 93.7 GB | ~68 GB + table mmap | tight on 96 GB |
| GGUF UD-Q4_K_XL | 111.4 GB | ~80 GB + table mmap | 128 GB tier |
| Strata IQ2_XS / IQ3_XXS / IQ3_S | ~68 GB | RAM gate: 44 / 56 / 58 GiB visible | expert tiering: hot experts VRAM, all experts host RAM (Strata) |
| MLX oQ4 (table resident) | 112 GB | 112 GB | 128 GB tier; this is the build that fails on 96 GB |
| MTPLX Bare Speed | 106.3 GB | ~74 GB, table on SSD | (pack card) |
| MTPLX Optimized Speed | 115.1 GB | ~83 GB, table on SSD | recommended |
| MTPLX Optimized Quality (8-bit) | 170 GB | ~128.5 GiB, table on SSD | 256 GB tier |
Radiance .rad (4-bit codebook experts, int8 trunk) |
113.6 GiB | experts tiered: hottest in 2x 32 GB VRAM, rest streamed from a pinned host pool (~32 GB free RAM) | GPTQ 4-bit into a non-uniform codebook, ten energy-heavy experts kept bf16, MTP + vision, served at 200K (model card) |
Qwen3.8-Flash-Next: decode and prefill by machine¶
| Machine | Memory | Bandwidth | Engine | Decode (tok/s) | Prefill | Source |
|---|---|---|---|---|---|---|
| DGX Spark (GB10) | 128 GB | 273 GB/s | vLLM NVFP4 + MTP | 41.7 median, 42-45 code, 27.4 prose; 17 without MTP | 1,666 tok/s at 32K | ai-muninn recipe |
| DGX Spark | 128 GB | 273 GB/s | llama.cpp Q4_K_XL | 20.5-23.4 | — | NVIDIA forum thread |
| DGX Spark (x2) | 256 GB | 273 GB/s each | vLLM TP2 | 55-60 | — | ai-muninn two-Spark |
| Ryzen AI Halo (Strix Halo, 8060S) | 128 GB | ~256 GB/s | EngramHalo.cpp + MTP | 82 code (3-bit-class quant), 55 JSON, ~22 prose | slow: 200K prompt took 17.3 min to first token | abliter8 recipe + results |
| MacBook Pro M5 Max | 128 GB | 614 GB/s | MTPLX 2.12 + MTP | 54.6-122.4 on real OpenCode tasks, 125.8 best request; 68.4 at 16K, 60.9 at 100K | 1,453 tok/s at 4K, 1,094 at 65K | MTPLX model page |
| MacBook Pro M5 Max | 128 GB | 614 GB/s | mlx-serve, 4-bit pack + MTP (llmprobe, self-reported, Aug 2026) | 91.2 serial; 105.3 on predictable prompts, 66.0 on novel text | 1,633 tok/s | repo hand-off plan |
| Mac Studio M5 Pro | 128 GB | 307 GB/s | — | no published numbers | — | — |
| Mac Studio M5 Ultra (64c) | 96 GB | 1.2 TB/s | oMLX (experts paged — wrong split) | 9.0 (see trap above) | 90.9 tok/s (also degraded) | oMLX benchmark |
| Mac Studio M5 Ultra (64c) | 96 GB | 1.2 TB/s | MTPLX Bare Speed (right split) | not yet benchmarked; ~74 GB resident fits | — | MTPLX pack card |
| Mac Studio M5 Ultra (80c) | 256 GB | 1.2 TB/s | oMLX recipe + lookup drafts | 158-165 fresh, 225-272 agent edit turns | 4,759 tok/s at 32K | sethforprivacy recipe |
| Mac Studio M5 Ultra (80c) | 256 GB | 1.2 TB/s | TensorFold 0.3.6.1 | 183-197 fresh, 437 edit turns (quality cost: 5.5x KLD) | 2,826 tok/s at 32K | oMLX engine comparison |
| Mac Studio M5 Ultra (80c) | 256 GB | 1.2 TB/s | oMLX stock 0.7.0rc1 | 81.2 at 8K code (MTP on); MacStories: 52 at 16K, 111.6 prose | 4,755 tok/s at 8K; 2,887 at 16K | oMLX benchmark · LLMCheck audit |
| Mac Studio M5 Ultra (256 GB) | 256 GB | 1.2 TB/s | mlx-serve 26.9.6, mixed 4-8-bit pack + MTP (llmprobe, self-reported) | 219 decode; 227 aggregate at 4 streams; up to 243-300+ on draftable turns | 3,150 tok/s at 8K (up to 3,362 claimed) | v26.9.6 release notes — box stated as M5 Ultra 256 GB in the verifying PRs |
| RTX 5090 + 5060 Ti + 128 GB | 32+16 GB VRAM | 1.79 TB/s | Basalt (NVFP4, Strata fork tuned for Blackwell) | 239-479 (author-reported; 665 on structured output) | 7,317 at 64K (claimed) | Basalt repo · launch post |
| RTX 5090 | 32 GB VRAM + ~60 GB RAM | 1.79 TB/s | fngine (GSQ-RCO IQ3_S experts, FP8 dense, CPU expert tier) | 146-209; 254-268 on coding-chat decode turns (model card) | — | fngine repo |
| RTX 5090 | 32 GB VRAM + host RAM | 1.79 TB/s | Infernix (grew from NInfer; experts offloaded) | ~140 | — | Infernix repo · intro post |
| RTX 5090 + 128 GB | 32 GB VRAM | 1.79 TB/s | Strata NVFP4 fork (sergqwer) | 177 | 7-8k claimed | strata-nvfp4 |
| 2x RTX 5070 Ti + 96 GB | 2x 16 GB VRAM | 896 GB/s each | capy-flash (NVFP4, TP2) | 171-183 (verified) | 9.2k at 4-16K, 6.2k at 64K (verified) | capy-flash repo |
| 2x RTX 5070 Ti + 96 GB | 2x 16 GB VRAM | 896 GB/s each | Strata IQ3_S (same box as capy-flash rows) | 195 (verified) | 2.2k at 3.5K, 3.8k at 15K, 5.2k at 55K | engine survey |
| 2x RTX 5070 Ti + 96 GB | 2x 16 GB VRAM | 896 GB/s each | Strata NVFP4 (same box) | 120 (verified) | 1.5k at 3.5K, 3.5k at 15K, 4.9k at 55K | engine survey |
| 2x RTX 5090 | 64 GB VRAM | 1.79 TB/s each | llama.cpp fork (finnff) | 142 | — | Oct 2026 engine survey |
| RTX 5090 | 32 GB VRAM + host RAM | 1.79 TB/s | Strata IQ2_XS | 129-172 (measured); 150-200 on current build with IQ3_S | 64K prompt in 11.1 s (~5,800 tok/s) | Strata page |
| Mac Studio M5 Max | 128 GB | 614 GB/s | MTPLX / Strata-Mac (Q2) | 126 warm cache, 119 | — | Oct 2026 engine survey |
| 2x RTX 3090 + 128 GB | 2x 24 GB VRAM | 936 GB/s each | vLLM TP2 + CPU cold experts, INT8 KV | 101-111 at 131K ctx | up to 2,100 | qwen38-flash-next-3090 |
| RTX 5060 Ti + 5080 | 16+16 GB VRAM | 448 / 960 GB/s | Strata IQ4_XS | 108 | — | Oct 2026 engine survey |
| DGX Spark (GB10) | 128 GB | 273 GB/s | ninfer-gb10 | 60 single-stream; 121 at 4 concurrent | — | NVIDIA forum |
| RTX 5060 Ti 16 GB | 16 GB VRAM | 448 GB/s | Strata IQ3_S | ~48 | — | Oct 2026 engine survey |
| RTX 4090 | 24 GB VRAM + 96 GB DDR5-5600 | 1,008 GB/s | Strata IQ3_XXS | 76.5 warm; 48-67 in real agentic sessions (median ~57) | ~600-1,700 fresh tokens (95% of a long session is KV reuse) | indrasmirror writeup |
| RTX 4090 | 24 GB VRAM + 96 GB DDR5-5600 | 1,008 GB/s | llama.cpp UD-IQ3_XXS, --fit on | 30 at 6K, 22 at 90K, 19.5 at 110K | ~1,360 at 6K, ~905 at 90K | ryan4yin gist |
| RTX 3090 | 24 GB VRAM + 64 GB RAM | 936 GB/s | Strata | ~100-140 (README estimate, ±20%; K8V4 KV measured 99 tok/s at 198K) | ~490-520 tok/s (older engine) | Strata DETAILS.md |
| RTX 5070 | 12 GB VRAM + 64 GB DDR5 | 672 GB/s | Strata | 93.5 Q2_0, 53 IQ3_S (measured) | 2,653 tok/s at 32K (Q2_0, engine 0.1.36) | Strata DETAILS.md |
| Radeon AI PRO R9700 (x2) | 2x 32 GB VRAM + host RAM | 644 GB/s each | Radiance TP2, 4-bit-expert/int8-trunk .rad + MTP |
137 sustained single-stream (replicated on two machines); 140-155 at 200K ctx; 1,103 aggregate at conc16 | ~6,400 tok/s at 26K (author-reported) | L1Techs guide · Reddit report · model card |
| Radeon AI PRO R9700 | 32 GB VRAM + 47 GB RAM | 644 GB/s | Strata HIP, Coder pack (half experts) IQ1_M + MTP | 45.5 at 4K, 53 after the v_perm fix, 60.5 warm (validated 2026-09-30) | 982 at 4K, 1,402 at 16K | Strata AMD_HIP.md |
| Radeon AI PRO R9700 | 32 GB VRAM + 64 GB DDR4 | 644 GB/s | Strata HIP + WMMA, full IQ3_S, 262K ctx (community bench, engine 0.1.43, Ryzen 7 5700X) | 75.5 chat, 82.9 agent-session decode (68.7% cache hit, ~1.8 s/turn TTFT) | 2,009 at 4K, 2,820 at 16K, 2,785 at 32K | Strata PR #2047 |
| Radeon AI PRO R9700 | 32 GB VRAM + host RAM | 644 GB/s | hipfire v0.4.3, gptq3/mq4-xts (engine-author self-reported, Oct 2026) |
36.9-49.4 plain AR on real text (coding 49.4, prose 41.6, code-edit 36.9); 44.3-49.0 prose / 66.1-73.9 code with DFlash --spec auto; synthetic tg128 33-35 understates real text |
6,431 at pp16K (16,384-row chunk default; ~5,400 at the 4,096-row decode-heavy override) | hipfire v0.4.3 release notes |
| 2x Radeon AI PRO R9700 | 2x 32 GB VRAM + 125 GB DDR4 | 644 GB/s each | Strata HIP layer-split, IQ3_S, 262K ctx (community bench, engine 0.1.43, Ryzen 9 5900X, PCIe x8 each) | 108.7 greedy / 101.3 sampled (stock); 115.0 / 106.2 with the gfx12 split-decode switches | +11-12% prompt reads at 4.6K/19K/69K | Strata PR #2629 |
Qwen3.8-27B (dense): decode and prefill by machine¶
| Machine | Memory | Bandwidth | Engine | Decode (tok/s) | Prefill | Source |
|---|---|---|---|---|---|---|
| DGX Spark | 128 GB | 273 GB/s | vLLM NVFP4 + MTP | ~28 single-stream; up to 77.3 with tuned speculative decoding across an 11-config study | — | recipe + harness · 11-config study |
| Ryzen AI Halo | 128 GB | ~256 GB/s | — | no direct 27B number published; dense 70B at 4-bit runs ~5 tok/s on this class of bandwidth, which is the honest anchor | — | — |
| MacBook Pro M5 Max | 128 GB | 614 GB/s | MLX | ~29-30 (labeled estimate) | — | LLMCheck estimate |
| MacBook Pro M5 Pro (16c GPU) | 48 GB | 307 GB/s | Splash 4-bit + DFlash 2 draft | 74 short-prompt; 54 at 32K single-stream; 170 aggregate at 4 concurrent | 363 tok/s at 32K; cached-TTFT 282 ms on 32K replay | model card · launch post |
| Mac Studio M5 Max / M5 Ultra | — | — | Splash | no published Splash numbers on Max or Ultra | — | — |
| Mac Studio M5 Ultra | 96 GB | 1.2 TB/s | — | fits at 4-bit (~16 GB) with room to spare; no direct 96 GB measurement, bandwidth is the same as the 256 GB model | — | — |
| Mac Studio M5 Ultra | 256 GB | 1.2 TB/s | LM Studio (plain decode) | ~55; LLMCheck's bandwidth estimate of 57 landed within 4% | 1,701 tok/s (MacStories) | BGR review |
| Mac Studio M5 Ultra | 256 GB | 1.2 TB/s | oMLX + MTP, long prompts | 48 (long 8-16K prompts slow decode even with MTP) | — | oMLX benchmarks |
| Mac Studio M5 Ultra (256 GB) | 256 GB | 1.2 TB/s | mlx-serve 26.10 + drafter (llmprobe, self-reported) | 245 at 4-bit, 190 at 6-bit, 168 at 8-bit (26.9.6 was 197/134/127 — +66% decode, identical output) | +14% prefill vs 26.9.6 | v26.10.1 release notes — box stated as M5 Ultra 256 GB in the verifying PRs |
| MacBook Pro M4 Max | 128 GB | 546 GB/s | mlx-serve 26.10, 4-bit + DFlash drafter (llmprobe, self-reported) | 104 (was 70-72 with MTP across 26.8-26.9 — +49%) | — | repo benchmarks.md |
| RTX 5090 | 32 GB | 1.79 TB/s | MegaCapybara (hand-written kernels, greedy, memory OC +20%) | 313-540 | ~7.6k at 8K (claimed) | MegaCapybara repo |
| RTX 5090 | 32 GB | 1.79 TB/s | SparkInfer + DSpark drafter (greedy) | 420 code / 265 avg | — | DSpark drafter · engine survey |
| 2x RTX 5070 Ti | 2x 16 GB | 896 GB/s each | capy-flash (TP2 + DFlash2 drafter) | 374 (verified) | 9.2k at 4-16K | capy-flash repo |
| RX 7900 XTX | 24 GB | 960 GB/s | hipfire (RDNA-native Rust, DFlash) | 318 | — | hipfire repo |
| Radeon AI PRO R9700 | 32 GB | 644 GB/s | Lucebox (DFlash2 drafter, block-16, exact-verify = lossless vs plain decode of same weights; stock Unsloth UD-IQ4_XS quant — KL 0.018 / 94% top-1 vs Q8_0, HumanEval & GSM8K at Q8_0 parity) | 208 HumanEval avg, peaks 227 on code, 133 math (up to 300.9 aggregate at 5 concurrent across a 27B + DeepSeek V4 Flash mix); prose accepts fewer drafts — expect lower than code | pp hits 4,500 (community report, ROCm build) | lucebox blog · lucebox scheduler post · Reddit r/ROCm |
| Radeon AI PRO R9700 | 32 GB | 644 GB/s | hipfire (RDNA-native Rust, paired DFlash draft default, v0.4.2/0.4.3, self-reported) | 74.5 at reasoning effort xhigh, 116.7 at medium (was 44.0 / 52.9 without the draft); 107.8 C1 greedy at width-8 serve, 324.6 aggregate at C8 | 3,809 -> 4,921 at 8.2K committed prompt (registry-paired draft) | hipfire v0.4.2 release · hipfire releases |
| Radeon AI PRO R9700 | 32 GB | 644 GB/s | vllm-radiance (MXFP4 W4A8 + DFlash2 spec-7, fp8 KV, 160K ctx, 220 W cap, PCIe 3.0 x4 host) | 63.7 sustained single-stream (peak burst 87); aggregate 117 / 187.8 / 197.8 at c2 / c4 / c8, peak burst 302 | ~2,300-2,600 short prompts; 2,598 at 0 ctx decaying to ~1,780 at 98K | Reddit r/LocalLLM setup+bench |
| Radeon AI PRO R9700 (Windows) | 32 GB | 644 GB/s | WHIRL (native C++/HIP, no WSL; MXFP4 fine-tune, self-reported) | 107-328 — fine-tuned model, not comparable to base-27B rows | up to 2.5x llama.cpp (claimed) | r/ROCm |
| 2x RTX 5090 | 64 GB | 1.79 TB/s each | SGLang fork (finnff, TP2) | 315 | — | engine survey |
| RTX 5090 | 32 GB | 1.79 TB/s | TensorFold (Zig engine) | 305 code / 220 chat; 126 at 215K | 1.9k at 215K (claimed) | TensorFold repo |
| RTX 3090 | 24 GB | 936 GB/s | tqLLM / ExLlamaV3 | 273 / 242 | — | tqLLM perf page · engine survey |
| RTX 5090 | 32 GB | 1.79 TB/s | llama.cpp + DFlash2 drafter | 205 | — | engine survey |
| RTX 5090 | 32 GB | 1.79 TB/s | NInfer nvfp4 | 184-204 (measured) | 12.8k at 7.7K (claimed) | NInfer page |
| RTX 5060 Ti 16 GB | 16 GB | 448 GB/s | tqLLM (3-bit TQ) | 156 | — | tqLLM perf page |
| 2x RTX 5060 Ti | 2x 16 GB | 448 GB/s each | ninfer-tp2 fork (lynx-gt / KimDaeHwan99, TP2) | 126 / 120 | — | lynx-gt/ninfer-tp2-5060ti · engine survey |
| RTX 4090 | 24 GB | 1,008 GB/s | llama.cpp UD-Q4_K_XL + MTP | 109.4 median (102.8-113.2 across runs); 47.0 without MTP; 46.5 at 262K ctx with q4_0 KV | 2,600-3,000 tok/s llama-bench; 1,621 tok/s at 197K depth | ComputingForGeeks · llamabench.ai |
| Radeon AI PRO R9700 (x2) | 2x 32 GB | 644 GB/s each | Radiance TP2, FP8 + DFlash2 drafter | 130 sustained single-stream; 596 aggregate at conc8 (MXFP4 pack: ~200 single-stream, 678 aggregate) | — | L1Techs guide |
| Radeon AI PRO R9700 | 32 GB | 644 GB/s | llama.cpp Vulkan (RADV), Atomic AD-Q6_K-Q5_K + MTP | ~42 at 224K ctx (was 15.3 on Windows without MTP; Linux drivers alone took it to 24.6) | ~2x the Windows baseline | Reddit report |
One engine worth watching on Apple silicon is Splash (Inco AI) — it builds the engine around the model rather than the other way around: a trained DFlash 2 draft (7 tokens/step, verified in one pass), shape-specialized 4-bit kernels, and a fixed memory plan. Its published 27B numbers are M5 Pro only so far. Beware the widely-circulated "Qwen3.8-27B at 144 tok/s on M5 Max" — that figure is a mix-up with Splash's 143 tok/s for the Qwen3.6-35B-A3B MoE on an M5 Pro (debunk). Splash also has hard floors: M3 or newer, macOS 26.4+, and 36 GB minimum unified memory (48+ recommended) — the memory floor is the cache-resident design, not the weights.
The other engine to watch is mlx-serve (ddalcu/mlx-serve) — a native Zig/Metal inference server with a Swift macOS app, OpenAI and Anthropic API compatible, no Python and no Electron, MIT-licensed. It's a young project (781 commits, date-based versioning) but the engineering is serious and the M5 Ultra numbers are the best Flash-Next decode figures published for that chip: 219 tok/s on the mixed 4-8-bit pack with MTP, 227 aggregate across 4 concurrent streams, and 243-300+ on draftable turns where the reply copies the context. Two things I like about how they measure: every number comes from llmprobe --bench-only (warmup discarded, median of 3), and the release-to-release history table is deliberately single-machine — M4 Max only, because mixing columns across hardware poisons the trend. The v26.10.1 release moved 27B decode +66% on M5 Ultra with byte-identical output verified across 18 models, which is the rarest kind of speedup. It also runs GGUF through its own MLX engine now instead of a llama.cpp fallback, and its agent features (mid-run steering, thinking not replayed into long-session context, all conversation images reaching the model) are aimed squarely at the agentic workloads this whole page is about. Caveats: every published number is self-reported, and the project moves fast enough that main is already 235 commits past the late-September release.
The AMD answer to the same problem is Radiance (StillDeadcode/radiance, mirrored on GitHub) — a from-scratch C++/HIP inference server with an OpenAI-compatible API, Apache-2.0, where architectures, kernels and quantisers are .so plugins. Its kernel library (libr4d) is hand-written for RDNA4 (gfx1200/gfx1201), and it started life as a patched vLLM fork before the author ditched the Python/PyTorch stack entirely; the Docker image is 423 MB against vLLM's 13+ GB. Models ship as single-file .rad containers — weights, tokeniser, chat template and drafter in one — and the Flash-Next build keeps the trunk int8 (W8A8) with GPTQ 4-bit codebook experts, ten energy-heavy experts kept bf16, plus MTP and vision, 113.6 GiB, served tensor-parallel across two 32 GB cards with the coldest experts streamed from a pinned host pool. The design premise matches the hardware: an R9700 has more compute than its 644 GB/s of GDDR6 can feed, so Radiance spends its effort shrinking bytes per token — int8/4-bit weights, a drafter (DFlash2 on the dense builds, the model's own MTP head on Flash-Next) accepting 3-4 tokens per pass, everything kept on the card's own memory bus instead of crossing PCIe. Level1Techs ran the whole stack on two machines to prove it replicates and measured 137 tok/s single-stream on Flash-Next and 130 on 27B FP8; a Redditor independently moved a dual-R9700 box from llama.cpp on Windows (31 tok/s, driver timeouts) to Radiance on Ubuntu and landed at 140-155 tok/s at 200K context. Caveats: it's a one-author project, RDNA4 kernels are the only device kernels in the tree (other AMD families need a kernel plugin), and every model uses both cards on one port, so you run one at a time.
Anecdotal configs worth knowing¶
Beyond the measured tables, the Strata community has pushed this model onto a lot of ordinary hardware. These are single reports, not benchmarks — treat them as anecdotal — but they show where the expert-tiering design goes:
- RTX 3060 12 GB + 48 GB RAM (Ryzen 5 3600): Q2_0 ran at ~44 tok/s — a 125B-class model on a 12 GB card from 2021 (writeup).
- 64 GB RAM + 16 GB VRAM: 40-50 tok/s at 256K context reported (summary).
- 64 GB RAM at 256K context with IQ3_S: users confirmed it runs with RAM to spare (Strata issue #406).
- RTX 5090 with the NVFP4 fork: 141 -> 158 tok/s just from widening the MTP draft vocabulary (Strata DETAILS.md).
- RTX 5080 + IQ3_S and an RTX 2070 (Turing) both contributed measured runs to the repo — the engine supports RTX 20/30/40/50 and a long list of Radeon cards (AMD_HIP.md).
- Dual-GPU: Strata does run one model across two cards — a pipeline (layer) split, not tensor parallelism, with the expert arena held once in host RAM and each card caching experts for its own layers (MULTI_GPU.md). No NVLink or P2P needed; x4 slots are fine. Mixed NVIDIA + AMD in one run is not supported — a dual-card Strata box must be same-vendor (
--backend hip --gpus 1,0for a Radeon pair). The AMD split was validated on an R9700 + 9070 XT: with the second card slower than the first, the split was slower (42-43 tok/s vs 51-52 solo) — a mismatched pair doesn't pay. Two matched R9700s split the model to 108-115 tok/s on a Ryzen 9 5900X (PR #2629); no solo number from that same box is published, so whether the matched pair's doubled VRAM expert cache beats a solo card is an open question rather than settled — see the table rows. Radiance, below, is tensor parallel across two cards and does pay off there.
The pattern across every one of these: the binding constraint is RAM capacity for the expert pool, and the second is VRAM for the hot-expert cache. A fast CPU with lots of RAM keeps a small GPU productive.
Media generation: image, video, music¶
Diffusion models are the opposite workload from LLM decode. A denoising step is a dense matmul over the whole network, every step, so media generation is compute-bound, not bandwidth-bound. This is why the M5 Ultra's Neural Accelerators moved image generation 4x over the M3 Ultra while decode only moved 1.5x (NetworkChuck measured both on the same box), and why the DGX Spark — decent tensor cores, but only 273 GB/s and a small GPU — falls ~2x behind a 5090 on the only image models it has published numbers for.
Image generation¶
Four models worth comparing in late 2026: FLUX.2 klein 4B (the small distilled family member — fp8, ~4 GB), FLUX.2 klein 9B (the bigger sibling — note it's 9B, not 12B; the 12B-class model is FLUX.2 [dev], which is much heavier), Qwen-Image 2.1 (7B DiT + Qwen3-VL 8B encoder, open weights, native 2K and alpha), and Z-Image Turbo (Alibaba's 6B Apache-2.0 distilled model — 8-9 steps, tiny, fast, and the community's go-to for interactive generation). Flux.1 dev is kept as the legacy anchor because it's the most cross-published number. All times are 1024x1024 unless noted. Every cell now states the quant and the measured peak memory where the source recorded it — footprint is what decides whether a card can run a model at all, and it varies hugely by quant: Qwen-Image 2.1 is 30.9 GB in bf16, 21.3 GB peak at 8-bit, 15.2 GB at 4-bit.
| Machine | Bandwidth | FLUX.2 klein 4B | FLUX.2 klein 9B | Qwen-Image 2.1 | Z-Image Turbo | Flux.1 dev (legacy) | Source |
|---|---|---|---|---|---|---|---|
| RTX 5090 | 1.79 TB/s, 32 GB | no direct published number | ~2 s distilled / ~35 s base (fp8; ~14 GB peak class) | 19.3 s fp8 (21.3 GB peak) / 19.2 s nf4 (15.2 GB peak), 40 steps | ~3.3 s/image nvfp4, 9 steps (shares VRAM with a resident 27B LLM; only 233 MB free at the tightest config) | ~8 s warm, FP4 Nunchaku (5090-mobile) | lucasgraphic · kgptalkie · wsjwong |
| RTX 4090 | 1,008 GB/s, 24 GB | ~1 s, fp8, 4 steps | ~3.7 s, FP16, 28 steps (site estimate; FP16 needs 29 GB — offload; fp8 fits at 14 GB) | no published run; 4-bit peaks at 15.4 GB so it fits | ~2.3 s, BF16, 8 steps (~13 GB peak) | 11.3 s warm, fp8, 20 steps | localaimaster · willitrunai · smeltcore · ComfyUI #9002 |
| RTX 3090 | 936 GB/s, 24 GB | — | — | — | 6.7 s, BF16, 7 steps (3090 Ti, steady-state) | ~26 s warm, fp8, 20 steps | miroleon benchmark · ComfyUI #9002 · GIGAGPU |
| DGX Spark (GB10) | 273 GB/s, 128 GB | — | — | no published number (original Qwen-Image: 98 s) | — | — | float16.cloud |
| Mac Studio M5 Ultra | 1.2 TB/s, 96/256 GB | 1.77 s, mflux, 4 steps (22.5 GB peak) | — | — | 9.25 s, mflux, 9 steps (37.0 GB peak) | no direct measurement (see head-to-head below) | NetworkChuck repo |
| Mac Studio M5 Max | 614 GB/s, 36 GB | — | — | 109 s Q8 (14 GB proc peak) / 157 s Q4 (11 GB), 40 steps, GGUF | — | ~28 s, 8-bit, 28 steps | kgptalkie · Contra Collective |
| Mac Studio M3 Ultra | 819 GB/s, 512 GB | 7.0 s, mflux, 4 steps (22.5 GB peak) | — | — | 27.9 s, mflux, 9 steps (37.0 GB peak) | 88 s cold | NetworkChuck repo · Olares |
| Mac Studio M4 Max | 546 GB/s | — | — | — | — | 136 s cold | Olares |
| Ryzen AI Halo (Strix Halo) | ~256 GB/s, 96/128 GB | — | — | — | — | ran, poorly (partial pass) | Olares |
The memory column is where the Mac pitch actually lives. Z-Image Turbo peaks at 37 GB — a 24 GB card can only run it quantized or offloaded, while a 96 GB Mac Studio holds it next to a resident 40-50 GB LLM with room to spare. Qwen-Image 2.1 tells the same story in quant form: 30.9 GB bf16, 21.3 GB at 8-bit, 15.2 GB at 4-bit — the 4-bit version fits a 4090 and runs at nearly 8-bit speed on CUDA, but on Apple Silicon the quant math inverts (see below).
Read the step count before comparing rows — mflux's defaults are 4 steps for klein 4B and 9 for Z-Image Turbo, kgptalkie's Qwen-Image 2.1 runs use 40 steps, the ComfyUI Flux templates use 20, willitrunai's klein estimate uses 28, and localaimaster's 4090 klein number is a 4-step distilled run. And note the Mac quirk kgptalkie measured: Q4 is slower than Q8 on Apple Silicon (157 s vs 109 s) because the GPU repacks 4-bit weights every step — the opposite of the CUDA world, where 4-bit is nearly free.
Video generation is its own story (it's MiniMax H3 now, and the bottleneck flips from compute to capacity) — see the dedicated section below. Music generation has no honest cross-hardware benchmark yet: YuE2, MusicGen, ACE-Step and Stable Audio Open are the same compute-bound shape as image gen, CUDA is where they're optimized, and from running YuE2 and MusicGen on a 5090 box a full song is a minutes-scale job — expect the Spark ~2x slower and Macs slower still, but that's reasoning, not measurement.
Three things to read out of the image table:
The Spark's LLM strength does not transfer to image gen. It wins or ties on vision-language and prefill because those are batched matmuls it has tensor-core throughput for, but a diffusion loop is a serial chain of matmuls on a small GPU — 2.1x behind the 5090 on every image test. Its 128 GB does help where the 5090 has to block-swap, but "fits without offload" is not the same as "fast". Video is the exception: H3 is capacity-bound, and there the Spark's memory is the whole story (below).
The M5 Ultra is the one Mac that takes image gen seriously. 1.77 s for a FLUX.2 klein 4B image is genuinely interactive — that's the Neural Accelerators doing dense matmuls, exactly the workload they were added for. But note the model: klein 4B is a small distilled model. On the same-class heavy models (Flux.1 dev), the Macs are still minutes behind CUDA boxes.
(Older video models for reference: LTX-2.5 ran 17.8 s on the M5 Ultra vs 48.4 s on the M3 Ultra — but H3 is the model that matters now.)
Video gen in 2026 is MiniMax H3¶
The model that actually separates hardware now is MiniMax H3 — open weights (Aug 2026), a 33B omni-transformer plus a full Qwen3-VL-32B encoder plus video and audio VAEs, generating video with native stereo sound in one pass (model card). It matters for this comparison because it's a capacity monster: the full pipeline is ~100 GB+ resident in BF16, ~42.5 GB of files even after INT8/NVFP4 repacking. That's a different bottleneck than compute, and it reshuffles the ranking:
| Hardware | Runtime | Workload | Time (type) | Source |
|---|---|---|---|---|
| 8x B300 | SGLang BF16 | 5.17 s @ 1344x768, 50 steps, warm | 19.0 s (latency) | SGLang cookbook |
| 4x H100 | SGLang TP2+Ulysses2 | 5 s @ 1344x768 (topology comparison) | 13.3 s (pipeline latency, not E2E) | SGLang cookbook |
| 4x H200 | SGLang Ulysses4 | 5 s @ 1344x768, 50 steps, warm | 74.4 s (E2E) | SGLang cookbook |
| 2x RTX 5090 | SGLang layerwise offload (26.3 GiB/GPU) | 5 s @ 1344x768, 50 steps | 559.7 s (E2E) | SGLang cookbook |
| DGX Spark (128 GB) | Diffusers INT8 + group offload | 5.17 s @ 768x576 | ~543 s (optimized avg) | community report |
| L40S (48 GB) | optimized raw weights | 10 s @ 960x544 | ~7.2 min | community report |
| RTX 4090 Laptop (16 GB) | ComfyUI repack, 20 steps | 5 s @ 960x540 | 182 s (E2E) | community report |
| Mac Studio M5 Ultra 96 GB | — | repack path (INT8 + offload) fits in unified memory; the open question is MPS throughput, not capacity | no published run | — |
| Mac Studio M5 Ultra 256 GB | — | repack path fits with room; no MLX port, Diffusers-on-MPS is the only path | no published run | — |
Read the workload and the measurement type before the number. The H100's 13.3 s is a pipeline-latency topology comparison, not an end-to-end run, so it is not directly comparable to the H200's 74.4 s E2E even though both are "4 cards at 1344x768" — SGLang itself flags them as different measured paths. The Spark's ~543 s is at 768x576 while the dual-5090's 560 s is at 1344x768, so they're not the same job either.
And note what the community numbers actually represent: the normal way to run H3 on consumer hardware is the ComfyUI repack path — INT8/NVFP4 weights with dynamic loading, which fits a 16 GB laptop GPU and runs comfortably on a 32 GB 5090 or 24 GB 4090. The multi-GPU SGLang recipes are for serving at full BF16 quality, not the desktop experience. So the practical ranking is: any decent CUDA card runs it (the 4090-laptop 182 s at 960x540 is the floor, a desktop 5090 is faster), the Spark runs it at ~543 s with light offload, and the question for each machine is throughput and resolution headroom, not whether it fits. The L40S report is the one to learn from: component placement (keeping the text encoder out of the way during denoise, not streaming the VAE leaf-by-leaf) mattered more than raw card speed.
For the Mac column: no published H3 numbers on Apple Silicon yet. Capacity isn't the blocker on either tier — the repack path is ~42.5 GB of files and unified memory holds it on 96 GB with offload, comfortably on 256 GB. The open question is throughput: no MLX port, Diffusers-on-MPS is the only path, and MPS was marginal for video even on M3/M4 Ultra in Olares' matrix. If H3 matters to your video workflow, that's a real question mark for the Mac and a real point for the Spark and every CUDA box.
The one comparison nobody has run: RTX 3090 vs M5 Ultra¶
There's no published head-to-head, but the pieces line up surprisingly close. Flux.1 dev fp8 at 1024², 20 steps: the 3090 does it in ~26 s warm (ComfyUI #9002, corroborated by GIGAGPU's 0.82 it/s). For the Mac side there's still no direct M5 Ultra Flux.1 dev measurement, but the best anchor is the same generation: an M5 Max does Flux.1 dev 8-bit in ~28 s warm at 28 steps (Contra Collective) — call it ~20 s at 20 steps, and the M5 Ultra's 2x bandwidth plus Neural Accelerators should beat that. (The older M3 Ultra took 88 s cold under MPS (Olares), and the M5 Ultra's measured image-gen speedup over the M3 Ultra is 4.0x (NetworkChuck) — both routes land in the same 15-25 s window.) So: roughly a tie on the same model, with the M5 Ultra pulling ahead on small distilled models (1.77 s on FLUX.2 klein 4B, 9.25 s on Z-Image Turbo — both faster than any 3090 result I can find) and the 3090 pulling ahead on anything that uses CUDA-only tricks — Nunchaku 4-bit, TensorRT, TeaCache — and on video, where H3's repack path is CUDA-only for now. The 3090 is also a third of the price.
What the tables actually say¶
Bandwidth sets the decode ceiling, but engines move 2-3x on the same chip. On the M5 Max, oMLX measured 33 tok/s and MTPLX measured 68-125 on the same model in the same month. On the Spark, the same checkpoint went 17 -> 41.7 tok/s with the MTP recipe and no weight changes. mlx-serve's own release history is the same lesson in one table: the same 27B pack on the same M4 Max went 70 -> 104 tok/s (+49%) in six weeks of engine work, no hardware change. The published tok/s is always an engine-plus-config number; the chip is only half of it.
Multi-GPU only buys speed with tensor parallelism. A useful taxonomy from the October 2026 engine survey: tensor-parallel engines (capy-flash, the ninfer-tp2 fork line, SGLang, vLLM, ExLlamaV3) add the two cards' bandwidth together for a single request; pipeline/layer-split engines (Strata, llama.cpp, the ninfer-3090 line) add memory, not speed (+0-16%). That matches what I measured on my own dual-card Strata box — the split only pays when no single card holds the experts.
Most headline numbers are greedy, code-heavy, and unverified. The survey's own legend is worth stealing: verified-vs-claimed on every row, and the observation that most of the "400+ tok/s" headlines are speculative decoding on code (DFlash2/DSpark drafters give x2-3 on the 27B, MTP x1.3-1.7 on Flash-Next). Read the drafter and the workload before comparing rows — this page's own 437-tok/s edit-turn row is the same effect.
On Flash-Next, quant format trades against engine on the same box. The survey ran Strata IQ3_S at 195 tok/s against Strata NVFP4 at 120 on the same 2x 5070 Ti machine — the 3-bit i-quant wins decode at roughly twice the KL (0.144 vs 0.070, Strata's own numbers). Format is not just a quality dial; it moves speed too.
MTP is the single biggest lever, and it's workload-shaped. Prose drafts well (predictable), code drafts worse, and agent edit turns — where the output copies the input — are the jackpot: sethforprivacy's prompt-lookup drafts hit 272 tok/s on edit turns because the continuation is already sitting in the context. My Strata setup runs the same trick with MTP on the 5090.
On RDNA4 the OS and the engine are worth more than the silicon. The same R9700 hardware measured 15.3 tok/s on llama.cpp under Windows, 24.6 under Linux with the same engine, ~42 with MTP, and 130-155 on Radiance — an order of magnitude between the worst and best config, no hardware change. The Windows number even came with GPU driver timeouts on the dual-card box. If you're buying an RDNA4 card for local inference, the ROCm/HIP engine ecosystem is the thing being evaluated, not just the card.
The M5 Ultra's headline is prefill, not decode. MacStories measured decode +40% over M3 Ultra but prefill +150-250% (16K prompt: 1,143 -> 2,887 tok/s, TTFT 13.9s -> 5.6s), from the Neural Accelerators. For agents that re-read long contexts, that's the gain that shows up in wall-clock time.
96 GB is the awkward tier for Flash-Next. The table-resident MLX builds don't fit and page experts, which is how you get 9 tok/s. The GGUF and MTPLX splits (table on SSD, experts hot) do fit, and the arithmetic says they run well, but nobody has published that benchmark on a 96 GB Studio yet — and that now includes the engine authors themselves. mlx-serve's mixed 4-8-bit pack is ~75 GB resident, which fits 96 GB with room, but every M5 Ultra number they publish (219 tok/s Flash-Next, 245 tok/s 27B) comes from a 256 GB box, stated in the PR bodies behind each release. The 96 GB base config's Flash-Next number is still an open question, and it deserves an independent run before anyone spends $5,499 on the strength of 256 GB results. 128 GB remains the floor where every engine's fast path works without questions.
The 5090 + Strata box is no longer the top of the decode table — and that's the ecosystem working. It still runs 150-200 tok/s on IQ3_S, and the NInfer measurements show what 1.8 TB/s of HBM buys. But the October 2026 survey put Basalt (NVFP4, Strata fork tuned for Blackwell) at 239-479 on a 5090 + 5060 Ti box, fngine at 146-209 on a single 5090, and hand-written single-model kernels (MegaCapybara) at 313-540 on the 27B — all author-reported, all greedy-friendly workloads, all worth reproducing before believing. The Mac's case was never raw decode — it's 96-512 GB of resident room, no slot swapping, and prefill speed.
Related¶
- Using Strata — the expert-tiering engine behind my 5090 numbers
- Using NInfer · Using llama.cpp · llama-slot-proxy