Two R9700s, one Rocket Lake board, and the PCIe atomics that went missing¶
I moved my i7-11700 out of the Dell XPS and onto a used Z490 AORUS MASTER late one night, bolted in two Sapphire Radeon AI PRO R9700 TOPs, and got exactly what I paid for: every device on the bus, both cards reporting 32 GB of VRAM, idle temps in the mid-20s. Then every single attempt to load a real model into VRAM killed the machine. It took the entire night to find out why, and the answer was not in any driver, engine, or firmware log I expected.
What failing looked like¶
The build itself was boring: XPS out of the closet, 11700 onto the AORUS MASTER, the two R9700s from Micro Center, an RM1000x. Fedora came up on the new board, my KVM VMs autostarted, lspci was clean. Both cards showed up in the kernel with full 32 GB BAR windows once I turned on Above-4G Decoding and Resizable BAR in the BIOS.
The trouble started the moment Radiance (or Strata — it honestly did not care which) began the model-load phase. One minute the engine was printing placement plans, the next the monitor was black and the machine needed a power button. The kernel transcript of each death was identical:
amdgpu 0000:04:00.0: [gfxhub] page fault (src_id:0 ring:40 vmid:8)
amdgpu 0000:04:00.0: MES(1) failed to respond to msg=REMOVE_QUEUE (x6)
amdgpu 0000:04:00.0: GPU reset begin!. Source: 3
amdgpu 0000:04:00.0: MODE1 reset
amdgpu 0000:04:00.0: GPU reset succeeded, trying to resume
amdgpu 0000:04:00.0: VRAM is lost due to GPU reset!
Sometimes the recovery itself failed, and that is worse than a hang — the card comes back on the bus but its security processor never re-initializes:
amdgpu 0000:04:00.0: PSP load kdb failed!
amdgpu 0000:04:00.0: PSP resume failed
amdgpu 0000:04:00.0: resume of IP block <psp> failed -62
After that, HIP sees zero usable GPUs and only a reboot clears it. Two of the failures took the whole machine down — no SSH, frozen desktop, hard power cycle.
What it wasn't¶
I spent hours eliminating suspects, and I want to record how many of them there were, because none of them was the answer:
- Not the cards. Radiance ships a self-test (its engine is verified by its own kernel-correctness suite —
rad-kbench): 1468 recorded kernel cases replayed against reference values, strict mode. 1468 passed on both cards. Idle, VRAM allocation, and basic GPU work were fine all night. - Not either card specifically. Both died with the identical fault, solo and alone, in both slots.
- Not the app. Strata hung with the same kernel signature; Radiance hung with the same kernel signature; llama.cpp-style reports elsewhere matched too.
- Not kernel version (within the drivers I had): 7.2.11 and 7.2.15 both failed identically.
- Not the IOMMU:
iommu=ptandintel_iommu=offboth failed identically. - Not ROCm runtime knobs:
HSA_SVM_DEFAULT=0,HSA_ENABLE_SDMA=0— the fault address moved but the fault stayed. - Not memory placement or context size: embedding table on-GPU instead of pinned host RAM, and shrinking the context from 200k to 40k so the KV cache fit entirely in VRAM with no host spillover — same fault, same stage, wild virtual address 29–64 TiB away from any real buffer.
The tell I kept missing
The fault report itself said the GPU was trying to write at an address it did not own, and the engine kept blaming host-pinned staging buffers. Every suspect I chased was a memory mapping theory. The actual problem was one layer lower, in the PCIe fabric underneath all of it.
The actual bug: PCIe atomics on Rocket Lake¶
Around 1 AM I searched for the exact kernel tail and landed on ROCm/TheRock issue #9279. Same signature, same PSP ... failed -62 transcript — and a commenter had posted, the day before I hit it, a bare-metal A/B experiment on a Rocket Lake host with two R9700s:
| Arm | Boots | Result |
|---|---|---|
| PCIe AtomicOps enabled (stock) | 3 | every boot failed on its first workload |
| AtomicOps blocked (Egress Blocking, DevCtl2 bit 7) | 4 | 4/4 passed, no measurable performance change |
Their minimal reproduction is a few dozen lines of HIP that do nothing but shader atomics to host memory — and on that board it stalls the GPU at 100% busy and ~22 seconds later the card drops off the bus. An AMD engineer replies in the thread:
"all three reports with the same fail signature are on Rocket Lake... you're good to continue with them disabled while we continue to investigate what's causing the conflict here."
The mechanism: PCIe AtomicOps (FetchAdd/CAS packets generated when GPU shaders do system-scope atomics on host memory) are optional for root complexes, and Intel's client root complexes of that era handle them badly — completions get lost with no AER error, no timeout, nothing logged. LLM inference engines are now exactly the workload that lives on those instructions: lockstep handshakes, atomic spins, shared counters across host and GPU. The whole MES/REMOVE_QUEUE/MODE1 storm I chased all night is aftermath, not cause. My i7-11700 is Rocket Lake. The bug was the CPU's PCIe root ports, and the "broken" R9700s were the victims.
The fix¶
Three pieces, and they have to happen in this order because the kernel will fight you otherwise.
Secure Boot has to be off in the BIOS. With Secure Boot on, Linux forces kernel lockdown and every PCIe config write from userspace returns EPERM — and the lockdown=none kernel parameter is ignored while lockdown is forced.
Then the actual workaround: set the AtomicOp Egress Blocking bit (Device Control 2, bit 7) on each card's on-card PCIe switch upstream port — the R9700s carry an internal switch, and that's the port, not the GPU endpoint — before amdgpu probes, so the driver sees the block and tells the runtime to take its no-atomics code paths:
# /etc/modprobe.d/pcie-atomics-block.conf
install amdgpu /usr/sbin/setpci -s 02:00.0 CAP_EXP+0x28.w=0480 ; /usr/sbin/setpci -s 05:00.0 CAP_EXP+0x28.w=0480 ; /sbin/modprobe --ignore-install amdgpu $CMDLINE_OPTS
The ordering bit is the trap: Fedora loads amdgpu from the initramfs, before any modprobe config exists, so the hook above would never run. Keep amdgpu out of the initramfs and it cold-plugs in the real root, where the hook fires first:
followed by sudo dracut -f. My switch upstream ports are 02:00.0 and 05:00.0; lspci shows them as "Navi 10 XL Upstream Port of PCI Express Switch". The value 0480 is the stock 0400 plus the blocking bit.
Three checks confirm it took, all of which I now see on every boot:
$ sudo setpci -s 05:00.0 CAP_EXP+0x28.w
0480
$ sudo dmesg | grep "atomic ops"
amdgpu 0000:04:00.0: PCIE atomic ops is not supported
amdgpu 0000:07:00.0: PCIE atomic ops is not supported
That second one is the driver itself reporting it fell back to no-atomics paths — the exact state the upstream A/B's passing arm was in.
Proof of life¶
The next attempt loaded the 27B model without incident and served it — first completion, asked for a one-sentence answer:
"I'm Qwen. One fun fact: I don't have direct access to the exact machine I'm running on..."
with a clean finish_reason: stop. Two naive end-to-end curls (wall-clock divided by total completion tokens including reasoning tokens, so a pessimistic figure) measured ~106–108 tokens/s decode for Qwen3.8-27B in fp8 on a single card at 40k context. The kernel log stayed completely silent — no page faults, no resets, no MES noise. And because the fix lives in a modprobe hook plus initramfs config rather than a hand-run command, the next reboot came up healthy with the same three checks passing.
One honest caveat
Blocking AtomicOps costs nothing for single-card inference — the upstream A/B measured no performance change, and neither did mine. But multi-GPU collective libraries (RCCL and friends) are known to want atomics, so my dual-card tensor-parallel tests on this board are the one place this workaround might bite. Single card: solved. Dual-card collective traffic: still an open question I intend to answer next.
The rest of the plan from that night — Flash-Next MoE streaming its cold experts from pinned host RAM onto this box, benchmarking against the 5090 machine, and pointing the voice pipeline at it for a day — goes on the Using Strata and hardware comparison pages once the numbers are real.
If you're putting RDNA4 cards on an Intel client board of that vintage and ROCm workloads die at load time with MES storms and page faults against pinned host memory, check setpci CAP_EXP+0x28.w on the switch upstream ports before you RMA anything. The cards were never the problem.