Proxmox VE¶

Proxmox VE is the hypervisor on my home server. It runs the Ubuntu VM that hosts the Docker stack, handles ZFS storage with snapshots, and pairs with Proxmox Backup Server for image-level backups. If you're turning a bare-metal box into a hypervisor, this is the guide.

Installing Proxmox¶

Download the Proxmox VE ISO and burn it to USB (Fedora Media Writer or BalenaEtcher both work), then boot the machine from it.
During install you'll be prompted to name the node — the machine Proxmox is installed on. Use the asset name (e.g. hp-z220) rather than a generic name like proxmox.

DNS¶

Make sure DNS works before doing anything else:

ping google.com

If it fails, set it at Node → System → DNS — Cloudflare (1.1.1.1) is a fine choice.

Repositories¶

Proxmox defaults to the enterprise repositories, which error on apt update without a subscription. Disable both enterprise repos at Node → Updates → Repositories, then Add the two no-subscription repositories.

Storage¶

Pools for VM drives¶

At the node level, look at Disks and create storage by type:

LVM-Thin for single disks storing VM disk volumes
ZFS when using an array (RAID)

Both support snapshots. If the disk you want isn't in the dropdown, go to Disks in the tree, select it, and wipe it — if it can't be wiped because it's mounted, unmount it first (Cockpit makes this easy; see below).

Thin-provisioned ZFS¶

LVM-Thin is obviously thin, but ZFS needs an extra step:

Create the ZFS pool under Disks at the node. I set compression to lz4 and ashift to 12. Starting with a single disk and moving to a mirror later is fine — you can add drives to the mirror with simple commands.
Before creating any VM drives on the pool, go to Datacenter → Storage, click the pool, click Edit, and check Thin provision.

Info

To change compression after the pool exists (only affects new writes):

zfs set compression=lz4 Personal4T

Mounting disks to the host (Cockpit)¶

Proxmox's UI doesn't offer a clean way to mount/unmount disks to the host, so I install Cockpit:

sudo apt update
sudo apt install cockpit
sudo systemctl enable --now cockpit.socket

Cockpit requires a non-root user:

adduser thomas
usermod -aG sudo thomas

GPU passthrough and RAM sizing¶

The GPU VM on the Powerspec box runs a 27B model that keeps its experts resident in RAM. The sizing numbers here are measured, not estimated.

IOMMU groups¶

Read the topology, don't guess:

lspci -tv

The Z890 quirk: only the top x16 slot is CPU-fed. The lower x16-length slots are chipset-fed and x4 electrical. Fine for inference, and each is in its own IOMMU group, so a second card is possible.

Don't trust the manual on slot speeds

The manual says Gen4 x4. The empty ports report Gen1 x4 in LnkCap. Check LnkSta on a populated slot before you plan around it.

The measured RAM numbers¶

Current hardware: 4x32 GiB (128 GiB installed), MemTotal 125.2 GiB.

The firmware/MMIO loss is ~2.8 GiB: the BIOS e820 map carries a 1.4 GiB hole below 4 GB (PCIe MMIO address space for the 5090's large BAR) plus kernel bookkeeping. BIOS-level, identical on any OS, and roughly fixed-size regardless of how much RAM is installed.
The kernel holds ~8.5 GiB in MIGRATE_CMA pageblocks for the integrated Intel GPU (enabled in BIOS, i915 loaded, never used — the box is headless). /proc/meminfo reports CmaTotal: 0 but grep free_cma /proc/zoneinfo and nr_free_cma in /proc/vmstat show the pool. Those pages count as free in every counter but are unspendable by normal host allocations.
Host kernel + PVE + QEMU overhead measures ~3–4 GiB.
Total effective host reservation: ~15 GiB, not the usual "2 GB for the host."
Guest RAM is charged one-to-one against host RSS when ballooning is off. The guest's page cache is host anon memory — a guest that fills its free RAM with cache (Linux does this by default) gets charged for all of it.

The sizing rule:

guest + ~15 GiB host reservation <= 125.2 GiB

Tested: 114000 MB OOM'd (114 + 8.5 + ~4 ≈ 126.5 > 125.2; the OOM census showed 9.3 GiB "free" of which 9.0 was CMA — effective free ~0.3 GiB). 110000 MB is stable, ~13 GiB MemAvailable.

Disabling the iGPU in BIOS should reclaim most of the 8.5 GiB CMA pool. Not yet done.

Ballooning with free-page-reporting is a polite request, not insurance. The host asks the guest for memory back and the guest can say no — and it can't reclaim page cache at all.

2DPC on Arrow Lake¶

Filling all four slots drops the negotiated speed from 6400 to somewhere around 4800–5600, and XMP often won't apply with four sticks. One DIMM per channel gets 6400. Measured on this box with four matched sticks: 4800 MT/s, no EXPO.

The story of how 56 GiB OOM'd the host is in the blog post The 56 GB that wasn't there, and how the 128 GB upgrade still OOM'd at 114 GB is in The 8.5 GB that showed up as free.

Device passthrough and shared directories¶

For an external USB drive that multiple trusted VMs need file-level access to:

Mount the drive on the host at a path, e.g. /mnt/external_drive (Cockpit, or mount + fstab). For NTFS drives install ntfs-3g.
At the Datacenter level, create a Directory Mapping pointing at that path.
On the VM: Hardware → Add → Virtiofs, select the Directory Mapping ID.

Inside the VM:

sudo mount -t virtiofs <mapping-name> /mnt/external_drive

and persist it in /etc/fstab:

<mapping-name>  /mnt/external_drive  virtiofs  defaults,nofail  0  0

Converting a RAID1 array to a ZFS mirror¶

When moving a Linux MD RAID array onto Proxmox, the metadata is still on the drives but the array must be reassembled:

sudo mdadm --detail --scan     # check if the array exists
sudo mdadm --assemble --scan   # reassemble it

The easiest conversion path is to free one disk from the array — the RAID1 keeps running degraded on the remaining drive:

# Mark the disk to remove as failed
mdadm --fail /dev/md0 /dev/sda

# Remove it from the array
mdadm --remove /dev/md0 /dev/sda

# Clear the mdadm metadata so it can be formatted as ZFS
sudo mdadm --zero-superblock /dev/sda

Warning

/dev/md0 is where my array was mounted. If you never mounted the array, check whether you can do this without mounting. Keep careful track of which disk is the old RAID and which is the new ZFS.

After creating the ZFS pool on the freed drive, mount both (the old RAID via Virtiofs) and copy the data across:

sudo rsync -ah --info=progress2 /mnt/md0/ /mnt/zfs/

Expanding a VM disk¶

Resize at VM → Hardware → Hard Disk → Resize (new size must be greater), then expand inside the guest — Proxmox does not do this for you. For an ext4 guest:

lsblk
sudo growpart /dev/sda 1
sudo resize2fs /dev/sda1

Double-check the right commands for whatever filesystem the VM uses.

Known pitfalls¶

Proxmox freezing on large rsync (Intel NIC bug)

If Proxmox freezes during large rsync transfers on older hardware, check the logs after a freeze for e1000e: Detected Hardware Unit Hang. Fix by disabling NIC offloading:

sudo ethtool -K eno1 tso off gso off gro off

To persist it, add to /etc/network/interfaces (replace eno1 with your interface from ip address):

iface eno1 inet manual
    post-up ethtool -K eno1 tso off gso off gro off

Reference: Reddit thread on the e1000e hang.

Proxmox Backup Server (PBS)¶

PBS gives image-level backups of VMs/LXCs to an external drive. The full walkthrough is in Proxmox Backup Server — the short version: mount the drive on the host, create a Directory Mapping, run PBS as a small VM (10 GB disk, 2 GB RAM, 2 CPU) with the datastore mounted via Virtiofs, create a DatastoreAdmin user, add PBS as a Datacenter storage, and schedule a backup job.