Skip to content

Backups

I run two backup layers because they do different jobs. Kopia handles file and snapshot backups — deduplicated, browsable snapshots of app data. Cron dump containers handle actual database backups. Backing up a live database volume mount can produce corrupted backups, because the container has to be stopped first and 24/7 containers never are. Dumps are the correct pattern for databases. For whole-VM image backups, use Proxmox Backup Server.

Kopia — file/snapshot backups

Attach an external drive (NTFS if you want Windows compatibility for restores) and mount it permanently:

lsblk -o NAME,MOUNTPOINT,UUID,FSTYPE,SIZE   # note the UUID
sudo nano /etc/fstab

UUID=3ED8D736D8D6EAE3  /mnt/external_drive  ntfs-3g  defaults,uid=1000,gid=1000,nofail  0  2

sudo mount -a          # umount first if it complains about already-mounted

Docker compose (web UI on 51515):

services:
  kopia:
    image: kopia/kopia:latest
    container_name: Kopia
    restart: unless-stopped
    ports:
      - 51515:51515
    command:
      - server
      - start
      - --disable-csrf-token-checks
      - --insecure
      - --address=0.0.0.0:51515
      - --server-username=${USERNAME}
      - --server-password=${SECRET_PASSWORD}
    environment:
      KOPIA_PASSWORD: ${SECRET_PASSWORD}
      USER: ${USERNAME}
      TZ: America/Denver
    volumes:
      - /mnt/external_drive/kopia/kopia-repo:/repository   # repository lives on the external drive
      - /mnt/md0/apps:/apps:ro                             # source data, READ-ONLY
      - /mnt/md0/restore:/restore
      - ./app/config:/app/config
      - ./app/cache:/app/cache
      - ./app/logs:/app/logs
      - ./data:/data:ro
      - ./tmp:/tmp:shared

Key points:

  • The repository path inside the container is /repository — create the repo in the UI pointing there.
  • Mount source data :ro — Kopia should read, never touch, the originals.
  • Create a snapshot per source folder (e.g. /apps/immich-app); hit Estimate first to confirm the mount resolves and the size looks right.

Database dumps (cron containers)

For live databases, run a sidecar container that dumps on a schedule:

Both run mysqldump / pg_dump on a cron interval into a mounted backup folder, so you get consistent, restorable SQL dumps without ever stopping the database container. Point the backup volume at the same external drive Kopia uses, and set the cron schedule + retention in the container env (see each repo's usage docs).

restic → OneDrive (cloud layer)

The second cloud engine, deliberately different from Kopia so the off-site copy doesn't share an engine with the on-site copy. Encrypted, deduplicated, incremental snapshots of Immich media plus daily DB dumps to a OneDrive Business drive, restic as the engine over a private unix socket served by rclone serve restic. Everything is Docker Compose.

The repo URL is the gotcha that took debugging:

RESTIC_REPOSITORY=rest:http+unix:///run/restic-sock/restic.sock:/

restic rest backend with the http+unix scheme. A bare unix:/// is invalid. Trailing slash because rclone serves the remote path at the HTTP root.

rclone config must be a directory bind mount

./rclone-config → /config/rclone, not a single file. rclone saves atomically via rename; a single-file bind mount can't be renamed in-container, so you get EBUSY: device or resource busy and the OAuth token never saves. And the rclone image runs as root — a re-auth writes rclone.conf root-owned and locks you out of your own creds. Pin the rclone services to user: "1000:1000".

The password file is the encryption key

restic.password is the repo encryption key. Losing it loses all backups, irrecoverably. Keep an off-box copy. Do not hand-edit the file to change the password — the password is bound to the keys encrypted inside the repo, editing it orphans the old key and the repo locks. Change it inside restic: restic key add new, verify, restic key remove old.

What gets backed up: media originals, uploads, and Immich's own daily DB dumps (no manual pg_dump needed). Excluded: thumbs/** and encoded-video/**, regenerable caches, about 74 GB saved per snapshot.

Probe the repo with restic snapshots, never restic init — init exits fatal when the repo already exists, so an until restic init loop never reaches the backup.

The story of the month this path ran dark is in The stale mount that ate half the backup windows.

PBS vzdump and the password file

PBS authenticates from each node to the backup server with a password file at /etc/pve/priv/storage/<storage-name>.pw. If a node doesn't have it, the storage shows inactive — not an auth error on the UI, just inactive. That's PVE saying "I can't authenticate to my backup server," which is exactly what you want to know before a backup silently fails.

The watchdog checks this per node. The false alarm that shaped the debounce design is in The false alarm that became the watchdog.

The live vzdump job for the GPU VM is snapshot mode at 03:00 UTC, VMID 100, on pbs-backup-local. Full detail on the Proxmox Backup Server page.

Which layer for what

Data Tool Why
App files (Immich uploads, documents) Kopia Deduplicated snapshots, browsable restore
Databases Dump containers Consistent dumps; volume copies of live DBs can corrupt
Whole VMs/LXCs PBS Image-level, verified snapshots

See also