Skip to content

Logging (GoAccess → Grafana → Loki)

My log stack went through three stages. GoAccess first — lightweight, real-time analytics per proxy host. Then Grafana + GeoIP for dashboards with country data. Now Loki + Promtail, which is log aggregation with LogQL. The classic ELK stack still works but is heavier than Loki for a home server. I'd pick Loki unless you already have ELK running.

Stage 1 — GoAccess (Nginx Proxy Manager)

Lightweight, real-time analytics. The justsky/goaccess-for-nginxproxymanager image tails NPM's logs:

  goaccess:
    image: justsky/goaccess-for-nginxproxymanager:latest
    restart: unless-stopped
    environment:
      - TZ=America/Denver
    ports:
      - '7880:7880'
    volumes:
      - ./data/npm_data/logs:/opt/log
      # per-host analytics: mount just one proxy host's log
      # - ./data/npm_data/logs/proxy-host-1_access.log:/opt/log/proxy-host-1_access.log

GoAccess can digest all logs, but it can't filter per hostname — for per-host analytics, mount a single proxy-host-N_access.log (NPM writes one per proxy host).

Real IP behind Cloudflare

In the NPM advanced tab, add real_ip_header CF-Connecting-IP; so GoAccess sees the real client IP through a Cloudflare tunnel.

Stage 2 — Grafana + GeoIP

The preconfigured npmGrafStats dashboard adds country-level analytics on top of NPM + GoAccess, fed by a JSON access log with GeoIP fields (the JSON log format is in SWAG). Full setup in the post.

Stage 3 — Loki + Promtail (current)

Follow the official Loki docker-compose (pin the compose file to a version — floating examples go stale). Practical adjustments:

  • Promtail mounts the Nginx logs read-only (:/var/log/nginx:ro) and tails them into Loki.
  • Grafana auto-provisions Loki as the default datasource; anonymous admin is fine for a private LAN instance.
  • Query with LogQL — filter by label, e.g. {job="nginx"}.

This replaced GoAccess for me: structured JSON logs + GeoIP fields make Grafana panels far more useful than GoAccess's fixed analytics view.

FluentBit (the alerting tap)

Alongside Promtail, I run FluentBit as a second tail on SWAG's JSON access log. It doesn't feed Loki — it tails json_access.log and writes to stdout, and Loggifly tails FluentBit's stdout, matches "status":"(401|403)", and pushes a templated Gotify alert. That's the path behind the "Access Denied" pings on the notifications page.

SWAG json_access.log ──> FluentBit (tail, stdout) ──> Loggifly (regex match) ──> Gotify

The compose mounts the log directory read-only and points the tail input at the file inside it:

services:
  fluent-bit:
    image: fluent/fluent-bit:latest
    container_name: fluentbit
    restart: unless-stopped
    volumes:
      - /home/thomas/docker/nginx-swag/swag/log/nginx:/logs:ro
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./parsers.conf:/fluent-bit/etc/parsers.conf:ro

[INPUT]
    Name             tail
    Path             /logs/json_access.log
    Read_from_Head   false
    Refresh_Interval 5
    Parser           nginx_json

[OUTPUT]
    Name   stdout
    Match  *
    Format json_lines

Mount the directory, not the file

This one bit me. The original compose bind-mounted the single file (…/json_access.log:/logs/json_access.log:ro). A single-file bind mount pins the container to one inode. SWAG's logrotate renames json_access.log to .1 and nginx creates a fresh file with a new inode — the container's mount still points at the dead old inode and can't see the new file. FluentBit then errors (inode=… cannot register file /logs/json_access.log … No such file or directory) and re-emits the last line it had, forever. The symptom is Gotify flooding with byte-identical 401/403 alerts, all the same old log line, landing on the same second (the flush tick). Restarting FluentBit "fixes" it until the next rotation, which is the trap.

The fix is to mount the directory (…/log/nginx:/logs:ro) so the container sees the recreated file, and add Refresh_Interval 5 so in_tail re-scans and re-attaches even if inotify misses the rename. Same family as the rclone lesson on Backups: atomic-rename log/config files need a directory mount, never a single-file bind.

To verify the tap end-to-end without waiting for a real attack, append a synthetic 403 line to the live json_access.log and watch it reach FluentBit stdout, then Loggifly, then a new Gotify row. A plain curl through SWAG often returns 400, which won't match the (401|403) regex — the synthetic append is the reliable probe.

Which to pick

Stack Strength Cost
GoAccess Zero-config real-time analytics Fixed views, no per-host filter
Grafana + GeoIP Country dashboards, custom panels Needs JSON logging
Loki + Promtail Full aggregation, LogQL, retention More moving parts
ELK Mature Heaviest footprint

See also