Skip to content

The holy grail of self-hosted notifications

In this post, I'm going to show you how to get amazing notifications from each of your self hosted serverices.

Loggifly Notifications

Install Gotify

We can reference the documentation (https://gotify.net/docs/install) to get a started docker-compose.yml file, but we'll add some environmental variables to it. Note the environmental variables that we'll want to include, https://gotify.net/docs/config#environment-variables.

docker-compose.yml
    services:
      gotify:
        image: gotify/server
        container_name: gotify
        ports:
          - 2212:80
        volumes:
          - './data:/app/data'
        # to run gotify as a dedicated user:
        # sudo chown -R 1234:1234 ./gotify_data
        user: "1000:1000"
        environment:
          GOTIFY_SERVER_PORT: 80
          GOTIFY_SERVER_KEEPALIVEPERIODSECONDS: 0
          GOTIFY_SERVER_LISTENADDR: 
          GOTIFY_SERVER_SSL_ENABLED: false
          GOTIFY_SERVER_SSL_REDIRECTTOHTTPS: false
          GOTIFY_SERVER_SSL_LISTENADDR: 
          GOTIFY_SERVER_SSL_PORT: 443
          GOTIFY_SERVER_SSL_CERTFILE: 
          GOTIFY_SERVER_SSL_CERTKEY: 
          GOTIFY_SERVER_SSL_LETSENCRYPT_ENABLED: false
          GOTIFY_SERVER_SSL_LETSENCRYPT_ACCEPTTOS: false
          GOTIFY_SERVER_SSL_LETSENCRYPT_CACHE: certs
          # GOTIFY_SERVER_SSL_LETSENCRYPT_HOSTS: [mydomain.tld, myotherdomain.tld]
          # GOTIFY_SERVER_RESPONSEHEADERS: {X-Custom-Header: "custom value", x-other: value}
          # GOTIFY_SERVER_TRUSTEDPROXIES: [127.0.0.1,192.168.178.2/24]
          # GOTIFY_SERVER_CORS_ALLOWORIGINS: [.+\.example\.com, otherdomain\.com]
          # GOTIFY_SERVER_CORS_ALLOWMETHODS: [GET, POST]
          # GOTIFY_SERVER_CORS_ALLOWHEADERS: [X-Gotify-Key, Authorization]
          # GOTIFY_SERVER_STREAM_ALLOWEDORIGINS: [.+.example\.com, otherdomain\.com]
          GOTIFY_SERVER_STREAM_PINGPERIODSECONDS: 45
          GOTIFY_DATABASE_DIALECT: sqlite3
          GOTIFY_DATABASE_CONNECTION: data/gotify.db
          GOTIFY_DEFAULTUSER_NAME: admin
          GOTIFY_DEFAULTUSER_PASS: admin
          GOTIFY_PASSSTRENGTH: 10
          GOTIFY_UPLOADEDIMAGESDIR: data/images
          GOTIFY_PLUGINSDIR: data/plugins
          GOTIFY_REGISTRATION: false

Info

Make sure to go ahead and create the /data directory.

Go ahead and create a new user with admin privilege, sign out and back in as the new user, and delete the initial admin user.

Send Test Notification

Go ahead and create a test application. The new application will have a token.

You can test this with a simply curl request

curl -X POST "http://192.168.5.8:2212/message?token=AGGLxv20ZmW_vaS" \
  -d title="hello world" \
  -d message="first message"

Install Apprise

Apprise will be essentially for a number of services that may not support Gotify directy to the same extent. Apprise is easily configured with Gotify to send notifications through Gotify.

Let's start with just the basic docker compose file that is found in the documentation (https://github.com/caronc/apprise-api?tab=readme-ov-file#docker-compose-examples).

docker-compose.yml
services:
  apprise:
    image: caronc/apprise:latest
    container_name: apprise
    ports:
      - "8000:8000"
    user: "${PUID:-1000}:${PGID:-1000}"
    environment:
      APPRISE_STATEFUL_MODE: simple
      APPRISE_WORKER_COUNT: "1"
      APPRISE_ADMIN: "y"
    volumes:
      - ./config:/config
      - ./plugin:/plugin
      - ./attach:/attach

Make sure to go ahead and create the bind mount folders before running the compose file.

Testing Apprise with Gotify

Go ahead and go to the web application at your binded port. Note the Config ID, we'll use this later.

Go to the configuration tab and set the format to YAML. Let's add a url:

urls:
  - gotify://192.168.5.8:2212/AGGLxv20ZmW_vaS:
    - tag: homelab

Tip

Apprise supports all kinds of notification endpoints. For a gotify http url we use gotify://<host-name>:<port>/app-token. For https, use gotifys.

The tag is very important, it allows you to send notifications to a particular tag, which is convenient so you don't have to use app ids and it allows you to have multiple notification providers for a tag.

Let's go over some of the main ways to send a notification through apprise.

Stateless solution

This just means that we're essentially disregading the yaml config we just set up and we are manually passing a url or using a default url specified by the APPRISE_STATELESS_URLS env variable. (Documentation: https://github.com/caronc/apprise-api?tab=readme-ov-file#stateless-solution)

curl -X POST \
    -F 'urls=gotify://192.168.5.8:2212/AGGLxv20ZmW_vaS' \
    -F 'body=test message' \
    -F 'title=test title' \
    http://192.168.5.8:2213/notify

Stateful solution

With this endpoint /notify/{KEY}, just add our Config ID, we noted before. Good chance the default for you is just "apprise". We can also now use a tag

curl -X POST \
    -F 'tag=test' \
    -F 'body=test message tag' \
    -F 'title=test title' \
    http://192.168.5.8:2213/notify/apprise

Note that all apprise endpoints are insecure by default. We'll come back to this.

Loggifly Installation

Loggifly will take care of notifications for several apps. It can monitor this system out logs of a docker container which means we don't have to rely on any kind of built in web hook. Super convenient. You'll want to install loggifly everywhere you have docker, i.e. if you have multiple VM's, you'll want loggifly on each (it's lightweight). Grab the docker compose file from the docs (https://clemcer.github.io/LoggiFly/guide/getting-started#docker-compose).

Info

Use the tecnative/docker-socket-proxy compose file.

Loggifly Config

Create an app in Gotify for every app that we're going to monitor via Loggifly. Loggifly only really supports stateless calls to apprise without any body parameter support so we won't be able to utilize tags here.

Here are some examples

config.yaml
containers:
  immich_server:  # Exact container name
    apprise_url: "gotify://192.168.5.8:2212/{GOTIFY_ID_IMMICH}" 
    notification_title: "Failed Login Attempt"  
    keywords:
      - regex: \bFailed login attempt for user\b

  vaultwarden:
    apprise_url: "gotify://192.168.5.8:2212/{GOTIFY_ID_VAULTWARDEN}" 
    notification_title: "Failed Login Attempt"  
    keywords:
      - regex: '(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}).*Username or password is incorrect. Try again. IP: (?P<ip_address>\d{1,3}(?:\.\d{1,3}){3}). Username: (?P<email>[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})'
        template: '🚨 Failed login!\n📧 Email: ''{email}''\n🔎 IP Address: {ip_address}\n🕐 {timestamp}'
        hide_pattern_in_title: true # Hide full regex pattern in notification title

  jellyfin:
    apprise_url: "gotify://192.168.5.8:2212/{GOTIFY_ID_JELLYFIN}" 
    notification_title: "{keywords} found in {container} logs"  
    keywords:
      - regex: "Authentication request.*denied"
        notification_title: "Failed login attempt"

      - regex: "Authentication request.*succeeded"
        notification_title: "Successful login attempt"

  mealie:  # Exact container name
    apprise_url: "gotify://192.168.5.8:2212/{GOTIFY_ID_MEALIE}" 
    keywords:
      - regex: \b401 Unauthorized\b

  grampsweb_celery:
    apprise_url: "gotify://192.168.5.8:2212/{GOTIFY_ID_GRAMPS}" 
    notification_title: "New User Registration Request"  
    keywords:
      - regex: send_email_(confirm_email|new_user)


# Optional. These keywords are being monitored for all configured containers. 
global_keywords:
  keywords:
    # - failed
    # - critical
    - unauthorized

notifications:     
  apprise:
    url: "gotify://192.168.5.8:2212/{GOTIFY_ID_LOGGIFLY}"    # Default Loggifly URL

There's probably other containers that are good for loggifly to monitor logs, but quite often the container logs don't give you the relevant details you actually want to monitor.

Radarr, Sonarr, Lidarr

The *arr applications have built it notifications which is awesome. The tricky thing is that these are often run in a Gluetun vpn network that doesn't have access to the local network. I kind of prefer keeping Prowlar in the glueutn network even though some people are comfortable only having their torrennt client in gluetun. But since prowlar needs bidirectional communiication with the other *arr apps, we really need these in the same network.

With this in mind, we need a publicly exposed endpoint to send out notifications, but of course, I want this to be zero trust protected. Note that the *arr apps support Apprise and Gotify, but only with Apprise do we have the option to pass a HTTP Basic Auth header. Since I already have pangolin set up, this is great because Pangolin allows for Basic auth to authenticate with an SSO protected resource.

Creating gotify apps

Create an app in Gotify for each arr application. Now add these to your Apprise config:

urls:
  - gotify://192.168.5.8:2212/{radarr-id}:
    - tag: radarr
  - gotify://192.168.5.8:2212/{sonarr-id}:
    - tag: sonarr
  - gotify://192.168.5.8:2212/{lidarr-id}:
    - tag: lidarr

Again, I really like just referencing apprise tags when I can, so you can give these individual unique tags.

Creating public Apprise domain and configure Arr apps

Create an apprise.domain.com public dns record and point this to a corresponding Pangolin resource that is SSO proected. In Pangolin, enable SSO, then enable Header Auth. Give it a username and password. Keep these credentials handy as we'll use them for the arr apps.

In each arr app go to settings -> Connect, add Apprise, put your public domain https://apprise.domain.com, use your Apprise Configuration Key, (i.e. Config ID, default is apprise), and then set the apprise tag, i.e. radarr. Again, I prefer using a tag over providing a stateless URL because it is much more dynamic.

Using Cloudflare instead of Pangolin

I focused on Pangonlin in this section, but there is an easy way to accomplish the same with cloudflare using a WAF security policy. Create a tunneled domain like apprise.domain.com. We're not going to use zero trust for this, because there is not an option for a bypass rule based on Header, so instead we're going to use a security rule. Go to the domain then go to Security -> Security Rules. Create a custom rule for Header | authorization | does not equal | {base64 encoded credentials}.

To get the encoded credentials simply run this in a terminal

echo -n 'username:password' | base64

Hit the And button, and set the hostname to your apprise domain and set the Action to block.

Jellyseer

We absolutely need notifications for request status from jellyseer. With jellyseer, we can use gotify directly (it's not in the gluetun network), or we can use Apprise using the Webhook function.

The direct Gotify method is particularly easy. Just put in the server url for gotify and create an application for Jellyseer and put the otken in the toiekn field. That's it.

For Apprise, use the Webhook, which also provides an Authorization header in the case that apprise is behind basic auth. Set the URL to the stateful url, i.e. http://192.168.5.8:2213/notify/apprise (apprise is my Config id), and modify the JSON paylog to include the tag for jellyseer

  "tag": "jellyseer"

Beszel

Beszel is an amazing server monitoring tool and can send customized notifications based on server stats (memory/cpu/disk usage etc.)

Go to Settings -> Notifications. And add a gotify url.

https

This seems like a bug in either Bezsel or it's Shoutrr provider, but even if you use gotify://<ip:port>/<gotify-app-id> it always tries to send an https request even though it SHOULD by convention send http for gotify and https for gotifys. So you will want to make sure you have an https domain for gotifiy, a local dns entry, etc.

Based on the above warning, your gotify domain in beszel should be gotifiy://go.domain.come/<gotify-app-id>. The gotify app id is, of course, the app id you create for bezsel in gotify.

Authentik

Ok, this one is a bit of a doozy but pretty amazing after setting it up.

Configure property mappings for Webhook body and header

Let's create the webhook header first since this is simple. Assuming this is running on a VPS, we need that basic auth header I mentioned before to authenticate with Pangolin or Cloudlfare.

Go to Customization -> Property Mappings -> Create -> Webhook Mapping. Give it a name of Apprise Authorization Header, and the expression should be

return {
  "Authorization": "Basic {base64 encoded username:password}"
}

Now for the body. I'll give you two version, a raw payload, and a parsed sexy payload. You see, the authentik notification body is itself a json string, that we can parse, giving use the ability to format the notification body as we like.

Let's start with just the raw body. Create anotoher Webhook Mapping, give it a name of Apprise Notifiication Body.

Set the expression to:

return { 
  "title": "Authentik: Login Event", 
  "body": request.context['notification'].body, 
  "type": "info", 
  "tag": "auth" 
  }

Note that we're going to be sticking with the stateful solution, and pass a tag with the payload, as again, this is my preference since it's much more flexible. When it comes to parsing the notification body, authentik is kind of weird, because the body is not a json object, but a serialized dict, so we need to dected the notification type (i.e. login for successful login and login_failed for failed login). Note that I've taken into account null checks for fields such as auth provider (if Google was used for example).

Apprise Notification Body (Sexy version) 
import ast

notification = request.context['notification']
raw_body = notification.body.strip()

# Detect if this is a successful or failed login
if raw_body.startswith("login:"):
    event_type = "login"
    raw_body_clean = raw_body[len("login:"):].strip()
elif raw_body.startswith("login_failed:"):
    event_type = "login_failed"
    raw_body_clean = raw_body[len("login_failed:"):].strip()
else:
    event_type = "unknown"
    raw_body_clean = raw_body

# Try to parse the dict
try:
    body_dict = ast.literal_eval(raw_body_clean)
except Exception:
    body_dict = None

# Extract relevant payload info
if body_dict:
    payload = body_dict
    source = payload.get('source', {})
    asn = payload.get('asn', {})
    geo = payload.get('geo', {})
    http_request = payload.get('http_request', {})

    # For failed logins, the attempted username is inside payload
    attempted_username = payload.get('username', None)

else:
    payload = {}
    source = asn = geo = http_request = {}
    attempted_username = None

# Decide what to show as "user"
if event_type == "login" and getattr(notification, "user", None):
    user_obj = notification.user
    user_str = f"{user_obj.username} ({user_obj.email})"
elif attempted_username:
    user_str = f"{attempted_username} (failed login)"
else:
    user_str = "Unknown user"

# Format the message
formatted_body = f"""
**👤 User**
{user_str}

**🔐 Source**
• Provider: {source.get('name', 'N/A')}
• Type: {source.get('model_name', 'N/A')}

**🌐 Network**
• ASN: {asn.get('asn', 'N/A')}
• Org: {asn.get('as_org', 'N/A')}
• Network: {asn.get('network', 'N/A')}

**📍 Location**
• City: {geo.get('city', 'N/A')}
• Country: {geo.get('country', 'N/A')}
• Continent: {geo.get('continent', 'N/A')}

**🧾 Request**
• Method: {http_request.get('method', 'N/A')}
• Path: {http_request.get('path', 'N/A')}
• Request ID: {http_request.get('request_id', 'N/A')}

**💻 Client**
{http_request.get('user_agent', 'N/A')}
"""

# Return the webhook payload
return {
    "title": f"Authentik: {'Login Failed' if event_type=='login_failed' else 'Login Success'}",
    "body": formatted_body,
    "type": "warning" if event_type=='login_failed' else "info",
    "tag": "auth"
}

Create the notification transport

We need to create the transport for this notification. Go to Events -> Notification Transports. Note that you already have a default-email-transport for email notifications for example. We need to create a transport for our Apprise url. Hit Create.

Give it a name like apprise-transport.

Set the mode to Webhook (generic).

The URL should be https://apprise.domain.com/notify/apprise. Again, this is the stateful end point with the Config ID at the end.

Webhook Body Mapping -> select the Apprise Notification Body we created.

Webhook Header Mapping -> select the Apprise Authorization Header we created.

Create the notification rule

Now we need to create the actual notification in Events -> Notification Rules -> Create. You can call it login-event-notification. Set the group to authentik Admins. Select the apprise-transport that we previously created. Optionally, go ahead and select default-email-transport if you want to receive email notifications too (assuming you have smtp configured already).

Great, now expand the rule, we need to add and bind policies for both a login success and a login failure. Click Create and bind Policy and choose Event Matcher Policy. You can call this Login Success Policy, and choose the Action of Login. The othe fields can be left blank. Do the same thing for a Login Failed Policy but choose the Action of Login Failed.

Authentik Notifications

Comments