> ## Documentation Index
> Fetch the complete documentation index at: https://docs.augustin.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

Health monitoring across three layers: is the container running, is the app internally functional, and is it reachable externally.

## Tools

* **Homepage** — dashboard with container status, internal site monitors, and per-service API widgets. Talks to the swarm API via `docker-socket-proxy_dockerproxy` on the `swarm` overlay, so it sees services scheduled on both nodes.
* **`tests/check-endpoints.sh`** — on-demand external HTTP check against all public URLs (run from anywhere). There is no continuous external prober; critical alerting is moving to a small Mastra scheduled health-check → ntfy on the endpoints that matter.
* **Dozzle** — log aggregation across both nodes (`mode: global` swarm service).
* **Beszel** — self-hosted system metrics dashboard at [https://beszel.augustin.ai](https://beszel.augustin.ai). Hub is a swarm stack on debian-1; agents run as plain compose on each node (swarm drops `devices:` which breaks SMART). Auto-registers via universal token. Telegram alerts via `@beszel_am_bot` (Shoutrrr URL configured per-user).

## Coverage Matrix

| Service                | Container Status | Internal Functionality                                                                                                | External Reachability                         |
| ---------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| aiclient2api           | Homepage         | Docker healthcheck (`healthcheck.js`) + `kiro-cli whoami` heartbeat from `sync-kiro-token.py` (catches OAuth wedging) | check-endpoints                               |
| audiobookshelf         | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| bazarr                 | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| bulwark                | Homepage         | `/api/health` via healthcheck                                                                                         | check-endpoints                               |
| calibre-web            | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| chaptarr               | Homepage         | Docker healthcheck                                                                                                    | check-endpoints                               |
| cloudflared            | Homepage         | Cloudflare API widget (external)                                                                                      | — (implicit; if down, every public URL fails) |
| docker-socket-proxy    | Homepage         | —                                                                                                                     | — (internal only)                             |
| dozzle                 | Homepage         | —                                                                                                                     | check-endpoints                               |
| draw                   | Homepage         | Port check only                                                                                                       | check-endpoints                               |
| element                | Homepage         | Docker healthcheck                                                                                                    | check-endpoints                               |
| flaresolverr           | Homepage         | Docker healthcheck (sessions.list probe)                                                                              | check-endpoints                               |
| gluetun                | —                | Docker healthcheck + Homepage API widget (real VPN status)                                                            | check-endpoints                               |
| homepage               | Homepage         | Docker healthcheck                                                                                                    | check-endpoints                               |
| jellyfin               | Homepage         | Widget (no healthcheck)                                                                                               | check-endpoints                               |
| matrix-db (postgres)   | Homepage         | Docker healthcheck                                                                                                    | —                                             |
| mautrix-discord        | Homepage         | —                                                                                                                     | —                                             |
| miniflux               | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| miniflux-db (postgres) | Homepage         | Docker healthcheck                                                                                                    | —                                             |
| n8n                    | Homepage         | —                                                                                                                     | check-endpoints                               |
| nextflux               | Homepage         | Port check only                                                                                                       | check-endpoints                               |
| openclaw               | Homepage         | Docker healthcheck (`/healthz`)                                                                                       | check-endpoints                               |
| openwebui              | Homepage         | Docker healthcheck                                                                                                    | check-endpoints                               |
| pocket-id              | Homepage         | Docker healthcheck                                                                                                    | check-endpoints                               |
| prowlarr               | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| qbittorrent            | Homepage         | Docker healthcheck (via gluetun)                                                                                      | — (behind VPN, only through seerr)            |
| radarr                 | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| reader/feed-scraper    | Homepage         | —                                                                                                                     | —                                             |
| reader/rsshub          | Homepage         | —                                                                                                                     | check-endpoints                               |
| registry               | Homepage         | —                                                                                                                     | — (internal only)                             |
| searxng                | Homepage         | —                                                                                                                     | check-endpoints                               |
| seerr                  | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| sonarr                 | Homepage         | Docker healthcheck + widget                                                                                           | check-endpoints                               |
| stalwart               | Homepage         | Homepage JMAP widget (real query)                                                                                     | check-endpoints                               |
| synapse                | Homepage         | Docker healthcheck                                                                                                    | check-endpoints                               |
| traefik                | Homepage         | Homepage API widget                                                                                                   | check-endpoints                               |
| yamtrack               | Homepage         | Docker healthcheck                                                                                                    | check-endpoints                               |
| yamtrack/redis         | Homepage         | —                                                                                                                     | —                                             |

## Drive Monitoring

`scripts/drive-check.sh` runs on i3 every 15 min via cron. For each drive + the mergerfs pool, it checks mountpoint status, read, and write — printing `OK`/`FAIL` per drive and exiting non-zero if any drive failed, so cron surfaces the failure (mail/log). There is no push monitor anymore.

Monitored drives: `sandisk-2tb`, `sandisk-1tb-lg`, `sandisk-1tb-dg` (pool members), `mergerfs-pool` (`/mnt/main/media`).

## Backup & Maintenance Scripts

The cron scripts print `OK`/`FAIL` and exit non-zero on failure; cron surfaces failures via its own mail/log. There is no external heartbeat monitor (Uptime Kuma was retired) — a small Mastra scheduled health-check → ntfy is the intended replacement for the alerting these push monitors used to provide.

| Script                            | Node    | Frequency                                         |
| --------------------------------- | ------- | ------------------------------------------------- |
| `scripts/backup-i3.sh`            | i3      | daily 03:00                                       |
| `scripts/backup-pentium.sh`       | Pentium | daily 04:00                                       |
| `scripts/restic-check.sh`         | i3      | weekly Sun 05:00                                  |
| `scripts/rebuild-images.sh`       | i3      | weekly Sun 04:00                                  |
| `aiclient2api/sync-kiro-token.py` | i3      | every 15 min (`kiro-cli whoami` is the heartbeat) |
| `scripts/drive-check.sh`          | i3      | every 15 min                                      |

## Gaps

* **No continuous external prober.** `tests/check-endpoints.sh` is on-demand only; nothing alerts automatically when a public URL goes down. The Mastra health-check → ntfy job is the planned replacement.
* **Healthchecks still missing** on: cloudflared, docker-socket-proxy, dozzle, draw, jellyfin, n8n, qbittorrent, registry, traefik, mautrix-discord, feed-scraper, nextflux, rsshub, searxng, stalwart, yamtrack/redis. Most of these ship no healthcheck upstream; a few could use a cheap `curl`-based one added in our compose file.
* **Cloudflared** has no external monitor. Its status is implicit — if the tunnel is down, every public URL fails at once.
* **Gluetun** — Homepage's widget (with API key) gets the real VPN-tunnel state.

Cheapest wins: add Docker healthchecks to draw, stalwart, n8n, and searxng. All have trivial HTTP endpoints.
