> ## Documentation Index
> Fetch the complete documentation index at: https://docs.augustin.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Disaster Recovery Playbook

Steps to recover the homelab from various failure modes. Assumes you have access to AWS account `389565021081`, the GitHub repo `amarcin/homelab`, and physical access (or SSH via an existing working node) to the remaining hardware.

## Recovery Keys You Need

All stored outside the homelab (1Password, hardware token, etc.):

* AWS admin credentials for account `389565021081`
* Restic repo password (also in `scripts/.restic-password` on each node — but you need it to bootstrap a node)
* GitHub SSH key (for cloning the repo to a replacement node)
* `.env-backup.tar.gz` restic snapshot recovery — the env files for every service

## Scenario 1: Single Service Corrupted

Fastest case. Restore from S3.

```
restic -r s3:s3.us-west-2.amazonaws.com/augustin-backups/<node> snapshots
restic -r s3:s3.us-west-2.amazonaws.com/augustin-backups/<node> restore <id> --target /tmp/restore --include /home/user/apps/<service>
```

Stop the service, replace its `data/` dir, bring it back up.

## Scenario 2: Pentium Total Loss

Pentium hosts: matrix, reader, stalwart, n8n, yamtrack, searxng, draw, pocket-id. All DBs are on Pentium.

1. Install Debian 13 on new hardware (or reinstalled Pentium)
2. Create `user` (UID 1000), add to `sudo`, enable passwordless sudo
3. `apt install docker.io git rsync restic`
4. Configure `~/.ssh/config` aliases for `i3` (192.168.1.100) and `ssh-copy-id` so SSH works both directions
5. Clone repo: `git clone git@github.com:amarcin/homelab.git ~/apps`
6. `bash ~/apps/scripts/systemd/install.sh` — enable the 5-min auto-pull
7. Put `/etc/docker/daemon.json` in place:
   ```
   {"insecure-registries": ["192.168.1.100:5000"]}
   ```
   Restart docker.
8. Restore Pentium's `/home/user/apps` from S3:
   ```
   export RESTIC_PASSWORD_FILE=~/apps/scripts/.restic-password
   source ~/apps/scripts/.aws-env
   restic -r s3:s3.us-west-2.amazonaws.com/augustin-backups/pentium snapshots
   restic -r s3:s3.us-west-2.amazonaws.com/augustin-backups/pentium restore latest --target /tmp/restore
   rsync -a /tmp/restore/home/user/apps/ ~/apps/
   ```
9. Restore DBs from the dumps inside the restored repo:
   * `matrix/matrix-db-dump.sql` → once the `matrix-db` container is up, `cat matrix-db-dump.sql | docker exec -i matrix_matrix-db.N psql -U synapse`
   * Same pattern for `reader/miniflux-db-dump.sql`
10. On i3, join pentium back to the swarm: get join token with `docker swarm join-token manager`, run the join command on pentium
11. Redeploy all Pentium stacks: from i3, `for s in matrix reader stalwart n8n yamtrack searxng draw pocket-id; do cd ~/apps/$s && set -a; source .env 2>/dev/null; set +a; docker stack deploy -c compose.yaml $s; done`

## Scenario 3: i3 Total Loss

More painful. i3 hosts media data on mergerfs SanDisks. Those are physical drives — recover them to the new box before anything else.

1. If SanDisks are intact: plug them into the new i3. Verify labels (`sandisk-2tb`, `sandisk-1tb-lg`, `sandisk-1tb-dg`) match `/etc/fstab`.
2. If SanDisks are lost: your media is gone. Repopulate via arr stack / torrent indexers over time. Jellyfin library metadata will rebuild, but watch progress/history is gone unless backed up separately.
3. Steps 1-6 from Scenario 2, substituting `pentium` → `i3`
4. Additional for i3:
   * Install mergerfs: `apt install mergerfs`
   * Verify `/mnt/main/media` mounts correctly on boot
   * Set up CDI for Jellyfin (see `docs/archive/media/jellyfin.md`)
5. Restore `/home/user/apps` from S3 (i3 repo)
6. On a still-running manager (if swarm survived on Pentium): `docker swarm leave --force` old dead i3 node. `docker swarm join-token manager` to get a new join command.
7. If swarm is totally dead (both nodes lost): `docker swarm init --advertise-addr 192.168.1.100` on new i3, then join Pentium. Stacks auto-recover from declarative compose files, but data restores must be complete first.
8. Traefik + cloudflared must come up first on i3 so everything else is reachable
9. Redeploy all stacks

## Scenario 4: Swarm Quorum Lost (both managers down simultaneously but hardware OK)

Likely cause: docker daemon crash on both nodes, or one crashed and the other rebooted. Swarm Raft needs a manager majority to function.

1. On one node, `docker swarm init --force-new-cluster --advertise-addr <node-ip>` — forces a new single-manager cluster
2. Rejoin the other node with `docker swarm join-token manager` from the forced manager
3. All services auto-recover from the stack definitions (they're stored in Raft, which the forced-new-cluster preserves)

## Scenario 5: Restic Repo Corruption

Detected by the weekly `restic check` cron. Symptoms: check fails, snapshots can't be read.

1. Don't run `restic forget --prune` until fixed — it'll make it worse
2. S3 versioning has your back: find the last-known-good pack files via `aws s3api list-object-versions --bucket augustin-backups --prefix <node>/data/`, restore the prior versions
3. Worst case: repo is unrecoverable. Start a new repo, accept data loss back to the last local snapshot. Keep old repo as read-only reference until sure new one is healthy.

## Scenario 6: Cloudflared Tunnel Dead

All public URLs and `ssh.augustin.ai` stop working. Check in this order:

1. `docker service ps cloudflared_cloudflared` — task running?
2. `docker service logs cloudflared_cloudflared` — registration errors?
3. Cloudflare dashboard → Zero Trust → Networks → Tunnels — is the tunnel showing "healthy"?
4. If tunnel credentials are compromised/rotated: regenerate `TUNNEL_TOKEN` at Cloudflare, update `cloudflared/.env`, `docker service update --force cloudflared_cloudflared`

If you need out-of-band access with cloudflared down: SSH into the node on its LAN IP (192.168.1.100 or .16) from a machine on the LAN.

## Known Gaps

* **Config for mergerfs mount order** is in `/etc/fstab` only, not in git. If /etc/fstab is lost, reconstruct from `docs/infrastructure/hardware.md`.
* **Cron jobs are not in git** either. Document: i3 has backup-i3.sh, stalwart backup, aiclient2api kiro sync, cwa-ingest-cron, torrent-to-cwa. Pentium has backup-pentium.sh, drive-check.sh (via other agent).
* **Docker daemon.json insecure-registries** list is machine-local. Must be restored on any node that pulls from the local registry.