Disaster Recovery Playbook

​

Recovery Keys You Need

• AWS credentials for the backup IAM user (S3 bucket augustin-backups)
• Restic repository password
• GitHub SSH key (to clone the repo to a replacement node)
• sealed-secrets sealing key — the private key that decrypts every SealedSecret in the repo. It lives in the cluster (ns kube-system), and a monthly CronJob age-encrypts it to s3://augustin-backups/sealed-secrets-keys (cluster/infra/sealed-secrets/key-backup-cronjob.yaml). The private age key that decrypts those backups is in 1Password — without it a lost control-plane means every SealedSecret is undecryptable.
• Cloudflare tunnel token (for cloudflared)

​

How Recovery Works Now

​

Scenario 1: Single Service / DB Corrupted

• Longhorn S3 backups (primary) — every platform/bulk volume is snapshotted daily to S3 by a Longhorn RecurringJob (StorageClasses auto-enroll new volumes). Restore a volume from the Longhorn UI (Backup → restore) and re-attach it, or clone into a new PVC.
• restic CronJob (cluster/infra/backup/) — covers what Longhorn snapshots can’t capture consistently: pg_dumpall of Postgres (miniflux, temporal) plus the legacy hostPath / local-path trees. Restore from S3:

restic -r s3:s3.us-west-2.amazonaws.com/augustin-backups/cluster snapshots
restic -r s3:s3.us-west-2.amazonaws.com/augustin-backups/cluster restore latest --target /tmp/restore

• Postgres (miniflux, temporal): restore the pg_dumpall dump from /tmp/restore, then kubectl exec into the DB pod and psql < dump.
• Everything else: restore the Longhorn volume backup rather than a file copy.

​

Scenario 2: Worker Node Loss (i3 or i5)

1. Reinstall Debian on the replacement, install open-iscsi + nfs-common (Longhorn needs them).
2. Join it to the cluster as a k3s agent (token from the control-plane: cat /var/lib/rancher/k3s/server/node-token on pentium).
3. Re-apply the node label (kubectl label node <name> kind=storage|compute) and, if it bears a Longhorn disk, add the disk in the Longhorn UI.
4. Longhorn rebuilds the missing replica automatically.
5. bulk data (the media library) is single-replica — if the node holding it is lost, that data is gone and must be re-downloaded. Only the media library is on bulk.

​

Scenario 3: Control-Plane Loss (pentium)

1. Reinstall Debian, install k3s server (control-plane). Restore the same cluster token / datastore if you want the same cluster identity; otherwise a fresh control-plane + re-join workers also works but is more disruptive.
2. registries.yaml is retired (2026-07-19) — nodes pull from GHCR directly, so there is nothing to restore here.
3. Recreate the local-storage default-class override (node-local, not in git): sudo touch /var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip, then kubectl annotate sc local-path storageclass.kubernetes.io/is-default-class=false --overwrite. Without the .skip sentinel, k3s re-applies its bundled local-storage.yaml on every restart and re-marks local-path as the cluster-default StorageClass. See “No default StorageClass” under Known Gaps.
4. Re-enable Secrets encryption-at-rest (node-local, not in git — see “Secrets encryption-at-rest” under Known Gaps). If you restored the original datastore, the AES key travels with it (it’s persisted as cluster bootstrap data), so you only need secrets-encryption: true back in /etc/rancher/k3s/config.yaml before starting k3s. On a fresh datastore, re-run the enable flow: echo "secrets-encryption: true" | sudo tee /etc/rancher/k3s/config.yaml → sudo k3s secrets-encrypt enable → sudo systemctl restart k3s (wait for /healthz = ok) → sudo k3s secrets-encrypt rotate-keys → sudo systemctl restart k3s. Verify with sudo k3s secrets-encrypt status (Enabled, stage reencrypt_finished, an AES-CBC active key).
5. Ensure the NoSchedule taint is back so workloads don’t land on it.
6. cloudflared + the Envoy Gateway proxy pods reschedule from git; verify https://<anything>.augustin.ai resolves again.

​

Scenario 4: Full Cluster Rebuild

1. Install k3s server on pentium, agents on i3 + i5. Label the workers, taint pentium.
2. Install Longhorn’s node prereqs (open-iscsi, nfs-common) and add disks on i3 + i5.
3. registries.yaml is retired — nodes pull from GHCR directly (no mirror to restore).
4. flux bootstrap github against amarcin/homelab (branch main). Restore the sealed-secrets sealing key into ns kube-system before Flux tries to unseal: fetch the latest age-encrypted bundle from s3://augustin-backups/sealed-secrets-keys, decrypt it with the private age key from 1Password, and kubectl apply the key Secret (label sealedsecrets.bitnami.com/sealed-secrets-key). The controller then decrypts every SealedSecret in the repo. (A fresh key means every SealedSecret must be re-sealed — restore the backed-up key to avoid that.)
5. Flux reconciles the whole cluster/ tree. Longhorn PVCs come up empty.
6. Restore application data from restic (Scenario 1 pattern) into the freshly-provisioned PVCs.
7. Media library (bulk) is re-downloaded, not restored.

​

Scenario 5: Restic Repo Corruption

1. Don’t restic forget --prune until fixed — it makes it worse.
2. S3 versioning has your back: recover prior pack-file versions via aws s3api list-object-versions --bucket augustin-backups --prefix cluster/data/.
3. Worst case: start a new repo, accept data loss to the last good snapshot; keep the old repo read-only until the new one is proven healthy.

​

Scenario 6: Cloudflared Tunnel Dead

1. kubectl -n mesh get pods -l app=cloudflared and kubectl -n mesh logs — running? registration errors?
2. Cloudflare dashboard → Zero Trust → Networks → Tunnels — tunnel “healthy”? origin still http://192.168.1.16:8890?
3. If the token is rotated/compromised: re-seal the cloudflared SealedSecret with the new token, commit, let Flux roll it.
4. Out-of-band access with the tunnel down: SSH into a node on its LAN IP from a machine on the LAN.

​

Known Gaps

• Node-level config is not in git. The k3s install itself and the Longhorn disk config are node-local, as is the local-storage default-class override on pentium (see next). (registries.yaml was retired 2026-07-19 — nodes pull from GHCR directly, nothing to restore.)
• No default StorageClass — enforced node-level on pentium. Callers must name a class explicitly (platform for precious state, bulk for media); a class-less PVC intentionally fails to bind rather than silently landing on node-local single-copy local-path. k3s ships local-path as the cluster-default via its bundled local-storage addon manifest and re-applies it on every restart, so the override is two node-local pieces on pentium: (1) a sudo touch /var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip sentinel that tells k3s to stop managing that addon (the running provisioner is left intact, so local-path stays available for anyone who names it), and (2) kubectl annotate sc local-path storageclass.kubernetes.io/is-default-class=false --overwrite. The .skip makes the annotation durable across k3s restarts. This is deliberately NOT in git — the StorageClass is owned by the k3s addon controller, and a Flux-managed copy would fight it; the sentinel is the k3s-sanctioned way to bow the addon out.
• Secrets encryption-at-rest is enabled but node-local on pentium. Kubernetes Secrets are encrypted at rest in the k3s datastore via k3s’s built-in AES-CBC provider (secrets-encryption: true in /etc/rancher/k3s/config.yaml, which points the apiserver at /var/lib/rancher/k3s/server/cred/encryption-config.json). Neither the config flag nor the AES key is in git — the key lives in the datastore’s cluster bootstrap data and travels with a datastore restore, but a fresh control-plane must re-run the enable flow (Scenario 3, step 4). Consumers read secrets transparently; encryption is invisible to workloads. Note: enabling only encrypts secrets going forward + a one-time reencrypt pass; stale pre-encryption plaintext revisions linger in the SQLite datastore until kine compaction purges them.
• bulk (media library) is single-replica — a disk/node loss there means re-download. This is deliberate (media is cheap to refetch); precious state is on 2-replica platform.
• The sealed-secrets sealing key is load-bearing. Without it the controller cannot decrypt any SealedSecret, so secret-bearing services stay down. It lives in ns kube-system and is age-encrypted monthly to s3://augustin-backups/sealed-secrets-keys; keep the private age key (1Password) off-site.