> ## Documentation Index
> Fetch the complete documentation index at: https://docs.augustin.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Node Prep (bare OS → cluster-ready)

The **imperative, out-of-git layer** that turns a bare Debian box into a working
member of the k3s + Flux cluster. Flux reconciles the `cluster/` tree, but it
does **not** manage `/etc/rancher/k3s/`, `/etc/fstab`, disk formatting, OS
packages, kernel modules, or SSH keys. Those are per-host and live only on the
node. This page is the single authoritative record of that layer, captured from
live node state so a node can be rebuilt from scratch.

Companion runnable helpers: [`scripts/node-prep/`](https://github.com/amarcin/homelab/tree/main/scripts/node-prep)
(idempotent; meant for rebuild time, not to be run against healthy nodes). The
disaster-recovery playbook ([recovery](/infrastructure/recovery)) references this
page for node-level steps.

> **All values below were read from the live nodes** (fstab UUIDs, IPs, k3s
> flags, Longhorn disk specs). Where a value is node-specific
> (disk UUIDs differ per box) it is listed per node — do NOT copy another node's
> UUID. Re-read with `blkid` on the actual disk when rebuilding.

## Fleet at a glance

| Node        | IP            | Role / k3s                                                                                                                                                               | Longhorn disks                                                                                                                              | Node-local special files                                                                      | OS pkgs                                           |
| ----------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| **pentium** | 192.168.1.16  | k3s **server** (control-plane), `NoSchedule` taint. `--disable traefik`, `--disable servicelb`, `--write-kubeconfig-mode 644`. `config.yaml`: `secrets-encryption: true` | **none** (diskless)                                                                                                                         | `config.yaml` (secrets-encryption), `local-storage.yaml.skip` + `coredns.yaml.skip` sentinels | open-iscsi, nfs-common                            |
| **i3**      | 192.168.1.100 | k3s **agent** (worker), label `kind=storage`                                                                                                                             | `disk0` `/mnt/longhorn/disk0` (sdb1), `disk1` `/mnt/longhorn/disk1` (sdc1), `sandisk2tb-bulk` `/mnt/sandisk-2tb/longhorn` (sdd, tag `bulk`) | media mergerfs + sandisk mounts, GPU `/dev/dri`                                               | open-iscsi, nfs-common, mergerfs                  |
| **i5**      | 192.168.1.10  | k3s **agent** (worker), label `kind=compute`                                                                                                                             | `disk0` `/mnt/longhorn/disk0` (sda1)                                                                                                        | ALSA mic state (ALC274, §7.5)                                                                 | open-iscsi, nfs-common, alsa-utils, alsa-ucm-conf |

(`registries.yaml` was retired 2026-07-19 — see §3.)

k3s version across all three: **v1.36.2+k3s1**. OS: **Debian 13 (trixie)**.
Only `pentium` carries a `config.yaml` and the `.skip` sentinel — the workers'
`/etc/rancher/k3s/config.yaml` does not exist (empty), which is correct; the
datastore + secrets-encryption live on the server.

***

## 1. Disk formatting + mounts (Longhorn data disks)

Longhorn stores replicas on plain ext4 filesystems mounted at well-known paths.
Each data disk is a whole spare drive, single-partition (or the whole device),
ext4, mounted **by UUID** with `nofail` so a missing disk never blocks boot.

### Formatting procedure (per data disk, at rebuild)

```bash theme={null}
# Identify the target device with lsblk FIRST — never assume sdX ordering.
sudo mkfs.ext4 -L <label> /dev/sdXN         # or partition first with parted/fdisk
sudo blkid /dev/sdXN                         # copy the UUID it prints
sudo mkdir -p /mnt/longhorn/diskN
# add the /etc/fstab line (below), then:
sudo systemctl daemon-reload
sudo mount -a
```

Helper: [`scripts/node-prep/format-mount-longhorn-disk.sh`](https://github.com/amarcin/homelab/tree/main/scripts/node-prep/format-mount-longhorn-disk.sh)
does exactly this idempotently (skips mkfs if the device already has a
filesystem; appends the fstab line only if absent).

### Live fstab entries (Longhorn + media only — copy per node)

**i3** (`/etc/fstab`):

```fstab theme={null}
# Media drives (migrated from Pentium 2026-04-24)
LABEL=sandisk-2tb     /mnt/sandisk-2tb     ext4 defaults,nofail 0 0

# mergerfs pool across all three SanDisks -> /mnt/main/media
/mnt/sandisk-2tb/media:/mnt/sandisk-1tb-lg/media:/mnt/sandisk-1tb-dg/media /mnt/main/media fuse.mergerfs defaults,allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=epmfs,moveonenospc=true,minfreespace=20G,nofail 0 0
UUID=7ec23b6c-2089-4543-8592-8f08117f2d37 /mnt/longhorn/disk0 ext4 defaults,nofail 0 2
UUID=18af6df0-9549-47c1-a2f7-cbaad807f750 /mnt/longhorn/disk1 ext4 defaults,nofail 0 2
```

* `sdb1` UUID `7ec23b6c-2089-4543-8592-8f08117f2d37` → `/mnt/longhorn/disk0` (932G)
* `sdc1` UUID `18af6df0-9549-47c1-a2f7-cbaad807f750` → `/mnt/longhorn/disk1` (932G)
* `sdd`  UUID `023583fd-8474-43bb-9213-ccb9292434ce`, LABEL `sandisk-2tb` → `/mnt/sandisk-2tb` (1.8T). Mounted **by LABEL**, not UUID. This drive is dual-purpose: it holds the media library **and** a Longhorn `bulk` disk at the `/mnt/sandisk-2tb/longhorn` subdir (see §2).
* The `mergerfs` line references `sandisk-1tb-lg`/`sandisk-1tb-dg` which are **not currently mounted** (a media migration is in flight onto the `bulk` PVC). `nofail` keeps boot clean while they're absent. Do not "fix" this during a rebuild without checking the current media-migration state.

**i5** (`/etc/fstab`):

```fstab theme={null}
UUID=2870b24b-fc59-4265-9f70-c341d36c70f5 /mnt/longhorn/disk0 ext4 defaults,nofail 0 2
```

* `sda1` UUID `2870b24b-fc59-4265-9f70-c341d36c70f5` → `/mnt/longhorn/disk0` (932G)

**pentium**: no Longhorn/data fstab entries — diskless for the cluster (only the
OS disk `sda` with `/`, `/boot/efi`, swap).

> `pass` field: OS root is `1`, Longhorn data disks are `2` (fsck after root),
> the LABEL-mounted sandisk and mergerfs are `0` (no fsck). `nofail` on every
> data mount is mandatory — a Longhorn disk that fails to mount must not wedge boot.

***

## 2. Longhorn disk registration (nodes.longhorn.io CRD)

Formatting + mounting a disk is not enough — Longhorn must be told to schedule
replicas onto it via the `spec.disks` map on the node's `Node` CRD in
`longhorn-system`. Longhorn auto-creates a `disk0` at the default data path on a
labeled node (`create-default-disk-labeled-nodes: true`, default-data-path
`/var/lib/longhorn/`) but the **extra disks and the `bulk` tag were added by
hand** and are node-local state (not in git).

Read live with:
`kubectl -n longhorn-system get nodes.longhorn.io <node> -o jsonpath='{.spec.disks}'`.

**i3** — three disks registered:

| Disk name         | path                        | allowScheduling | tags       | storageReserved |
| ----------------- | --------------------------- | --------------- | ---------- | --------------- |
| `disk0`           | `/mnt/longhorn/disk0`       | true            | `[]`       | 0               |
| `disk1`           | `/mnt/longhorn/disk1`       | true            | `[]`       | 0               |
| `sandisk2tb-bulk` | `/mnt/sandisk-2tb/longhorn` | true            | `["bulk"]` | 0               |

**i5** — one disk:

| Disk name | path                  | allowScheduling | tags | storageReserved |
| --------- | --------------------- | --------------- | ---- | --------------- |
| `disk0`   | `/mnt/longhorn/disk0` | true            | `[]` | 0               |

**pentium** — no Longhorn disks (diskless, tainted).

The `bulk` **disk tag** on i3's `sandisk2tb-bulk` is what pins the `bulk`
StorageClass's single-replica media volume onto that 2TB drive. The `platform`
StorageClass (2 replicas across i3+i5) uses the untagged disks. (StorageClasses
are in git: `cluster/infra/longhorn/storageclasses.yaml`; disk tags are not.)

To re-register a disk after a rebuild (all data paths must already be mounted):

```bash theme={null}
# Patch the node CRD's spec.disks. Example: add the bulk-tagged disk on i3.
kubectl -n longhorn-system patch nodes.longhorn.io i3 --type merge -p '{
  "spec": { "disks": {
    "sandisk2tb-bulk": {
      "path": "/mnt/sandisk-2tb/longhorn",
      "allowScheduling": true,
      "storageReserved": 0,
      "tags": ["bulk"],
      "diskType": "filesystem"
    }
  }}}'
```

(Or do it in the Longhorn UI: Node → Edit node and disks → Add Disk.) Longhorn
will not schedule onto a path it can't see mounted — create the mount first.

***

## 3. Registry access — `registries.yaml` **retired 2026-07-19**

The in-cluster registry was deleted; builds now push to **GHCR**
(`ghcr.io/amarcin/<name>:<short-sha>`, built by in-cluster BuildKit — see
`cluster/infra/build/README.md`). Nodes pull directly from GHCR, so the old
insecure-mirror `/etc/rancher/k3s/registries.yaml` is no longer needed.

On each node the file was **renamed to `registries.yaml.retired-2026-07-19`**
(kept as a breadcrumb, not read by containerd). On a fresh rebuild there is
nothing to write here — skip it. If a node still has an active
`registries.yaml` pointing at the dead ClusterIP `10.43.0.100`, rename it and
restart k3s (`k3s-agent` on workers, `k3s` on pentium).

***

## 4. k3s config & systemd flags (server vs agent)

k3s is installed from the official installer (`get.k3s.io`). The install writes a
systemd unit (`k3s.service` on the server, `k3s-agent.service` on workers) whose
`ExecStart` flags are baked at install time, plus it reads
`/etc/rancher/k3s/config.yaml` at startup. Both were read live.

### pentium — server (`k3s.service` ExecStart)

```
/usr/local/bin/k3s server \
  --disable traefik \
  --disable servicelb \
  --write-kubeconfig-mode 644 \
  --node-name pentium \
  --node-ip 192.168.1.16
```

Plus `/etc/rancher/k3s/config.yaml`:

```yaml theme={null}
secrets-encryption: true
```

* `--disable traefik` — Traefik retired; Envoy Gateway is the sole router.
* `--disable servicelb` — klipper/servicelb off; only mail uses a LoadBalancer (handled elsewhere). *(Note: mail's LoadBalancer is served by klipper on i3 per the mail docs — confirm servicelb handling if reworking; this flag is what live pentium reports.)*
* `--write-kubeconfig-mode 644` — makes `/etc/rancher/k3s/k3s.yaml` world-readable so non-root can run kubectl.
* `secrets-encryption: true` — enables k3s's built-in AES-CBC encryption-at-rest for Kubernetes Secrets in the datastore. See §9 and recovery.md Scenario 3.

Install equivalent (fresh server):

```bash theme={null}
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.2+k3s1 sh -s - server \
  --disable traefik --disable servicelb --write-kubeconfig-mode 644 \
  --node-name pentium --node-ip 192.168.1.16
# then write config.yaml with secrets-encryption and follow the enable flow (§9)
```

### i3 / i5 — agents (`k3s-agent.service` ExecStart)

```
/usr/local/bin/k3s agent --node-name i3 --node-ip 192.168.1.100    # i3
/usr/local/bin/k3s agent --node-name i5 --node-ip 192.168.1.10     # i5
```

No `config.yaml` on the workers. Join a worker with the server URL + node token
(`cat /var/lib/rancher/k3s/server/node-token` on pentium):

```bash theme={null}
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.2+k3s1 \
  K3S_URL=https://192.168.1.16:6443 K3S_TOKEN=<node-token> sh -s - agent \
  --node-name i5 --node-ip 192.168.1.10
```

Both units also `modprobe br_netfilter` and `overlay` as `ExecStartPre` (k3s
installer adds these).

### Node labels & taint (post-join, via kubectl)

```bash theme={null}
kubectl label node i3 kind=storage
kubectl label node i5 kind=compute
kubectl taint node pentium node-role.kubernetes.io/control-plane=true:NoSchedule
# (workers also carry node-role.kubernetes.io/worker=true; pentium is control-plane)
```

Live state confirms: i3 `kind=storage`, i5 `kind=compute`, pentium tainted
`node-role.kubernetes.io/control-plane=true:NoSchedule`, workers untainted.

***

## 5. local-storage default-class override (pentium only)

There is intentionally **no cluster-default StorageClass**: a class-less PVC
should fail to bind (forcing explicit `platform`/`bulk` intent) rather than
silently landing on node-local single-copy `local-path`. But k3s ships a bundled
`local-storage` addon that annotates `local-path` as `is-default-class=true` and
**re-applies it on every restart**. The override is two node-local pieces on
pentium (the server owns addon manifests):

1. A zero-byte sentinel that tells k3s to stop managing that addon:

   ```bash theme={null}
   sudo touch /var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip
   ```

2. Turn the default annotation off (durable once the `.skip` exists):

   ```bash theme={null}
   kubectl annotate sc local-path storageclass.kubernetes.io/is-default-class=false --overwrite
   ```

Verified live on pentium: `local-storage.yaml.skip` (0 bytes) sits next to
`local-storage.yaml` in `/var/lib/rancher/k3s/server/manifests/`. The `.skip`
makes k3s bow out of managing that addon (the running provisioner is left intact,
so `local-path` stays available for anyone who names it explicitly) — so the
annotation stays off across restarts. This is deliberately **not** in git: the
StorageClass is owned by the k3s addon controller and a Flux-managed copy would
fight it. See recovery.md "No default StorageClass" and
`cluster/infra/longhorn/storageclasses.yaml`.

### 5.1. CoreDNS — the second `.skip` sentinel

Same mechanism, opposite conclusion: cluster DNS **is** owned in git
(`cluster/infra/coredns/`, `replicas: 2`) — k3s ships 1 replica on the tainted
control-plane, a fleet-wide DNS single point of failure.

```bash theme={null}
sudo touch /var/lib/rancher/k3s/server/manifests/coredns.yaml.skip
```

Unlike `--disable coredns` (which **deletes** the live objects), the sentinel only
stops re-application, so the handover has no DNS outage. Apply the leaf first, then
the sentinel. Full rationale, the post-upgrade diff command and the `NodeHosts`
trap live in the `cluster/infra/coredns/kustomization.yaml` header comment.

***

## 6. OS packages Longhorn needs

Longhorn's engine attaches volumes over iSCSI and serves RWX volumes over NFS, so
every node that could run a Longhorn workload needs:

| Package      | Why                                                                            |
| ------------ | ------------------------------------------------------------------------------ |
| `open-iscsi` | iSCSI initiator — Longhorn attaches every volume via iSCSI                     |
| `nfs-common` | NFS client — Longhorn RWX (`bulk`) volumes are served over NFS (share-manager) |

Verified installed on **all three** nodes (version `open-iscsi 2.1.11-1+deb13u2`,
`nfs-common 1:2.8.3-1`). `iscsid` is **enabled + active** on all three.

```bash theme={null}
sudo apt-get update
sudo apt-get install -y open-iscsi nfs-common
sudo systemctl enable --now iscsid
```

Helper: [`scripts/node-prep/install-longhorn-prereqs.sh`](https://github.com/amarcin/homelab/tree/main/scripts/node-prep/install-longhorn-prereqs.sh).

pentium is diskless and runs no Longhorn workloads, but it has these installed
too (harmless, keeps the fleet uniform).

***

## 7. Kernel modules & sysctls

### iscsi\_tcp — persisted on all three nodes

Longhorn needs the `iscsi_tcp` module. It is made persistent via a drop-in
(verified present on all three nodes):

```
# /etc/modules-load.d/longhorn-iscsi.conf
iscsi_tcp
```

`lsmod` confirms `iscsi_tcp` loaded on all three. (Recreate the file and
`sudo modprobe iscsi_tcp` if missing after a rebuild.)

### i915 — Intel GPU, auto-loaded

`i915` is loaded on all three nodes automatically (in-tree module, loads for the
Intel iGPU at boot) — no explicit modules-load drop-in needed. i3 and i5 expose
`/dev/dri/{card0,renderD128}` (`card0` = `video` group, `renderD128` = `render`
group). pentium's Pentium G3220 loads i915 but has no usable QuickSync
(`hardware.md`). The Intel GPU **device plugin** (advertising
`gpu.intel.com/i915`) is GitOps (`cluster/infra/intel-gpu/`) — nothing to do at
node level beyond having the module + `/dev/dri`, which come from the stock
Debian kernel.

### sysctls

k3s manages the networking sysctls it needs. The k3s systemd unit modprobes
`br_netfilter` + `overlay` on start (`ExecStartPre`). Live values on pentium:
`net.bridge.bridge-nf-call-iptables=1`, `net.ipv4.ip_forward=1` — these are set
by k3s/flannel, not by a hand-authored sysctl drop-in. No custom sysctl file is
required for node prep.

***

## 7.5. i5 audio input — mic capture for Home Assistant voice (ALSA)

i5 is a Dell Inspiron AIO with a **Realtek ALC274** codec on `card 0: PCH [HDA
Intel PCH]`, capture device `plughw:0,0`. The consumer is the `kiosk` pod (ns
`peripherals`, privileged, `/dev/snd`): its Chrome captures the mic via
`getUserMedia` and plays TTS back through the speaker, both on card 0. The
"hey jarvis" wake word is detected in-browser by the kiosk-ui SPA (LiveKit
participant `kiosk`); there is no host-level or in-pod wake-word daemon. The
host-level ALSA state below (mixer levels, capture unmute, `alsactl` persistence)
is what makes that mic path deliver signal at all.

**Failure mode seen (2026-07-11):** the mic captured only silence — a consumer
opening `plughw:0,0` read a flat signal and no audio ever reached the app. Root
cause was purely host-level ALSA state: **`alsa-utils` was never installed** on
this fresh headless box, so `alsactl`/`amixer` had never run and the codec
powered up with **capture muted and Internal Mic Boost at zero**. A capture of
`plughw:0,0` streamed dead silence (measured `peak 0, rms 0.0` over a 3 s
capture). (The mixer *controls* were always exposed by the kernel driver — there
was just no userspace tool present to read or set them, and nothing to unmute
capture at boot.)

**Fix (host-level on i5, out-of-git):**

```bash theme={null}
# 1. Install ALSA userspace (official Debian repos). Also pulls alsa-ucm-conf.
sudo apt-get update
sudo apt-get install -y alsa-utils alsa-ucm-conf

# 2. Unmute + set the internal-mic capture path on card 0.
amixer -c 0 sset Capture 60% cap          # ADC: 60% (+11.25 dB), capture ON
amixer -c 0 sset 'Internal Mic Boost' 0%  # boost 0 dB — 60% cap alone is plenty
amixer -c 0 sset 'Internal Mic' cap       # select internal DMIC as capture source
# (Master output was also muted by Auto-Mute; unmute so the TTS *response* is audible)
amixer -c 0 sset Master 80% unmute

# 3. Persist across reboot.
sudo alsactl store                        # writes /var/lib/alsa/asound.state
```

**Level tuning:** `Capture 100% + Boost 100%` (=42 dB) pegged the ADC at
full-scale (`peak 32767`, clipping). `Capture 60% + Boost 0%` gives a clean
`peak ~13700` with headroom and a low noise floor — the level stored above.

**Persistence mechanism:** installing `alsa-utils` drops in
`/usr/lib/udev/rules.d/90-alsa-restore.rules`, which runs
`alsactl restore` when the sound control device appears at boot — no unit to
`enable` (`alsa-restore.service` is `static`, driven by that udev rule). Verified
by muting Capture to 0%/off, running the exact
`sudo alsactl -E HOME=/run/alsa -E XDG_RUNTIME_DIR=/run/alsa/runtime restore 0`
the rule fires, and confirming it recovered to 60% `[on]`.

**Verify capture has signal** (free the device first — the `kiosk` pod's Chrome
holds it):

```bash theme={null}
kubectl scale deploy kiosk -n peripherals --replicas=0   # release mic
ssh i5 'arecord -D plughw:0,0 -f S16_LE -r 16000 -c 1 -d 3 /tmp/t.wav'
# make noise during the 3 s; then check peak/rms is non-zero (python wave module)
kubectl scale deploy kiosk -n peripherals --replicas=1   # restore
```

i3 has no equivalent mic requirement; this section is **i5-only**.

***

## 8. SSH topology (bastion + inter-node)

pentium is the SSH bastion; the Mac reaches it over the Cloudflare tunnel
(`ssh ssh.augustin.ai`). pentium reaches the workers by key auth over the LAN.

pentium's `~/.ssh/config` (verified live):

```sshconfig theme={null}
Host i3
    HostName 192.168.1.100
    User user

Host i5
    HostName 192.168.1.10
    User user
    IdentityFile ~/.ssh/id_ed25519
```

* **pentium → i3**: works (key auth). i3 uses the default id key.
* **pentium → i5**: **works** (verified live — `ssh i5` succeeds; i5's
  `authorized_keys` has pentium's pubkey. This corrects the stale "currently
  fails" note in `AGENTS.md`, which predates the key being installed.)

pentium's public key (`~/.ssh/id_ed25519.pub`) is:
`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3ekCUNW00tqzjFg0lU61jhJi8pn9qDWRy+0Iespb34 user@debian-2`

### Re-establishing keys after a rebuild

If **pentium** is rebuilt (new keypair): copy its new pubkey onto each worker.
If a **worker** is rebuilt: append pentium's existing pubkey to the new worker's
`authorized_keys`.

```bash theme={null}
# From pentium, once its keypair exists (ssh-keygen -t ed25519 if new):
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@192.168.1.100   # i3
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@192.168.1.10    # i5
# If the worker has password auth disabled, append the pubkey at its physical
# console instead:
#   echo 'ssh-ed25519 AAAA... user@debian-2' >> ~/.ssh/authorized_keys
```

(Never copy private keys off the bastion. The pubkey above is safe to paste.)

***

## 9. Secrets encryption-at-rest (pentium / datastore)

Kubernetes Secrets are encrypted at rest in the k3s datastore via k3s's built-in
AES-CBC provider. Node-local on pentium; neither the flag nor the AES key is in
git.

* `secrets-encryption: true` in `/etc/rancher/k3s/config.yaml` (see §4).
* The AES key lives in the datastore's cluster bootstrap data and **travels with
  a datastore restore** — so restoring the original datastore needs only the flag
  back before starting k3s.
* On a **fresh** datastore, re-run the enable flow:

  ```bash theme={null}
  echo "secrets-encryption: true" | sudo tee /etc/rancher/k3s/config.yaml
  sudo k3s secrets-encrypt enable
  sudo systemctl restart k3s          # wait for /healthz = ok
  sudo k3s secrets-encrypt rotate-keys
  sudo systemctl restart k3s
  sudo k3s secrets-encrypt status     # Enabled, stage reencrypt_finished, AES-CBC active key
  ```

Full detail: recovery.md Scenario 3 + "Secrets encryption-at-rest" under Known Gaps.

***

## Rebuild order (bare OS → cluster-ready)

1. Install Debian 13. Create the `user` account. Configure SSH (§8).
2. Install Longhorn prereqs on workers (and pentium for uniformity) — §6
   (`install-longhorn-prereqs.sh`). Persist `iscsi_tcp` — §7.
3. **Workers only:** format + mount each Longhorn data disk — §1
   (`format-mount-longhorn-disk.sh`). Confirm `/dev/dri` exists on i3/i5 (§7).
4. Install k3s (server on pentium, agents on i3/i5) with the exact flags — §4.
   On pentium also write `config.yaml` + run the secrets-encryption enable flow — §9.
5. `registries.yaml` is retired — nothing to write here (§3). Restart k3s
   workers-first only if a stale file is still active.
6. On pentium: create the `local-storage.yaml.skip` sentinel + un-default
   `local-path` — §5. Apply node labels + the pentium taint — §4.
7. `flux bootstrap github`. Secrets are **SealedSecrets** — the controller's
   sealing key decrypts them in-cluster, so there is no key to install on
   bootstrap (unlike the old SOPS age key). If the sealing key was lost with the
   control-plane, restore it first (recovery.md Scenario 4). Flux reconciles
   Longhorn.
8. **Workers:** register Longhorn disks on the node CRDs (extra disks + the `bulk`
   tag on i3) — §2.
9. Restore application data from restic into the fresh Longhorn PVCs (recovery.md
   Scenario 1). Media library on `bulk` is re-downloaded, not restored.

***

## Drift / surprises found (as of this capture)

* **i3 has THREE Longhorn disks, not two.** Beyond `disk0`/`disk1`, the 2TB
  SanDisk (`/dev/sdd`) hosts a Longhorn `bulk`-tagged disk at
  `/mnt/sandisk-2tb/longhorn` (disk name `sandisk2tb-bulk`) — the same physical
  drive also holds the media library mount. Prior notes framed the 2TB as
  media-only; it is dual-purpose.
* **`AGENTS.md` says `ssh i5` fails — it now works.** i5's `authorized_keys`
  contains pentium's pubkey (2 keys present). The "physical-console step in
  progress" note in AGENTS.md is stale.
* **`iscsi_tcp` drop-in + open-iscsi/nfs-common are on pentium too**, despite it
  being diskless — deliberate fleet uniformity, harmless.
* **The mergerfs fstab line on i3 references two SanDisks that aren't mounted**
  (`sandisk-1tb-lg`, `sandisk-1tb-dg`) — a media migration onto the `bulk` PVC is
  in flight. `nofail` keeps boot clean. Do not resolve this blindly.
* Workers have **no `/etc/rancher/k3s/config.yaml`** (empty) — correct; only the
  server carries `secrets-encryption`.