Node Prep (bare OS → cluster-ready)

The imperative, out-of-git layer that turns a bare Debian box into a working member of the k3s + Flux cluster. Flux reconciles the cluster/ tree, but it does not manage /etc/rancher/k3s/, /etc/fstab, disk formatting, OS packages, kernel modules, or SSH keys. Those are per-host and live only on the node. This page is the single authoritative record of that layer, captured from live node state so a node can be rebuilt from scratch. Companion runnable helpers: scripts/node-prep/ (idempotent; meant for rebuild time, not to be run against healthy nodes). The disaster-recovery playbook (recovery) references this page for node-level steps.

All values below were read from the live nodes (fstab UUIDs, IPs, k3s flags, Longhorn disk specs). Where a value is node-specific (disk UUIDs differ per box) it is listed per node — do NOT copy another node’s UUID. Re-read with blkid on the actual disk when rebuilding.

Fleet at a glance

Node	IP	Role / k3s	Longhorn disks	Node-local special files	OS pkgs
pentium	192.168.1.16	k3s server (control-plane), `NoSchedule` taint. `--disable traefik`, `--disable servicelb`, `--write-kubeconfig-mode 644`. `config.yaml`: `secrets-encryption: true`	none (diskless)	`config.yaml` (secrets-encryption), `local-storage.yaml.skip` + `coredns.yaml.skip` sentinels	open-iscsi, nfs-common
i3	192.168.1.100	k3s agent (worker), label `kind=storage`	`disk0` `/mnt/longhorn/disk0` (sdb1), `disk1` `/mnt/longhorn/disk1` (sdc1), `sandisk2tb-bulk` `/mnt/sandisk-2tb/longhorn` (sdd, tag `bulk`)	media mergerfs + sandisk mounts, GPU `/dev/dri`	open-iscsi, nfs-common, mergerfs
i5	192.168.1.10	k3s agent (worker), label `kind=compute`	`disk0` `/mnt/longhorn/disk0` (sda1)	ALSA mic state (ALC274, §7.5)	open-iscsi, nfs-common, alsa-utils, alsa-ucm-conf

(registries.yaml was retired 2026-07-19 — see §3.) k3s version across all three: v1.36.2+k3s1. OS: Debian 13 (trixie). Only pentium carries a config.yaml and the .skip sentinel — the workers’ /etc/rancher/k3s/config.yaml does not exist (empty), which is correct; the datastore + secrets-encryption live on the server.

1. Disk formatting + mounts (Longhorn data disks)

Longhorn stores replicas on plain ext4 filesystems mounted at well-known paths. Each data disk is a whole spare drive, single-partition (or the whole device), ext4, mounted by UUID with nofail so a missing disk never blocks boot.

Formatting procedure (per data disk, at rebuild)

# Identify the target device with lsblk FIRST — never assume sdX ordering.
sudo mkfs.ext4 -L <label> /dev/sdXN         # or partition first with parted/fdisk
sudo blkid /dev/sdXN                         # copy the UUID it prints
sudo mkdir -p /mnt/longhorn/diskN
# add the /etc/fstab line (below), then:
sudo systemctl daemon-reload
sudo mount -a

Helper: scripts/node-prep/format-mount-longhorn-disk.sh does exactly this idempotently (skips mkfs if the device already has a filesystem; appends the fstab line only if absent).

Live fstab entries (Longhorn + media only — copy per node)

i3 (/etc/fstab):

# Media drives (migrated from Pentium 2026-04-24)
LABEL=sandisk-2tb     /mnt/sandisk-2tb     ext4 defaults,nofail 0 0

# mergerfs pool across all three SanDisks -> /mnt/main/media
/mnt/sandisk-2tb/media:/mnt/sandisk-1tb-lg/media:/mnt/sandisk-1tb-dg/media /mnt/main/media fuse.mergerfs defaults,allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=epmfs,moveonenospc=true,minfreespace=20G,nofail 0 0
UUID=7ec23b6c-2089-4543-8592-8f08117f2d37 /mnt/longhorn/disk0 ext4 defaults,nofail 0 2
UUID=18af6df0-9549-47c1-a2f7-cbaad807f750 /mnt/longhorn/disk1 ext4 defaults,nofail 0 2

sdb1 UUID 7ec23b6c-2089-4543-8592-8f08117f2d37 → /mnt/longhorn/disk0 (932G)
sdc1 UUID 18af6df0-9549-47c1-a2f7-cbaad807f750 → /mnt/longhorn/disk1 (932G)
sdd UUID 023583fd-8474-43bb-9213-ccb9292434ce, LABEL sandisk-2tb → /mnt/sandisk-2tb (1.8T). Mounted by LABEL, not UUID. This drive is dual-purpose: it holds the media library and a Longhorn bulk disk at the /mnt/sandisk-2tb/longhorn subdir (see §2).
The mergerfs line references sandisk-1tb-lg/sandisk-1tb-dg which are not currently mounted (a media migration is in flight onto the bulk PVC). nofail keeps boot clean while they’re absent. Do not “fix” this during a rebuild without checking the current media-migration state.

i5 (/etc/fstab):

UUID=2870b24b-fc59-4265-9f70-c341d36c70f5 /mnt/longhorn/disk0 ext4 defaults,nofail 0 2

sda1 UUID 2870b24b-fc59-4265-9f70-c341d36c70f5 → /mnt/longhorn/disk0 (932G)

pentium: no Longhorn/data fstab entries — diskless for the cluster (only the OS disk sda with /, /boot/efi, swap).

pass field: OS root is 1, Longhorn data disks are 2 (fsck after root), the LABEL-mounted sandisk and mergerfs are 0 (no fsck). nofail on every data mount is mandatory — a Longhorn disk that fails to mount must not wedge boot.

2. Longhorn disk registration (nodes.longhorn.io CRD)

Formatting + mounting a disk is not enough — Longhorn must be told to schedule replicas onto it via the spec.disks map on the node’s Node CRD in longhorn-system. Longhorn auto-creates a disk0 at the default data path on a labeled node (create-default-disk-labeled-nodes: true, default-data-path /var/lib/longhorn/) but the extra disks and the bulk tag were added by hand and are node-local state (not in git). Read live with: kubectl -n longhorn-system get nodes.longhorn.io <node> -o jsonpath='{.spec.disks}'. i3 — three disks registered:

Disk name	path	allowScheduling	tags
`disk0`	`/mnt/longhorn/disk0`	true	`[]`
`disk1`	`/mnt/longhorn/disk1`	true	`[]`
`sandisk2tb-bulk`	`/mnt/sandisk-2tb/longhorn`	true	`["bulk"]`

i5 — one disk:

Disk name	path	allowScheduling	tags	storageReserved
`disk0`	`/mnt/longhorn/disk0`	true	`[]`	0

pentium — no Longhorn disks (diskless, tainted). The bulk disk tag on i3’s sandisk2tb-bulk is what pins the bulk StorageClass’s single-replica media volume onto that 2TB drive. The platform StorageClass (2 replicas across i3+i5) uses the untagged disks. (StorageClasses are in git: cluster/infra/longhorn/storageclasses.yaml; disk tags are not.) To re-register a disk after a rebuild (all data paths must already be mounted):

# Patch the node CRD's spec.disks. Example: add the bulk-tagged disk on i3.
kubectl -n longhorn-system patch nodes.longhorn.io i3 --type merge -p '{
  "spec": { "disks": {
    "sandisk2tb-bulk": {
      "path": "/mnt/sandisk-2tb/longhorn",
      "allowScheduling": true,
      "storageReserved": 0,
      "tags": ["bulk"],
      "diskType": "filesystem"
    }
  }}}'

(Or do it in the Longhorn UI: Node → Edit node and disks → Add Disk.) Longhorn will not schedule onto a path it can’t see mounted — create the mount first.

3. Registry access — `registries.yaml` retired 2026-07-19

The in-cluster registry was deleted; builds now push to GHCR (ghcr.io/amarcin/<name>:<short-sha>, built by in-cluster BuildKit — see cluster/infra/build/README.md). Nodes pull directly from GHCR, so the old insecure-mirror /etc/rancher/k3s/registries.yaml is no longer needed. On each node the file was renamed to registries.yaml.retired-2026-07-19 (kept as a breadcrumb, not read by containerd). On a fresh rebuild there is nothing to write here — skip it. If a node still has an active registries.yaml pointing at the dead ClusterIP 10.43.0.100, rename it and restart k3s (k3s-agent on workers, k3s on pentium).

4. k3s config & systemd flags (server vs agent)

k3s is installed from the official installer (get.k3s.io). The install writes a systemd unit (k3s.service on the server, k3s-agent.service on workers) whose ExecStart flags are baked at install time, plus it reads /etc/rancher/k3s/config.yaml at startup. Both were read live.

pentium — server (`k3s.service` ExecStart)

/usr/local/bin/k3s server \
  --disable traefik \
  --disable servicelb \
  --write-kubeconfig-mode 644 \
  --node-name pentium \
  --node-ip 192.168.1.16

Plus /etc/rancher/k3s/config.yaml:

secrets-encryption: true

--disable traefik — Traefik retired; Envoy Gateway is the sole router.
--disable servicelb — klipper/servicelb off; only mail uses a LoadBalancer (handled elsewhere). (Note: mail’s LoadBalancer is served by klipper on i3 per the mail docs — confirm servicelb handling if reworking; this flag is what live pentium reports.)
--write-kubeconfig-mode 644 — makes /etc/rancher/k3s/k3s.yaml world-readable so non-root can run kubectl.
secrets-encryption: true — enables k3s’s built-in AES-CBC encryption-at-rest for Kubernetes Secrets in the datastore. See §9 and recovery.md Scenario 3.

Install equivalent (fresh server):

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.2+k3s1 sh -s - server \
  --disable traefik --disable servicelb --write-kubeconfig-mode 644 \
  --node-name pentium --node-ip 192.168.1.16
# then write config.yaml with secrets-encryption and follow the enable flow (§9)

i3 / i5 — agents (`k3s-agent.service` ExecStart)

/usr/local/bin/k3s agent --node-name i3 --node-ip 192.168.1.100    # i3
/usr/local/bin/k3s agent --node-name i5 --node-ip 192.168.1.10     # i5

No config.yaml on the workers. Join a worker with the server URL + node token (cat /var/lib/rancher/k3s/server/node-token on pentium):

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.2+k3s1 \
  K3S_URL=https://192.168.1.16:6443 K3S_TOKEN=<node-token> sh -s - agent \
  --node-name i5 --node-ip 192.168.1.10

Both units also modprobe br_netfilter and overlay as ExecStartPre (k3s installer adds these).

Node labels & taint (post-join, via kubectl)

kubectl label node i3 kind=storage
kubectl label node i5 kind=compute
kubectl taint node pentium node-role.kubernetes.io/control-plane=true:NoSchedule
# (workers also carry node-role.kubernetes.io/worker=true; pentium is control-plane)

Live state confirms: i3 kind=storage, i5 kind=compute, pentium tainted node-role.kubernetes.io/control-plane=true:NoSchedule, workers untainted.

5. local-storage default-class override (pentium only)

There is intentionally no cluster-default StorageClass: a class-less PVC should fail to bind (forcing explicit platform/bulk intent) rather than silently landing on node-local single-copy local-path. But k3s ships a bundled local-storage addon that annotates local-path as is-default-class=true and re-applies it on every restart. The override is two node-local pieces on pentium (the server owns addon manifests):

A zero-byte sentinel that tells k3s to stop managing that addon:

sudo touch /var/lib/rancher/k3s/server/manifests/local-storage.yaml.skip

Turn the default annotation off (durable once the .skip exists):

kubectl annotate sc local-path storageclass.kubernetes.io/is-default-class=false --overwrite

Verified live on pentium: local-storage.yaml.skip (0 bytes) sits next to local-storage.yaml in /var/lib/rancher/k3s/server/manifests/. The .skip makes k3s bow out of managing that addon (the running provisioner is left intact, so local-path stays available for anyone who names it explicitly) — so the annotation stays off across restarts. This is deliberately not in git: the StorageClass is owned by the k3s addon controller and a Flux-managed copy would fight it. See recovery.md “No default StorageClass” and cluster/infra/longhorn/storageclasses.yaml.

5.1. CoreDNS — the second `.skip` sentinel

Same mechanism, opposite conclusion: cluster DNS is owned in git (cluster/infra/coredns/, replicas: 2) — k3s ships 1 replica on the tainted control-plane, a fleet-wide DNS single point of failure.

sudo touch /var/lib/rancher/k3s/server/manifests/coredns.yaml.skip

Unlike --disable coredns (which deletes the live objects), the sentinel only stops re-application, so the handover has no DNS outage. Apply the leaf first, then the sentinel. Full rationale, the post-upgrade diff command and the NodeHosts trap live in the cluster/infra/coredns/kustomization.yaml header comment.

6. OS packages Longhorn needs

Longhorn’s engine attaches volumes over iSCSI and serves RWX volumes over NFS, so every node that could run a Longhorn workload needs:

Package	Why
`open-iscsi`	iSCSI initiator — Longhorn attaches every volume via iSCSI
`nfs-common`	NFS client — Longhorn RWX (`bulk`) volumes are served over NFS (share-manager)

Verified installed on all three nodes (version open-iscsi 2.1.11-1+deb13u2, nfs-common 1:2.8.3-1). iscsid is enabled + active on all three.

sudo apt-get update
sudo apt-get install -y open-iscsi nfs-common
sudo systemctl enable --now iscsid

Helper: scripts/node-prep/install-longhorn-prereqs.sh. pentium is diskless and runs no Longhorn workloads, but it has these installed too (harmless, keeps the fleet uniform).

7. Kernel modules & sysctls

iscsi_tcp — persisted on all three nodes

Longhorn needs the iscsi_tcp module. It is made persistent via a drop-in (verified present on all three nodes):

# /etc/modules-load.d/longhorn-iscsi.conf
iscsi_tcp

lsmod confirms iscsi_tcp loaded on all three. (Recreate the file and sudo modprobe iscsi_tcp if missing after a rebuild.)

i915 — Intel GPU, auto-loaded

i915 is loaded on all three nodes automatically (in-tree module, loads for the Intel iGPU at boot) — no explicit modules-load drop-in needed. i3 and i5 expose /dev/dri/{card0,renderD128} (card0 = video group, renderD128 = render group). pentium’s Pentium G3220 loads i915 but has no usable QuickSync (hardware.md). The Intel GPU device plugin (advertising gpu.intel.com/i915) is GitOps (cluster/infra/intel-gpu/) — nothing to do at node level beyond having the module + /dev/dri, which come from the stock Debian kernel.

sysctls

k3s manages the networking sysctls it needs. The k3s systemd unit modprobes br_netfilter + overlay on start (ExecStartPre). Live values on pentium: net.bridge.bridge-nf-call-iptables=1, net.ipv4.ip_forward=1 — these are set by k3s/flannel, not by a hand-authored sysctl drop-in. No custom sysctl file is required for node prep.

7.5. i5 audio input — mic capture for Home Assistant voice (ALSA)

i5 is a Dell Inspiron AIO with a Realtek ALC274 codec on card 0: PCH [HDA Intel PCH], capture device plughw:0,0. The consumer is the kiosk pod (ns peripherals, privileged, /dev/snd): its Chrome captures the mic via getUserMedia and plays TTS back through the speaker, both on card 0. The “hey jarvis” wake word is detected in-browser by the kiosk-ui SPA (LiveKit participant kiosk); there is no host-level or in-pod wake-word daemon. The host-level ALSA state below (mixer levels, capture unmute, alsactl persistence) is what makes that mic path deliver signal at all. Failure mode seen (2026-07-11): the mic captured only silence — a consumer opening plughw:0,0 read a flat signal and no audio ever reached the app. Root cause was purely host-level ALSA state: alsa-utils was never installed on this fresh headless box, so alsactl/amixer had never run and the codec powered up with capture muted and Internal Mic Boost at zero. A capture of plughw:0,0 streamed dead silence (measured peak 0, rms 0.0 over a 3 s capture). (The mixer controls were always exposed by the kernel driver — there was just no userspace tool present to read or set them, and nothing to unmute capture at boot.) Fix (host-level on i5, out-of-git):

# 1. Install ALSA userspace (official Debian repos). Also pulls alsa-ucm-conf.
sudo apt-get update
sudo apt-get install -y alsa-utils alsa-ucm-conf

# 2. Unmute + set the internal-mic capture path on card 0.
amixer -c 0 sset Capture 60% cap          # ADC: 60% (+11.25 dB), capture ON
amixer -c 0 sset 'Internal Mic Boost' 0%  # boost 0 dB — 60% cap alone is plenty
amixer -c 0 sset 'Internal Mic' cap       # select internal DMIC as capture source
# (Master output was also muted by Auto-Mute; unmute so the TTS *response* is audible)
amixer -c 0 sset Master 80% unmute

# 3. Persist across reboot.
sudo alsactl store                        # writes /var/lib/alsa/asound.state

Level tuning: Capture 100% + Boost 100% (=42 dB) pegged the ADC at full-scale (peak 32767, clipping). Capture 60% + Boost 0% gives a clean peak ~13700 with headroom and a low noise floor — the level stored above. Persistence mechanism: installing alsa-utils drops in /usr/lib/udev/rules.d/90-alsa-restore.rules, which runs alsactl restore when the sound control device appears at boot — no unit to enable (alsa-restore.service is static, driven by that udev rule). Verified by muting Capture to 0%/off, running the exact sudo alsactl -E HOME=/run/alsa -E XDG_RUNTIME_DIR=/run/alsa/runtime restore 0 the rule fires, and confirming it recovered to 60% [on]. Verify capture has signal (free the device first — the kiosk pod’s Chrome holds it):

kubectl scale deploy kiosk -n peripherals --replicas=0   # release mic
ssh i5 'arecord -D plughw:0,0 -f S16_LE -r 16000 -c 1 -d 3 /tmp/t.wav'
# make noise during the 3 s; then check peak/rms is non-zero (python wave module)
kubectl scale deploy kiosk -n peripherals --replicas=1   # restore

i3 has no equivalent mic requirement; this section is i5-only.

8. SSH topology (bastion + inter-node)

pentium is the SSH bastion; the Mac reaches it over the Cloudflare tunnel (ssh ssh.augustin.ai). pentium reaches the workers by key auth over the LAN. pentium’s ~/.ssh/config (verified live):

Host i3
    HostName 192.168.1.100
    User user

Host i5
    HostName 192.168.1.10
    User user
    IdentityFile ~/.ssh/id_ed25519

pentium → i3: works (key auth). i3 uses the default id key.
pentium → i5: works (verified live — ssh i5 succeeds; i5’s authorized_keys has pentium’s pubkey. This corrects the stale “currently fails” note in AGENTS.md, which predates the key being installed.)

pentium’s public key (~/.ssh/id_ed25519.pub) is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3ekCUNW00tqzjFg0lU61jhJi8pn9qDWRy+0Iespb34 user@debian-2

Re-establishing keys after a rebuild

If pentium is rebuilt (new keypair): copy its new pubkey onto each worker. If a worker is rebuilt: append pentium’s existing pubkey to the new worker’s authorized_keys.

# From pentium, once its keypair exists (ssh-keygen -t ed25519 if new):
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@192.168.1.100   # i3
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@192.168.1.10    # i5
# If the worker has password auth disabled, append the pubkey at its physical
# console instead:
#   echo 'ssh-ed25519 AAAA... user@debian-2' >> ~/.ssh/authorized_keys

(Never copy private keys off the bastion. The pubkey above is safe to paste.)

9. Secrets encryption-at-rest (pentium / datastore)

Kubernetes Secrets are encrypted at rest in the k3s datastore via k3s’s built-in AES-CBC provider. Node-local on pentium; neither the flag nor the AES key is in git.

secrets-encryption: true in /etc/rancher/k3s/config.yaml (see §4).
The AES key lives in the datastore’s cluster bootstrap data and travels with a datastore restore — so restoring the original datastore needs only the flag back before starting k3s.

On a fresh datastore, re-run the enable flow:

echo "secrets-encryption: true" | sudo tee /etc/rancher/k3s/config.yaml
sudo k3s secrets-encrypt enable
sudo systemctl restart k3s          # wait for /healthz = ok
sudo k3s secrets-encrypt rotate-keys
sudo systemctl restart k3s
sudo k3s secrets-encrypt status     # Enabled, stage reencrypt_finished, AES-CBC active key

Full detail: recovery.md Scenario 3 + “Secrets encryption-at-rest” under Known Gaps.

Rebuild order (bare OS → cluster-ready)

Install Debian 13. Create the user account. Configure SSH (§8).
Install Longhorn prereqs on workers (and pentium for uniformity) — §6 (install-longhorn-prereqs.sh). Persist iscsi_tcp — §7.
Workers only: format + mount each Longhorn data disk — §1 (format-mount-longhorn-disk.sh). Confirm /dev/dri exists on i3/i5 (§7).
Install k3s (server on pentium, agents on i3/i5) with the exact flags — §4. On pentium also write config.yaml + run the secrets-encryption enable flow — §9.
registries.yaml is retired — nothing to write here (§3). Restart k3s workers-first only if a stale file is still active.
On pentium: create the local-storage.yaml.skip sentinel + un-default local-path — §5. Apply node labels + the pentium taint — §4.
flux bootstrap github. Secrets are SealedSecrets — the controller’s sealing key decrypts them in-cluster, so there is no key to install on bootstrap (unlike the old SOPS age key). If the sealing key was lost with the control-plane, restore it first (recovery.md Scenario 4). Flux reconciles Longhorn.
Workers: register Longhorn disks on the node CRDs (extra disks + the bulk tag on i3) — §2.
Restore application data from restic into the fresh Longhorn PVCs (recovery.md Scenario 1). Media library on bulk is re-downloaded, not restored.

Drift / surprises found (as of this capture)

i3 has THREE Longhorn disks, not two. Beyond disk0/disk1, the 2TB SanDisk (/dev/sdd) hosts a Longhorn bulk-tagged disk at /mnt/sandisk-2tb/longhorn (disk name sandisk2tb-bulk) — the same physical drive also holds the media library mount. Prior notes framed the 2TB as media-only; it is dual-purpose.
AGENTS.md says ssh i5 fails — it now works. i5’s authorized_keys contains pentium’s pubkey (2 keys present). The “physical-console step in progress” note in AGENTS.md is stale.
iscsi_tcp drop-in + open-iscsi/nfs-common are on pentium too, despite it being diskless — deliberate fleet uniformity, harmless.
The mergerfs fstab line on i3 references two SanDisks that aren’t mounted (sandisk-1tb-lg, sandisk-1tb-dg) — a media migration onto the bulk PVC is in flight. nofail keeps boot clean. Do not resolve this blindly.
Workers have no /etc/rancher/k3s/config.yaml (empty) — correct; only the server carries secrets-encryption.

​Fleet at a glance

​1. Disk formatting + mounts (Longhorn data disks)

​Formatting procedure (per data disk, at rebuild)

​Live fstab entries (Longhorn + media only — copy per node)

​2. Longhorn disk registration (nodes.longhorn.io CRD)

​3. Registry access — registries.yaml retired 2026-07-19

​4. k3s config & systemd flags (server vs agent)

​pentium — server (k3s.service ExecStart)

​i3 / i5 — agents (k3s-agent.service ExecStart)

​Node labels & taint (post-join, via kubectl)

​5. local-storage default-class override (pentium only)

​5.1. CoreDNS — the second .skip sentinel

​6. OS packages Longhorn needs

​7. Kernel modules & sysctls

​iscsi_tcp — persisted on all three nodes

​i915 — Intel GPU, auto-loaded

​sysctls

​7.5. i5 audio input — mic capture for Home Assistant voice (ALSA)

​8. SSH topology (bastion + inter-node)

​Re-establishing keys after a rebuild

​9. Secrets encryption-at-rest (pentium / datastore)

​Rebuild order (bare OS → cluster-ready)

​Drift / surprises found (as of this capture)