Multi-node sensor fabric (cameras + mics on any node)

• Camera → the model SEES a JPEG. Opus 4.8 is vision-capable. camera-snapshot already fetches one frame and hands it to the model as a viewable image. This path works today and generalizes cleanly to multi-node — it is just more registry entries.
• Mic → the model reads a TRANSCRIPT. Opus 4.8 cannot process raw audio. A “listen” tool therefore cannot hand the model audio. It must capture N seconds of PCM from the node’s senses pod, run it through shared STT → text, and return a transcript (+ metadata). This path depends on the shared STT service the voice access-point is being built around (see kiosk-voice-visual-architecture §4) — the mic tool consumes that endpoint; it does not build a second STT.

​

0. Hardware findings — what is ACTUALLY on each node

1. Augustin’s “USB camera on i3 or pentium” is on i3 — a Logitech Brio 101, and it’s a combo device (camera and mic). Pentium has nothing attached.
2. Two camera+mic nodes exist today: i5 (built-in) and i3 (USB Brio). The fabric has a real second node to prove multi-node against — it isn’t a hypothetical.
3. pentium is a non-starter for sensors on two independent grounds: no hardware, and the NoSchedule control-plane taint (§1). Even if a USB cam were plugged into pentium, it’s the bastion/ingress box we deliberately keep workload-free — the right move would be to plug the device into i3 or i5 instead of tolerating the taint. Design notes the toleration path for completeness but recommends against sensors on pentium.
4. The i5 mic and i3 mic are both capture-capable (verified via the capture column), so a mic-listen tool has real targets on both worker nodes once STT exists.

​

1. Per-node senses pods

​

The pattern (parameterized from the live i5 pod)

• the camera tool’s static registry (STATIC_REGISTRY.i5.snapshotUrl = http://senses.peripherals.svc.cluster.local:8080/snapshot), and
• nothing else references it by name, but renaming would still churn the running pod the kiosk path and the camera tool exercise.

​

The device-gate reality (why privileged: true, on every node)

​

pentium and the NoSchedule taint

tolerations:
  - key: node-role.kubernetes.io/control-plane
    effect: NoSchedule
    operator: Exists

​

Device contention (the one real per-node caveat)

• The camera is fine — ustreamer is the single owner and everyone reads its HTTP /snapshot (multi-reader over HTTP, single-owner of the device). The fabric preserves this: the pod owns the device, HTTP fans out.
• The mic needs care — if the voice pod (§4) and the mic-listen tool both want raw /dev/snd capture on the same node, they contend. The design resolves this by making the senses pod the single owner of audio capture too, exposing capture over HTTP (§3), so the mic-listen tool and any other consumer read the endpoint, not the device. One owner per device node, HTTP fan-out — the same discipline as the camera.

​

2. The unified sensor registry

​

What exists (camera-snapshot.ts, read-only)

• A static Record<name, {snapshotUrl, note}> baked into the image, seeded with i5.
• An optional dynamic table read per-call from config/cameras.json on the durable PVC (CONFIG_DIR = sibling of PROMPTS_DIR; on the live mastra pod that resolves to /app/data/config/cameras.json, on the mastra-data PVC).
• Merge rule (safety): dynamic may ADD names but may NOT override a static entry — a runtime write can’t repoint an existing sensor at a hostile URL.
• SSRF bound: a sensor URL must resolve in-cluster (*.svc.cluster.local / short name / private RFC1918 / loopback) — enforced both at registration-read and at fetch.

​

The extension: one registry, kind + node

type SensorEntry = {
  kind: "camera" | "mic";     // what the sensor is → which tool reads it, which path (image vs transcript)
  node: string;                // which box it's physically on: "i5" | "i3" | …  (documentation + human addressing)
  endpoint: string;            // in-cluster URL of the senses-pod endpoint:
                               //   camera → …/snapshot   (GET → one JPEG, the model SEES it)
                               //   mic    → …/capture?seconds=N  (GET → audio bytes, tool STT-transcribes)
  note?: string;
  source?: "static" | "dynamic";
};

​

Static seed (from §0’s real hardware)

{
  "i5":     { "kind": "camera", "node": "i5", "endpoint": "http://senses.peripherals.svc.cluster.local:8080/snapshot",     "note": "i5 AIO built-in UVC webcam (Integrated_Webcam_FHD). 1280x720." },
  "i3":     { "kind": "camera", "node": "i3", "endpoint": "http://senses-i3.peripherals.svc.cluster.local:8080/snapshot",  "note": "i3 USB Logitech Brio 101 webcam." },
  "i5-mic": { "kind": "mic",    "node": "i5", "endpoint": "http://senses.peripherals.svc.cluster.local:8080/capture",      "note": "i5 AIO built-in mic array (ALC274 analog). Needs shared STT." },
  "i3-mic": { "kind": "mic",    "node": "i3", "endpoint": "http://senses-i3.peripherals.svc.cluster.local:8080/capture",   "note": "i3 Brio 101 USB mic (or onboard ALC3220). Needs shared STT." }
}

​

Alignment path — do NOT break the live camera tool

• Preferred — additive superset. Add kind/node/endpoint as the canonical fields; keep reading snapshotUrl as an alias for a kind:"camera" endpoint so existing config/cameras.json entries and the i5 static seed keep working untouched. The camera tool filters the registry to kind === "camera" (or entries with a snapshotUrl); the mic tool filters to kind === "mic". One registry file, two consumers, zero migration.
• File name: keep config/cameras.json or promote to config/sensors.json with cameras.json read as a legacy fallback. Recommend sensors.json as canonical + cameras.json fallback so the concept name matches the fabric, without orphaning any runtime-registered camera. This is a mastra-src change (owned by the other agent) — spec’d here, not done.

​

3. The senses-pod audio-capture endpoint (new; camera path already exists)

​

Requirements

• GET /capture?seconds=N → return N seconds of captured audio as a single response (a bounded WAV/PCM blob, Content-Type: audio/wav), then stop. Not a stream — a sample, to mirror the camera’s “one frame” semantics. Bound N (e.g. 1–30s) so a call can’t run away.
• Contained / shell-free-ish, same ethos as the camera path: no arbitrary command surface, in-cluster-only, no state change. The endpoint is a read of the mic, exactly as /snapshot is a read of the camera.
• Single owner of /dev/snd on its node (§1 contention note) — the senses pod owns audio capture; the voice pod and the mic-listen tool both read this HTTP endpoint, not the device.

​

Implementation options (evaluated)

1. A tiny purpose-built HTTP audio-capture service in the senses image (recommended). Add a small binary/daemon to images/senses/ (alongside ustreamer) that, on GET /capture, opens the ALSA capture PCM, records N seconds to an in-memory WAV, and returns it. Language: a ~60-line Go binary (single static binary, no interpreter, matches the shell-free ethos) or a minimal C using ALSA libasound directly. No arecord shell-out — call the ALSA API in the process, so there is no command string in the loop (same reasoning the agent tools are shell-free). Runs on a second port (e.g. :8081) in the same pod, or the same pod with two containers (ustreamer + audiocap) sharing /dev/snd. This keeps “one senses pod owns the node’s sensors” true for both camera and mic.
2. arecord-over-HTTP shim (rejected as the primary). A shim that shells arecord -d N -f cd file.wav then serves the file is the quickest to write but reintroduces a shell/command surface into a device-owning privileged pod — exactly the thing the rest of the stack avoids. Acceptable as a throwaway Phase-1 spike to prove the mic captures at all, not the shipped design.
3. ustreamer’s audio (rejected — doesn’t fit). ustreamer can carry audio, but only as part of its Janus/WebRTC path (WITH_JANUS), which the senses Dockerfile explicitly builds OFF (WITH_JANUS=0) and which produces a stream for a browser, not a bounded sample for a tool. Wrong shape (stream vs sample) and wrong transport (WebRTC vs a plain GET). Don’t turn it on for this.

​

Sample rate / format

​

4. The mic-listen tool (spec for the mastra agent)

​

Flow

listen({ mic: "i3-mic", seconds: 8 })
   │
   ├─ resolve "i3-mic" in the unified registry → { kind:"mic", node:"i3",
   │     endpoint:"http://senses-i3.peripherals.svc.cluster.local:8080/capture" }
   │     (reject if kind !== "mic"; reject if endpoint not in-cluster — SSRF bound)
   │
   ├─ GET {endpoint}?seconds=8  → 16kHz mono WAV blob from the i3 senses pod (§3)
   │
   ├─ POST that WAV to the SHARED STT endpoint  (the voice pipeline's STT — §5 dependency)
   │     e.g. POST http://voice.peripherals.svc.cluster.local:<port>/stt  (multipart audio/wav)
   │     → { text, ... }
   │
   └─ return { ok, mic, node, seconds, transcript, ... }   ← TEXT, the model reads it

​

Tool interface (drop-in shape, mirrors camera-snapshot)

export const micListen = createTool({
  id: "listen",
  description:
    "Capture a few seconds of audio from an in-fleet MICROPHONE and return a TRANSCRIPT of what " +
    "was said/heard (speech-to-text). Use it to hear a physical space on demand: is someone " +
    "talking near a node, what did they say, is there noise. Mics are named; known: " +
    "i5-mic, i3-mic (plus any registered in config/sensors.json). NOTE: this returns TEXT, not " +
    "audio — the audio is transcribed by the shared STT service first (the model cannot process " +
    "raw audio). Read-only, shell-free, in-cluster only. An unknown/unreachable mic, or STT " +
    "being unavailable, returns a structured {ok:false,detail} — it never throws.",
  inputSchema: z.object({
    mic: z.string().default("i5-mic")
      .describe("mic name, e.g. 'i5-mic' or 'i3-mic'. Defaults to 'i5-mic'."),
    seconds: z.number().int().min(1).max(30).default(8)
      .describe("how many seconds of audio to capture and transcribe (1–30)."),
  }),
  outputSchema: z.object({
    ok: z.boolean(),
    mic: z.string(),
    node: z.string().optional(),
    seconds: z.number().optional(),
    transcript: z.string().optional().describe("STT text of the captured audio — the model reads THIS"),
    detail: z.string(),
  }),
  // NO toModelOutput image path — unlike camera-snapshot, there is nothing for the model to SEE.
  // The transcript is a plain string field the model reads directly. (If STT returns word/segment
  // timings or a confidence score, carry them in `detail` or extra fields, still as text.)
  execute: async ({ mic, seconds }) => { /* resolve registry → GET /capture → POST STT → return transcript */ },
});

​

Design points

• Returns a transcript, not audio. There is no toModelOutput media part (the camera tool’s whole trick). The transcript string is the payload; the model reads text. This is the single most important consequence of the “Opus can’t hear” constraint and it must be unmistakable in the tool so the agent never expects to “listen to” audio.
• Two network hops, both fail-soft. /capture on the node’s senses pod, then the shared STT. Each is a structured {ok:false, detail} on failure (unknown mic, capture failed, STT unavailable), never a throw — a bad listen degrades the turn gracefully, same as a bad snapshot.
• SSRF-bounded, same as camera-snapshot: both the /capture endpoint and the STT endpoint must be in-cluster. The mic tool reuses the camera tool’s isInClusterHost bound verbatim.
• Same registry, filtered to kind:"mic". No second registry, no second config file.
• Default mic = i5-mic (mirrors camera default i5), the always-present built-in.

​

5. Sensor-TOOL vs the voice access-point — the boundary

​

6. Phasing + what’s built vs designed

​

Buildable now (camera path — no STT dependency)

• P1 — Generalize senses to per-node; add i3 (SAFE, ADDITIVE, staged + validated). The i3 Brio is real hardware (§0). Add a senses-i3 Deployment + Service (pinned kind=storage, privileged, /dev/video0+/dev/snd mounts) alongside the untouched senses i5 pod — the bare senses name is grandfathered so the live camera tool + kiosk are undisturbed. Wire senses-i3 into cluster/apps/peripherals/kustomization.yaml. This was drafted and offline-validated clean this session (kustomize build | kubeconform -strict → 2/2 valid). It is the lowest-risk step and proves the multi-node fabric with a second real camera.
• P2 — Register i3 camera in the unified registry. Add i3 (kind:camera, node i3, senses-i3.../snapshot) to config/sensors.json on the mastra PVC → camera-snapshot can see the i3 Brio immediately, no rebuild. (Registry-file write; the tool already reads dynamic entries.)
• P3 — Unify the registry type in the camera tool (mastra src, other agent). Add kind/node/endpoint as the canonical superset, keep snapshotUrl as a back-compat alias, promote sensors.json (fallback cameras.json). Backward-compatible; no migration. Spec’d §2; not editable here.

​

Waits on the shared STT (mic path)

• P4 — Add the /capture audio endpoint to the senses image (images/senses/ — a small in-image ALSA→WAV HTTP service, §3). Buildable independent of STT (it just returns audio), but only useful once STT exists to transcribe it. Reasonable to build alongside P5.
• P5 — The voice pod exposes POST /stt (the coordination point, §5) — owned by the voice-pipeline agent. This is the gate for the whole mic path.
• P6 — The listen mic tool (mastra src, other agent) — capture from a node’s /capture → shared STT → return transcript. Spec’d §4; blocked on P4 + P5; not editable here.

​

What this session built vs designed

• Built (safe, additive): nothing committed to the live tree beyond this doc — the senses-i3 manifest set was authored and offline-validated (kustomize build | kubeconform -strict → 2 resources, 0 errors) but staged in /tmp for review rather than committed, because adding a second privileged device-pod that grabs the i3 Brio is a live-fleet change worth an explicit go-ahead, not something to slip in under “additive.” The exact manifests are in §1’s table + reproduced below; dropping them into cluster/apps/peripherals/senses-i3/ and adding - senses-i3 to the peripherals kustomization is a one-commit, Flux-reconciled step whenever approved.
• Designed (not built): the unified registry type change (mastra src, other agent), the /capture endpoint on the senses image, the listen mic tool (mastra src, other agent), and the POST /stt coordination with the voice-pipeline agent.

​

The staged senses-i3 manifests (validated, ready to commit when approved)

​

Constraints honored

• k3s + OS only — every sensor is a device-mount pod; no host daemon (the peripherals-doc invariant).
• Opus 4.8 vision-but-not-audio — camera returns an image the model sees; mic returns a transcript the model reads. Never audio to the model.
• One STT, shared — the mic tool is a client of the voice pipeline’s STT, not a second STT.
• Shell-free / SSRF-bounded — the /capture endpoint uses the ALSA API in-process (no arecord shell-out); both tools bound sensor + STT URLs to in-cluster hosts, reusing the camera tool’s existing bound.
• Least-privilege-where-possible, privileged-for-hardware-where-forced — privileged: true on senses pods is the documented cgroup-v2 device-gate necessity, contained by namespace + no API token + no ingress, exactly as the live senses/kiosk pods are.
• Don’t fork the registry — one unified name → {kind, node, endpoint} table, the camera tool’s existing pattern extended, static-wins-on-collision + PVC-extensible preserved.