Kiosk SPA design — what the i5 screen shows

The runtime question is settled: keep Chrome+cage, and make the kiosk boot ONE URL that never changes — a self-navigating single-page app that re-renders itself over a live feed, so the browser never navigates again and the relaunch-to-navigate hack becomes dead code (docs/kiosk-display-runtime-eval.md, Option C; docs/kiosk-voice-visual-architecture.md, §5). This doc is the other half: WHAT that one page shows — its modes, how it switches between them client-side, where its data comes from, how it’s served, and a phased plan to build it. It is a design, not an implementation; nothing here is built yet. The one hard recommendation: ship a v1 that boots to a clock/ambient dashboard, killing today’s about:blank idle screen, before touching voice or agent-push. This builds on two decided facts and one just-shipped fix:

• Runtime = Chrome+cage, unchanged. The self-navigating SPA is the whole point of keeping it.
• Content pipeline = deploy-web-app. The agent already ships static apps to apps.augustin.ai/<name> in one atomic git commit (ConfigMap → Caddy). No new tooling.
• Touch now works (this session’s Part-1 fix): the kiosk is a real touchscreen again, and the compositor hides the cursor on touch / shows it on mouse. So the SPA is touch-first.

---

​

0. Ground truth (what actually exists today)

Read out of cluster/apps/** + images/** this session: The takeaway that shapes everything below: the SPA is a single static HTML file, served from a ConfigMap, running in a browser that lives inside the cluster network. That is simultaneously very powerful (direct in-cluster HTTP to HA/senses/agent) and very constrained (no build step, ≤1 MiB, no iframe of sibling apps, no raw MQTT).

---

​

1. The modes and how ONE page switches between them

From the voice+visual arch doc, the SPA is a client-side state machine with five modes. The whole design principle: mode transitions are DOM/CSS changes in one long-lived page, never navigations. The page loads once, opens its feed, and swaps what it renders. The switching mechanism (client-side): a single top-level state variable (mode) drives a <main data-mode> element; each mode is a component/section shown or hidden by CSS, and transitions are CSS opacity/transform cross-fades (300–500 ms). The feed (§4) pushes {mode, payload} messages; the SPA reduces them into state. So “fade to black → wake → cross-fade to dashboard → cross-fade to voice orb” is a sequence of state changes in one page, exactly what the relaunch model could never do. The IDLE fade is literally a CSS transition on the persistent DOM — a real cross-fade, not a page reload. “Show me X” — the hard one, because iframes are blocked. The founding intent is “show me images of X → pop up on the screen.” Three ways to realize AGENT_PUSH, given Caddy sends X-Frame-Options: DENY so the SPA cannot iframe a sibling apps.augustin.ai/<x>:

1. Render-in-a-pane (recommended default). The agent doesn’t push a URL to iframe; it pushes structured content the SPA knows how to render — {mode:"agent_push", kind:"gallery", items:[{img,caption}]} or {kind:"markdown", body:"…"} or {kind:"iframe_external", url:"…"}. The SPA has a small set of built-in renderers (image gallery, markdown/text card, a big headline, a chart). This covers the overwhelmingly common “show me images/a summary/a number” cases without any cross-origin iframe. It’s also the most touch/glanceable-friendly.
2. Hard navigate (escape hatch). For a genuinely arbitrary web app the agent built with deploy-web-app, fall back to the existing relaunch/kiosk/show {url} path — the SPA (or the navigator) points Chrome at apps.augustin.ai/<x> directly. This is a real navigation (accepts the cold-start flash) reserved for the rare “I built you a whole interactive app” case. The self-nav SPA stays the default boot page; this is the deliberate exception.
3. Iframe an allowlisted internal origin. If a pushed app is served from an origin that does not set X-Frame-Options: DENY (e.g. a dedicated content host, or Caddy relaxed to SAMEORIGIN/frame-ancestors self for a /embed/* prefix), the SPA can iframe it. Requires a Caddy config change — flagged as an option, not the v1 path.

Recommendation: default to #1 (render-in-a-pane with structured payloads) — it’s touch-native, needs no iframe, and keeps the persistent-page property. Keep #2 as the escape hatch for full apps. Treat #3 as a future nicety if arbitrary internal apps need embedding. The agent’s push payload gains a kind discriminator; the SPA owns the renderers.

---

​

2. Data sources and control flow

Because the SPA runs in a browser inside the cluster, it talks to services directly over HTTP/WS — the same reach the agent’s call-service has.

• Smart-home dashboard ← Home Assistant. Two options:
  • HA WebSocket API (ws://home-assistant.peripherals.svc.cluster.local:8123/api/websocket) — authenticate with a long-lived token, subscribe_events for state_changed, get push updates. Best for a live dashboard (no polling).
  • HA REST API (GET /api/states) — simpler, poll every few seconds. Fine for v1.
  • Recommend: REST-poll for v1 (dead simple, single file), upgrade to WS once the dashboard is real. Either needs a long-lived HA token minted and stored as a k8s secret; the SPA reads it… — and here’s a wrinkle: a static SPA has no server to hide a token behind. The token would be embedded in the page (in-cluster, output-only device, no inbound port, so the blast radius is contained, but it IS a token in client JS). Cleaner: proxy HA through the agent — the SPA calls the agent (which holds the token via call-service) instead of HA directly. Trade-off: adds the agent to the hot path for dashboard reads. For v1 the in-page token is acceptable given the device is inbound-portless and in-cluster; revisit if it bothers.
• Light / device controls → HA. Per the arch doc’s split:
  • Touch → HA directly (REST/WS from the browser): a tap on a light tile calls POST /api/services/light/turn_on. Sub-100 ms, no LLM. Right for direct manipulation.
  • Voice → agent → call-service → HA: “turn the lights warm” goes through the voice loop so the agent interprets intent. Same HA endpoint, two callers, chosen by modality.
• Voice viz ← the voice pipeline (another agent’s design): the SPA connects to the voice pod’s WebSocket for amplitude/partial-transcript/speaking events and drives the visualizer from that signal. Adopt the visual components (openai-realtime-blocks / agent-starter-react MIT widgets), re-driven from our WS. Out of scope to design here; the SPA just needs a voice mode section and a WS client stub.
• Presence / wake ← touch (now fixed) / camera-motion (senses) / feed events. Touch is a DOM event in-page (wakes IDLE→AMBIENT instantly, no round-trip). Motion comes from a senses-fed detector publishing a wake event onto the feed. The feed can also carry an explicit wake from the agent.

---

​

3. Framework choice — and should it share the console’s styling?

The dominant constraint is deploy-web-app: single index.html, ≤1 MiB, no build step. That rules out the console’s stack outright — you cannot ship Next.js/React-19/Tailwind-v4/ assistant-ui through a ConfigMap with no bundler. So the honest options are: Recommendation: build the SPA as a single index.html using Preact + htm (or Lit) via ESM CDN import, with inline CSS. It satisfies deploy-web-app natively, gives a real component/reactive model for the mode machine, stays well under 1 MiB, and the agent can regenerate it. Vanilla JS is the acceptable fallback if even the CDN dependency is unwanted. Should it share the console’s styling? Partially — share the tokens, not the toolchain, and diverge on layout. Arguments:

• For sharing: a consistent “Cortex” look across the desk console and the wall kiosk is nice; the console already defined a considered OKLCH dark palette (near-black surfaces, green primary, violet secondary). Copying those CSS custom-property values into the SPA’s inline :root{} is cheap and gives visual kinship for free.
• Against full sharing: the two surfaces have genuinely different jobs. The console is a lean-in, mouse+keyboard, dense chat/observability app (assistant-ui components, small text, thin scrollbars). The kiosk is a lean-back, glanceable, touch-first, across-the-room display: huge type, big tap targets (≥48 px), no hover-dependent UI, high-contrast tiles, and a dark ambient aesthetic that fades to black. Those layout/interaction needs are different enough that reusing the console’s components would fight the medium.

Verdict: adopt the console’s color tokens (the OKLCH @theme values) for brand kinship, but author kiosk-specific layout and components tuned for touch/glanceability. Same palette, different clothes. This keeps them recognizably one family without forcing a touch wall-display into a desktop chat app’s component set.

---

​

4. How it’s served + deployed — and the feed transport (the real gap)

Serving: ship the SPA via deploy-web-app as terminal → apps.augustin.ai/terminal. Then set the kiosk’s IDLE_URL env to https://apps.augustin.ai/terminal (one line in cluster/apps/peripherals/kiosk/deployment.yaml). The kiosk boots straight into it and never navigates again. Updating the UI = the agent re-runs deploy-web-app terminal (one commit, Flux reconciles, ~1 min). Zero new agent tooling — exactly the “don’t invent a pipeline” principle. A dedicated pod/image (console-style) is the alternative only if the SPA outgrows the 1 MiB / no-build ceiling; it should not for a glanceable dashboard. The feed transport is the one piece of missing plumbing, and the design must pick a lane: The SPA needs a live push feed to switch modes. mosquitto today is raw TCP 1883 only — a browser cannot connect to it. Options:

1. Add MQTT-over-WebSocket to mosquitto — add listener 9001 + protocol websockets to the mosquitto ConfigMap and expose 9001 on its Service; the SPA uses MQTT.js over WS. Keeps the existing kiosk/* topic design intact; the agent keeps publishing to MQTT as planned. This is the cleanest fit for the arch doc’s MQTT-centric mode machine. One small manifest change.
2. Use HA’s WebSocket as the bus — the SPA is already connecting to HA for the dashboard; HA can carry custom events too, and the agent already reaches HA via call-service. Fewer new listeners, but overloads HA with kiosk-control semantics and couples the feed to HA being up.
3. A tiny WS relay in the SPA’s own backend — not available, since deploy-web-app apps are static (no server). Rejected.

Recommendation: option 1 — add a WebSocket listener to mosquitto. It’s a ~3-line ConfigMap addition + one Service port, it preserves the whole kiosk/mode/kiosk/show/kiosk/wake topic design from the arch doc unchanged, and it makes the SPA a first-class MQTT client. This is the critical dependency to land before the mode machine can work end-to-end. (Note: it’s a peripherals-namespace central change — coordinate; it’s the mosquitto owner’s manifest, not the kiosk’s.)

---

​

5. Touch-first UX (the screen is a touchscreen again)

Now that touch works and the cursor hides on touch / shows on mouse (Part-1 fix), design for fingers, not a pointer:

• Big tap targets — tiles and buttons ≥ 48×48 px (ideally larger for across-the-room use), generous spacing so fat fingers don’t misfire.
• No hover-dependence — every affordance must be visible/reachable without hover. No hover tooltips carrying essential info; no hover-to-reveal controls. Use explicit expand/tap.
• Glanceable hierarchy — huge clock/primary info readable across the room; secondary info smaller but high-contrast. Dark theme, minimal chrome, content-forward.
• Direct manipulation — light sliders, toggle tiles respond instantly (HA REST/WS, no LLM in the loop for touch). Optimistic UI (flip the tile immediately, reconcile on the state event).
• The cursor — hidden during touch by the compositor patch, so a finger UI shows no stray arrow; a mouse (rare, for debugging) reveals it. The SPA can additionally set cursor:none and toggle on PointerEvent.pointerType as defense-in-depth, but the compositor already handles it page-independently, so this is optional belt-and-suspenders.
• Idle → fade to black — a full-screen CSS opacity transition on inactivity; touch anywhere wakes. No jarring reloads.

---

​

6. Phased plan (v1 first — kill about:blank)

Ordered to deliver value early and defer the hardest parts, mirroring the arch doc’s phasing but scoped to the SPA content:

• v1 — Boot to an ambient clock/dashboard (kills about:blank). Single index.html (terminal) with just IDLE + AMBIENT: a big clock + weather + a few HA tiles (REST-poll), fade-to-black on idle, tap-to-wake. deploy-web-app terminal, flip IDLE_URL to it. This alone replaces the blank idle screen with something worth looking at, and proves the persistent-page + HA-data path. No feed/MQTT needed yet (touch wake is a local DOM event; HA is polled). Smallest useful thing.
• v2 — LOCAL_CONTROL + live HA + the feed. Add device-control tiles (touch → HA directly), upgrade HA reads to the WebSocket subscription for live state, and stand up the MQTT-over-WS feed (§4) so the SPA can receive mode events. Now the walk-up local path is complete.
• v3 — AGENT_PUSH. Wire the agent’s kiosk/show {url|payload, priority, ttl} into the feed; implement the render-in-a-pane renderers (gallery/markdown/headline) + the hard-navigate escape hatch; add the interrupt-vs-toast arbitration from the arch doc. Delivers the remote “show me X” path unified with the walk-up path.
• v4 — VOICE_ACTIVE. Add the voice visualizer mode + transcript, driven from the voice pod’s WS (that pipeline is another agent’s design). Last because it depends on the voice pod and is the highest-risk piece.

Each phase is a deploy-web-app terminal re-ship — one commit, no new infra except the mosquitto WS listener at v2.

---

​

7. Riskiest unknown (flagged)

The browser feed transport, and specifically the HA-token-in-a-static-page tension. The mode machine is only as good as the live feed, and there is no browser-reachable bus today — mosquitto is TCP-only, and deploy-web-app apps are static (no server to broker a WS or hide a token). v1 dodges this (poll HA, local touch wake), but v2+ requires solving it. The clean answer (add MQTT-over-WS to mosquitto) is a central manifest change in someone else’s service, and the HA long-lived token either lives in client JS (contained but not pretty) or forces every dashboard read through the agent (adds latency + couples the dashboard to the agent). Neither is hard, but both are decisions that touch other components — so the SPA’s mode machine can’t be fully proven until the feed + token story is nailed down. Everything visual (modes, tiles, transitions, touch UX) is low-risk single-file work; the plumbing to feed it live is where the unknowns and cross-service coordination sit. Prove v1 (which needs none of it) first, then land the mosquitto WS listener before committing to v2’s live feed.

---

​

Sources / inputs

• docs/kiosk-display-runtime-eval.md — runtime decision (keep Chrome, self-nav SPA).
• docs/kiosk-voice-visual-architecture.md — the TERMINAL/CONTENT/AGENT split, modes, arbitration, voice.
• images/mastra/src/mastra/tools/deploy-web-app.ts + runtime/skills/deploy-web-app/SKILL.md — content pipeline (single index.html, ≤1 MiB ConfigMap, no build, one commit).
• images/mastra/src/mastra/tools/call-service.ts — agent’s in-cluster HTTP reach to HA (raw home-assistant…:8123).
• cluster/apps/webapps/ (Caddyfile ConfigMap, deployment, ingress) — apps.augustin.ai serving + the X-Frame-Options: DENY iframe constraint.
• cluster/apps/peripherals/{home-assistant,mosquitto,senses}/ — data sources; mosquitto has NO WebSocket listener (verified).
• images/console/ (package.json, app/globals.css) — the console’s Next/assistant-ui stack + OKLCH “Cortex” tokens (styling-share input).
• apps-src/mileage-tracker/index.html — reference single-file SPA pattern (inline CSS design system, CDN libs) to model the terminal app on.
• Part-1 fix (this session): touch enumerates in the compositor; patched cage hides cursor on touch / shows on mouse.