Skip to main content

The Self-Capable Arc — a Broad Code-Eval Capability + a Safe Write-Wall Takedown

Status: PLAN / design doc for review. No code changed, nothing deployed. Author: architecture pass (Opus 4.8), 2026-07-06. Scope: design a broad, general-purpose “execute + evaluate arbitrary code” capability for the homelab Mastra agent, and the safe removal of the write-wall that currently keeps the agent out of its own source (images/). The last step of the arc is letting the agent commit code changes — but only behind a real, tested verification loop.
Extension — concurrent workstreams + conflict resolution: once the agent self-edits AND self-deploys (all 9 images incl. its own brain, turn 109) with MULTIPLE workstreams running at once, their commits collide. The branch-per-workstream isolation + git-does-the-mechanical-merge + LLM-resolver-for-intent design lives in concurrent-workstreams-design.md. It builds directly on this doc’s verify() Layer-1 primitive (re-verifying every merge before deploy) and slots into Phases 4–5 below as a new Phase 6.

1. Goal + principles

Goal. Give the main agent a broad capability to safely modify and verify code — its own AND any other code — and, behind that verification, let it commit code changes to the repo. Two distinct powers fall out of this:
  1. Evaluate arbitrary code (broad; no repo write needed): run + test any code in an isolated sandbox and get a real pass/fail. This works for a fresh Rust CLI in a scratch dir, a new service, someone else’s repo — anything, not just the agent’s own image repo.
  2. Commit a verified change to a repo path (needs a widened WRITE_ROOTS): once a change has passed the verification loop, land it in images/** so CI rebuilds and Flux reconciles.
Augustin’s design principles (hard requirements; tensions called out in §6):
  • P1 — Broad modules, not narrow ones. “A few broad tools rather than lots of general-specific ones.” Large, general-purpose capabilities that can be used for anything.
  • P2 — Code-eval must be general. “If we’re giving it the power to evaluate code, it should do that for ANY code, not just its own image repo.” The capability is “run + evaluate arbitrary code in an isolated sandbox,” full stop.
  • P3 — The main agent stays shell-free. The main agent must never wield a host execute_command / command-interpreter tool. The established reconciliation (already in the codebase, REWRITE-NOTES turn 105): running arbitrary code inside an isolated throwaway pod, driven from within a workflow/tool and never attached to an Agent, does NOT violate shell-free — because attaching a sandbox to an Agent is precisely what auto-injects execute_command. This plan preserves that: broad execution power, isolated execution, main-agent tool surface never gains a raw host shell.
  • P4 — Do more with less code. Capability-per-line is the north star. Don’t build a grand framework.

2. Investigation findings (ground truth)

2.1 Mastra’s current relevant features

Pinned version, from images/mastra/package.json: @mastra/core: ^1.47.0 (caret; any 1.x ≥ 1.47.0 resolves). REWRITE-NOTES and code comments say 1.49.0 — that’s the resolved lockfile version, consistent with the ^1.47.0 range. @mastra/docker and @mastra/agentcore are not dependencies. Sources: https://mastra.ai/llms.txt and the doc pages it indexes. All version markers below are the docs’ own “Added in @mastra/core@x.y.z” lines.
FeatureWhat it actually doesMaturityFit
Code Mode (docs/agents/code-mode.md, reference/tools/create-code-mode.md)The model authors one TypeScript function per query that orchestrates your existing tools (exposed to the generated code as external_<toolId>). That orchestration code runs in a Workspace sandbox; each external_* call bridges back to the real tool on the host. Adds ONE tool, execute_typescript (not a raw execute_command). Requires a sandbox; LocalSandbox “runs the function as a host node process with host privileges.”BETA (“breaking changes may occur without a major bump”). Added in 1.38.0 (available under ^1.47.0).Solves a tool-fan-out problem this single-user agent doesn’t have. Its generated code runs sandboxed, but the point of the feature is orchestrating the agent’s own tools, not “evaluate arbitrary code.” Not the shape we want.
Workspace family (docs/workspace/overview.md): Filesystem, Sandbox, LSP, Search, SkillsA Workspace bundles five capability areas. Attached via the workspace config key on a Mastra instance or an Agent. Attaching a Sandbox auto-injects execute_command (+ get_process_output / kill_process if the sandbox supports background processes); a Filesystem injects read_file/write_file/list_directory/grep; LSP injects mastra_workspace_lsp_inspect.Added 1.1.0; no beta label. Process manager added 1.7.0.The Sandbox interface is exactly what we already implement (see §2.2). The auto-injection-on-attach behavior is the thing P3 forbids at the agent — so we use the interface but never attach.
WorkspaceSandbox interface (reference/workspace/sandbox.md)Required: start(), executeCommand(command, args?, options?) → CommandResult, destroy(), getInfo(). Optional: stop?(), getInstructions?(), processes (a SandboxProcessManager; omit → no background-process tools). File I/O is NOT on this interface — it lives on WorkspaceFilesystem.Added 1.1.0.This is the interface K8sExecSandbox already implements. We’re conformant today.
Built-in sandbox implementationsLocalSandbox (host child_process, isolation: 'none' by default — no OS sandbox), AgentCoreRuntimeSandbox (AWS Bedrock AgentCore; ships in @mastra/agentcore, one-shot only, no background processes, no persistent state, no filesystem mounts), plus cloud/OCI: E2BSandbox, ModalSandbox, DaytonaSandbox, BlaxelSandbox, RailwaySandbox, AppleContainerSandbox, VercelSandbox.LocalSandbox 1.1.0; others per provider.Every one either runs on the host (LocalSandbox — rejected for prod) or leaves the homelab for a cloud (E2B/Modal/etc. — new dep + API key + egress). None is homelab-native.
Background Tasks (docs/long-running-agents/background-tasks.md)Lets an agent dispatch a slow tool call without blocking the agentic loop; built on the same dountil under the hood.Added 1.29.0.Agent-tool-loop-coupled, not a detached workflow executor. Already evaluated + rejected for the harness (HARNESS.md §2).
Is there a Mastra built-in for “run + evaluate ARBITRARY code in an isolated sandbox, driven from a workflow step (not attached to an agent)”? No single named/branded feature. But the primitives fully support it: you can instantiate a Workspace/sandbox standalone and call sandbox.executeCommand(...) directly from any code (including a workflow step) without attaching it to an agent. A custom WorkspaceSandbox called directly from a workflow step is a supported, first-class pattern — which is exactly what we already do. The one honest gap: the docs never show a worked “sandbox inside a workflow step” recipe; we’re composing primitives, not using a labeled feature.

2.2 What we already own — K8sExecSandbox (this is most of the answer)

images/mastra/src/mastra/durable/executor.ts is a working, isolated, arbitrary-code-execution sandbox that already implements Mastra’s WorkspaceSandbox interface. It is not aspirational — it is shipped and live on main (REWRITE-NOTES turns 107–108: “the durable-build harness is now genuinely LOAD-BEARING”). What it does (mechanism):
  • start() — creates a bare Pod (not a Job) in the agent namespace, pinned to i5 (kind: compute), image DURABLE_EXECUTOR_IMAGE (default node:22; has node/npm/tar/git, pnpm via corepack). sleep infinity, waits for Ready.
  • writeFiles() — tars files and streams them over exec … tar -xmf - stdin into /work. No shell; fixed argv.
  • executeCommand(command, args, opts) — execs the argv in the pod, captures stdout/stderr/exit. Fixed argv, no string concatenation.
  • readFile() / listFiles()cat / find a file back out.
  • destroy() — deletes the pod (best-effort; activeDeadlineSeconds self-reaps a leak).
Isolation (verified in cluster/apps/agent/mastra/rbac.yaml):
  • The pod runs as durable-build-executor SA, which has zero role bindings and automountServiceAccountToken: false (the SA manifest also pins it off) — generated code has no k8s API access at all, cannot touch the control plane, cannot read a secret.
  • The pod is cap-dropped (drop: ALL), runAsNonRoot (uid 1000), seccompProfile: RuntimeDefault, resource-capped (2 CPU / 2Gi RAM / 2Gi emptyDir), restartPolicy: Never, terminationGracePeriodSeconds: 0, activeDeadlineSeconds hard wall (default 3h).
  • mastra itself only holds the namespace-scoped mastra-durable-build Role (create/get/list/delete pods, pods/status, pods/exec create+get, pods/log — agent ns only). No cross-ns, no secrets, no nodes.
Shell-free held (verified): the sandbox is NEVER attached to an Agent — it is driven only from workflow steps. The main agent’s tool surface (main.ts) sees only run_durable_build / durable_status; it never gets a command string into the pod. The catch — it is web-app-specific today. The generic primitive (create → inject → exec → read → teardown) is fully general, but the caller around it (the durable-build harness) is hardwired for web apps:
  • workflow.ts step generate-and-evaluate always runs evaluateLiveApp(ws, kind, selectors), which serves the app on :3000 and drives browserless.
  • The hard gate at workflow.ts:315: passed = verdict.passed && live.started && live.consoleErrors.length === 0. live.started requires an HTTP server to have come up. A Rust CLI or a typecheck-only change can never set started: true, so it can never pass — the harness is web-only in practice.
  • Evaluation evidence is DOM/console/selectors — meaningless for non-web code.
So: we own a general sandbox wearing a web-app-shaped harness. The realization to act on is that the sandbox is the reusable asset; the web-app judging is one policy on top of it.

2.3 The write-wall reality

  • images/mastra/src/mastra/tools/git-commit.ts:19const WRITE_ROOTS = ["cluster/", "apps-src/"]. Anything outside these roots is refused. This walls the agent out of images/ (its own source + all custom-image source).
  • Write path: lib/github.ts commitFile / commitFiles — GitHub Contents/Data API with a fine-grained PAT scoped to only this repo, committing straight to main. No SSH, no local clone, no node shell. Blast radius is one repo.
  • git-commit runs a structural gate before push (validate()): parses YAML for cluster//*.yaml, rejects unencrypted *.sops.yaml. Web assets under apps-src/ skip it. There is no code-level gate today (there was never code in the write roots).
  • self-edit.ts is a separate narrow write path: it writes the agent’s own prompt/config to the PVC (live in seconds) and mirrors to images/mastra/runtime/{prompts,config} in git. It proves the agent can already version-control changes to itself — but only its prompt/config, never its code. REWRITE-NOTES turn 105: “self-modification is DOUBLY blocked: can’t read its own code, can’t write it.”
  • images/ is bigger than assumed. It holds 9 custom images, not 4: mastra, console, feed-scraper, cluster-backup, riverbot, senses, kiosk, stt, tts. Any WRITE_ROOTS widening to “all of images/” covers all nine.
  • Self-deploy path today: edit images/<name>/ → CI (.github/workflows/images.yml, a matrix over all nine, builds on push to images/**) → new ghcr.io/amarcin/<name> image → bump the image: tag in cluster/apps/<...>/deployment.yaml → Flux reconciles. The tag bump lives in cluster/already inside WRITE_ROOTS, so the agent can already trigger its own redeploy; it just can’t change the code that gets built.
  • mastra’s own deployment is replicas: 1 + strategy: { type: Recreate } (libsql single-writer). Consequence: a bad mastra image doesn’t roll gracefully — Recreate kills the old pod, then the new one crashloops with nothing serving. The agent editing its own running code can ship a crashlooping image and cannot recover a pod it just killed. This is the core danger the verification loop must defend against.

2.4 Addressing the prior SKIP verdicts (REWRITE-NOTES turn 102/103, verbatim)

The prior design doc recorded these SKIPs:
  • “Code mode: SKIP — reintroduces a host exec surface (violates shell-free) and solves a fan-out problem this single-user agent doesn’t have.”
  • “Mastra Workspaces family: keep Skills + BM25; SKIP filesystem / sandbox / LSP; Search-vector = SKIP.”
Does that reasoning still hold now that the goal is general code-eval + safe self-modification?
  • Code mode: STILL SKIP, and the reasoning is unchanged. Code mode as a Mastra agent feature means attaching execute_typescript to the main agent — an exec surface on the agent, exactly what P3 forbids. And its purpose (orchestrating the agent’s own tools in generated code) is a fan-out problem we still don’t have. We are NOT reversing this. Note the distinction that matters: our capability runs arbitrary code in an isolated pod driven from a workflow, which is the turn-105 carve-out — categorically different from bolting a code-execution tool onto the agent.
  • “Workspace sandbox = SKIP”: ALREADY REVERSED, and correctly. Turn 102 said “SKIP … sandbox”; turn 105 then built K8sExecSandbox — but reconciled it by NOT attaching a Mastra Workspace sandbox to the agent. The turn-102 SKIP was about attaching Mastra’s sandbox to the agent (which auto-injects execute_command). The thing we built and now propose to generalize is a workflow-internal sandbox. These look contradictory on the surface; the turn-105 reconciliation (“shell-free binds the main agent’s tool surface, not a workflow step running generated code in isolation”) is exactly what separates them. This plan is consistent with the reversal, not a new one.
  • Workspace Filesystem / LSP: STILL SKIP. We don’t need Mastra’s WorkspaceFilesystem (we have readFile/listFiles/tar-inject on the sandbox) or LSP (typecheck via tsc in the pod gives the same signal without a language-server dependency). Adding them buys nothing for this goal and costs surface area (P4).
Net: no prior SKIP is being reversed by this plan. Code mode stays skipped. The sandbox “reversal” already happened at turn 105 and this plan builds on it. We are generalizing what we own, not adopting a new Mastra feature.

3. Options for the broad code-eval capability

Option A — Generalize the sandbox we already own

Extract the general “create isolated pod → inject files → run commands → read results → teardown” loop from under the web-app harness into a broad, kind-agnostic “execute + evaluate arbitrary code” capability. Web-app-build becomes ONE caller (it keeps the browserless live-app judge as a web-specific evaluation policy); non-web code (Rust CLI, mastra typecheck+test, a fresh service) is judged by running its own build/test commands and reading exit codes + output — no HTTP server required.
  • Pros: Reuses a shipped, isolation-hardened, RBAC-verified, Mastra-conformant asset — the hardest 80% (the isolated executor + the exec plumbing + the teardown discipline) is already built and load-bearing. It’s the most homelab-native (k3s on i5, no cloud, no key, no egress). It directly satisfies P2 (any code) and P3 (workflow-internal, never agent-attached). It’s the highest capability-per-line (P4): we’re removing a hard gate and adding a kind-generic evaluation path, not writing a new engine. It honors P1 by producing a few broad verbs, not many narrow ones.
  • Cons: The current harness is opinionated around the planner→generator→evaluator, spec.md/findings.md, browserless flow. Generalizing it means carefully separating “run arbitrary code and report” from “build a whole app with a planner + rubric.” If done sloppily it becomes a leaky mega-workflow. The single long-lived pod is a single point of failure (HARNESS.md §8), 2Gi RAM may be tight for a real node_modules build, and the node/next serve path is less proven than the static one.

Option B — Adopt a Mastra built-in sandbox

Swap in E2BSandbox / ModalSandbox / DaytonaSandbox (cloud) or AgentCoreRuntimeSandbox (AWS), or LocalSandbox with isolation: 'bwrap'.
  • Pros: Less code we maintain for the executor itself; the provider handles pod/VM lifecycle. E2B/Modal are purpose-built for “run untrusted code.”
  • Cons: Every cloud option leaves the homelab — a new external dependency, an API key, and egress off the fleet for every eval. That contradicts the whole self-hosted posture and was explicitly rejected for the harness (executor.ts header). AgentCoreRuntimeSandbox is one-shot only (no background processes, no persistent state) — a poor fit for a multi-command build, and needs @mastra/agentcore + AWS creds. LocalSandbox runs on the mastra pod itself — the exact isolation failure P3/the mandate calls out (generated code as the agent’s own process on i5). None is homelab-native; all are strictly worse than what we already run. We’d be adding a dependency to replace a working, more-isolated component. Fails P4 hard.

Option C — Hybrid: our sandbox as the default, a pluggable backend for overflow

Keep K8sExecSandbox as the prod backend but formalize the WorkspaceBackend seam (already present in workspace.ts via setWorkspaceFactory) so an E2B/Modal backend is a one-line swap for a future heavy workload our 2Gi pod can’t hold.
  • Pros: All of Option A’s benefits, plus a documented escape hatch if a build genuinely outgrows the homelab pod.
  • Cons: The seam already exists and is dev/test-only today; “formalizing” it for prod overflow is speculative work for a capacity problem we don’t have yet (P4). Risks over-engineering.

4. RECOMMENDATION

Option A: generalize the sandbox we already own. Do NOT adopt a new Mastra feature. The honest answer is that we already own the load-bearing capability. K8sExecSandbox is an isolated, tokenless, Mastra-conformant, homelab-native arbitrary-code executor that is shipped and proven. Every Mastra built-in alternative is either less isolated (LocalSandbox on the mastra pod), off-fleet (E2B/Modal/Daytona/AgentCore), or crippled (AgentCore one-shot). The only thing standing between “we have a web-app builder” and “we have a broad code-eval capability” is that the harness around the sandbox hard-codes web-app judging. Fix that, and web-app-build becomes one caller of a general capability. Adopting a Mastra sandbox would mean adding a dependency to replace a better component — the opposite of P4. Keep the setWorkspaceFactory seam as-is (it already exists for dev/test); do NOT build Option C’s prod-overflow backend until a real workload needs it.

4.1 Shape of the broad capability

Introduce one broad workflow + one broad tool, sitting beside (not replacing) the existing durable-build. The design keeps the existing harness working and adds the general primitive underneath. The broad primitive: an “evaluate code” workflow. A kind-agnostic workflow that:
  1. Takes: a set of files (or a git ref / repo path to fetch), an optional setup step, and a verification command list (e.g. ["pnpm install", "pnpm typecheck", "pnpm test"] for a TS package; ["cargo build", "cargo test"] for Rust; ["ruff check", "pytest"] for Python). The caller declares what “verified” means as commands + expected exit 0.
  2. Spins up a K8sExecSandbox (the existing executor, unchanged), injects the files, runs each verification command in order via sandbox.executeCommand(argv), captures exit codes + stdout/stderr tails.
  3. Returns a structured VerificationReport: per-command exit code + output tail, an overall passed (all commands exit 0), and the final file snapshot. No HTTP-server requirement. The web-app live-app judge becomes an optional evaluation policy the caller can request, not a mandatory gate.
Web-app-build becomes one caller. The existing planner→generator→evaluator harness keeps its browserless live-app judge — but expressed as one evaluation policy (kind: web-app → run evaluateLiveApp and apply the console-error gate) layered on the general verify step, rather than as an unconditional gate in the workflow. Refactor the workflow.ts:315 hard gate so live.started/consoleErrors only apply when the caller asked for the web-app policy. For a non-web kind, “passed” comes from the verification-command exit codes instead. Non-web code, concretely:
  • A fresh Rust CLI in a scratch dir: inject the crate, verification commands ["cargo build --release", "cargo test"]. passed = both exit 0. No server, no browser.
  • A mastra typecheck+test change: inject the changed images/mastra/** files (plus enough of the package to build — see §5 phasing / open questions), verification ["pnpm install --frozen-lockfile", "pnpm typecheck", "pnpm test"]. This is the exact loop that gates a self-modification (§5).
  • Someone else’s repo: fetch by git ref into the pod, run its declared build/test. Purely evaluate; nothing gets committed.
How it stays shell-free at the main agent (P3): identical to today. The workflow instantiates and drives the sandbox from inside its steps; the sandbox is never attached to an Agent, so execute_command is never injected. The main agent gains at most one broad toolevaluate_code (kick a verification run; poll status) — mirroring run_durable_build/durable_status. That tool takes files + a command list and returns a report; the model never gets a shell handle into the pod. The verification commands are fixed argv chosen by the caller of the workflow, executed inside the isolated pod, exactly as the generator’s run action works today. On P1 (broad, few tools): the main-agent tool surface grows by ONE broad tool (evaluate_code, plus its status reader), not a family of run_rust / run_python / typecheck_mastra tools. The kind-specific behavior lives in the command list the caller passes and in optional evaluation policies inside the workflow — not in the tool count. This is the “large module usable for anything” shape.

4.2 Why one workflow, not two separate ones

Durable-build (planner→generator→evaluator, builds an app from a prompt) and evaluate-code (verify a given set of files) are genuinely different jobs. But they share the sandbox and the “run commands, collect a report” core. The clean decomposition:
  • Layer 0 (unchanged): K8sExecSandbox — the isolated executor.
  • Layer 1 (new, broad): a verify(files, commands, policy?) capability — inject, run commands, optional policy (web-app live check), return a report. This is the general “execute + evaluate arbitrary code” module.
  • Layer 2 (existing, refactored to sit on Layer 1): durable-build — the planner/generator/evaluator loop, whose evaluation step now calls Layer 1 with the web-app policy.
Two callers (durable-build; the new evaluate_code tool), one broad primitive underneath. That’s the P1/P4 sweet spot: one new broad module, the app-builder becomes a consumer of it.

5. The write-wall takedown, coupled to the verification loop

The write-wall comes down in a way that makes an unverified code change structurally impossible to commit. The coupling is the point: WRITE_ROOTS widening is worthless (dangerous, even) without a mandatory verification gate in front of it.

5.1 What gets verified, where, before a code commit lands

A code commit (a write whose path is under images/**) must pass Layer-1 verification before commitFile is called. Concretely, the widened write path for code:
  1. The agent proposes a change to images/<name>/** (one or more files).
  2. The write tool assembles the buildable unit — the changed files plus enough of the package to build/test (for mastra: the images/mastra/ package; §7 open question on how much context to inject).
  3. It runs Layer-1 verify() in an isolated pod with the package’s declared verification commands — for mastra that is at minimum pnpm install --frozen-lockfile && pnpm typecheck && pnpm test (the test/typecheck scripts already exist in package.json). For other images, the per-image command list (e.g. a Dockerfile lint + the image’s own tests) — declared once per image in a small config, not inferred.
  4. Only if verify() passes does the tool call commitFile/commitFiles to land the change in images/**. A failing verification returns the report to the agent and commits nothing.
This is offline validation before the commit — the same discipline git-commit’s YAML gate already applies to manifests, extended to code via the sandbox. The verification runs in the isolated pod (P3 held), never on the mastra process.

5.2 The WRITE_ROOTS change

  • Widen WRITE_ROOTS from ["cluster/", "apps-src/"] to include all of images/ (covers all nine custom images). Proposed: ["cluster/", "apps-src/", "images/"].
  • But gate by path class, not one flat allow. Split the write path in git-commit.ts (or a sibling tool) into:
    • Manifest/asset writes (cluster/, apps-src/) — unchanged: structural YAML/SOPS gate, then commit. No code verification (they aren’t code).
    • Code writes (images/**) — must go through Layer-1 verification first (§5.1). A raw commitFile to images/** that bypasses verification is refused.
  • Keep self-edit.ts as-is for prompt/config — it’s a fast, non-code runtime path and shouldn’t be forced through a container-build verification.
Distinguishing the two powers (per P2 + the prompt): “evaluate arbitrary code” (Layer 1, evaluate_code tool) needs NO write and NO WRITE_ROOTS change — it runs anywhere on any code. “commit a verified change to a repo path” (the widened code-write path) needs WRITE_ROOTS to include images/ AND the verification gate. Both are in scope; they’re separate tools with separate blast radii.

5.3 Self-deploy safety

Self-deploy stays through git → CI → Flux — never the agent’s own kubectl apply of its own new image (AGENTS.md rule). The agent already can bump its own image: tag (it’s in cluster/). The dangers and backstops:
  • Crashloop risk (the sharp edge): mastra is replicas: 1 + Recreate. A bad mastra image, once the tag is bumped and Flux reconciles, kills the old pod and crashloops the new one — the agent cannot recover a pod it just killed. Backstops:
    1. Verification gate (§5.1)pnpm typecheck && pnpm test in the sandbox catches the large class of “it doesn’t even build/compile” and “tests fail” before the code is committed, so a broken build never reaches CI.
    2. The commit is revertable — every code change is a git commit on main. If a crashloop slips past verification (e.g. a runtime-only failure tests didn’t catch), the revert path is: revert the image: tag bump in cluster/apps/agent/mastra/deployment.yaml to the last-good SHA → Flux rolls back → the previous known-good image runs. Because the previous image is already built and in GHCR, the rollback is a pure tag change, fast.
    3. Human-in-the-loop for the agent’s OWN image (recommendation, open decision §7): a self-modification of images/mastra that requires a redeploy of the agent that is doing the modifying is uniquely risky — the agent can brick itself mid-operation. Consider requiring that the tag bump for mastra specifically (not the code commit, not other images) either (a) needs an explicit confirm gate, or (b) is left for Augustin to trigger, while other images (riverbot, feed-scraper, etc.) can self-deploy fully. The code commit + verification can be fully autonomous; the “redeploy the brain” step is where a human backstop earns its keep.
  • CI as a second gate: CI rebuilds the image on push to images/**. If the Dockerfile build itself fails, no new image is produced and the old tag keeps running — a natural backstop. (Verification in §5.1 runs the package’s tests, not the full Docker build; the Docker build is CI’s job and is the belt to verification’s suspenders.)
  • Offline manifest validation (AGENTS.md) still applies to any cluster/ change the agent makes — one bad manifest wedges all of Flux.

6. Where the principles create tension (called out honestly)

  • P1 (broad) vs P4 (less code): a truly broad “evaluate any code” module wants to handle many languages/build systems. The temptation is a big framework that knows about npm/cargo/pip/go. The resolution that keeps both: the module doesn’t know languages — the caller passes the command list. verify(files, ["cargo test"]) and verify(files, ["pnpm test"]) are the same module. Broad because it’s dumb about specifics. If we start baking per-language logic into the module, we’ve violated P4; push that into caller-supplied config.
  • P2 (general) vs P3 (isolation): general code-eval means running genuinely arbitrary, untrusted code. P3’s isolation (zero-token pod, no cluster API, cap-dropped, resource-capped, deadline-bounded) is what makes “run anything” safe. These reinforce rather than fight — but only as long as the pod stays zero-privilege. The risk is scope creep: someone later wants the eval pod to reach a real database or the cluster API “just for this build.” That must be refused — the moment the eval pod gains privilege, P2 (run anything) becomes a liability. Keep the executor SA at zero bindings, always.
  • P3 (shell-free) vs the goal (self-modification): self-modification feels like it needs a shell. It doesn’t. The write is a GitHub API call (commitFile), the verification is workflow-internal sandbox exec, the deploy is a git tag bump + Flux. At no point does the main agent get execute_command. The turn-105 carve-out is what makes this legal, and this plan stays strictly inside it.
  • Broad tool vs blast radius: one broad evaluate_code tool that runs arbitrary commands is powerful. The blast radius is bounded entirely by the isolated pod, not by the tool’s breadth — which is why the isolation (not tool-narrowing) is the safety mechanism. This is the correct P1-compatible way to be safe: broad tool, tight sandbox.

7. Risks, open questions, and decisions needed from Augustin

Open questions (need a decision before building):
  1. How much context to inject for a mastra self-verification? A pnpm install --frozen-lockfile && pnpm typecheck && pnpm test needs the whole images/mastra/ package (not just the changed file) plus the lockfile in the pod, and 2Gi RAM may be tight for node_modules + tsc + vitest (HARNESS.md §8). Options: (a) inject the full package each time (simple, heavy); (b) fetch the package from git at HEAD into the pod and overlay only the changed files (lighter injection, needs git-in-pod fetch, which the pod supports). Decision needed: which, and whether to bump the eval pod’s RAM for code-verification runs.
  2. Autonomy level for redeploying the agent’s own image. Fully autonomous self-deploy of images/mastra (verification passes → commit → tag bump → Flux → agent restarts on the new image) vs. requiring a confirm / human trigger for the mastra-specific tag bump (§5.3). Other images could be fully autonomous either way. This is the sharpest decision — it’s the difference between “the agent can brick its own brain” and “the agent proposes, Augustin ships the brain.”
  3. Where verification commands per image are declared. A small per-image config (e.g. images/<name>/verify.json listing the command list) vs. inferring from package.json/Cargo.toml. Explicit config is more predictable (and self-editable), inference is less setup. Recommend explicit config (predictable, versioned, honors “the caller declares what verified means”).
  4. Does evaluate_code need to fetch remote repos? P2 says “someone else’s repo.” Fetching an arbitrary git URL into the pod is easy (pod has git), but it means the eval pod makes outbound network calls. Confirm that’s acceptable (the pod is isolated from the cluster, but it does have egress for pnpm install/cargo fetch today). Decision: is arbitrary outbound fetch in the eval pod OK, or should remote-repo eval be gated?
Risks:
  • Verification is not a proof of correctness. typecheck + test catches compile errors and test regressions, not every runtime failure. A self-modification with passing tests can still crashloop (missing env var, bad startup ordering). The revert path (§5.3) is the mitigation; verification lowers but doesn’t eliminate the crashloop risk. Be honest that “verified” means “builds + tests pass,” not “cannot break.”
  • Single long-lived pod SPOF + sizing (HARNESS.md §8) — inherited from the existing sandbox; a code-verification run that OOMs the pod fails the verify (safe — it refuses to commit), but a flaky OOM could block legitimate self-improvement. Size the pod for the mastra build.
  • The node/next serve path is less proven than static (HARNESS.md §6, §8) — only relevant to the web-app policy caller, not to non-web verification.
  • Refactoring the workflow.ts:315 gate risks regressing the working web-app harness. Keep the web-app policy behavior byte-identical; only change when it applies.

8. Rough build sequence (phased, delegable)

Phase 1 — Extract Layer 1 (the broad verify primitive). New module (e.g. durable/verify.ts): verify(files, commands, policy?) that spins up K8sExecSandbox (unchanged), injects, runs the command list, returns a VerificationReport. Pure extraction of the run-commands-collect-report core already present in workflow.ts/workspace.ts; no new isolation work. Unit-test with the existing local backend (setWorkspaceFactory), like harness-e2e.test.ts. Phase 2 — Refactor durable-build to sit on Layer 1. Make evaluateLiveApp + the :315 hard gate a web-app policy invoked by Layer 1 only when the caller asks; non-web kinds pass/fail on command exit codes. Keep the web-app path behavior-identical (regression-guard with the existing e2e test). No change to the executor or RBAC. Phase 3 — The broad evaluate_code tool. One broad main-agent tool (+ a status reader) over Layer 1: takes files/ref + command list, returns the report. Mirrors run_durable_build/durable_status wiring in main.ts + index.ts. Satisfies P2 (evaluate any code) with zero WRITE_ROOTS change. This is a complete, shippable capability on its own — the agent can now verify arbitrary code before Augustin decides on the write-wall. Phase 4 — The verified code-write path. Split git-commit.ts write handling by path class: widen WRITE_ROOTS to include images/, but route images/** writes through a mandatory Layer-1 verification (per-image command list from §7 Q3) before commitFile. Manifest/asset writes unchanged. Resolve open questions §7 Q1/Q3 before this phase. Phase 5 — Self-deploy safety wiring. Implement the §5.3 decision for the mastra-specific tag bump (confirm gate or human trigger, per §7 Q2). Document the revert-to-last-good-SHA runbook. Other images can self-deploy per the chosen autonomy level. Phases 1–3 deliver the broad code-eval capability (P1/P2/P3/P4 satisfied) with no write-wall change and are independently shippable. Phases 4–5 take the wall down safely, gated on Augustin’s §7 decisions. Phase 6 — Concurrent workstreams + conflict resolution (separate doc: concurrent-workstreams-design.md). Once multiple self-editing workstreams run at once, their branches must merge safely: branch-per-workstream isolation (a one-parameter widening of lib/github.ts, folds into Phase 4), a merge-coordinator thread driving a durable merge+re-verify(this doc’s verify())+deploy-once workflow, and a conversational LLM resolver (a spawned federation thread, on beta Signals behind lib/autonomy.ts) that fires only for a true git conflict or a clean-merge-that-fails-verify. The re-verify coupling is why a git-clean merge of two images/mastra edits can’t ship a broken brain. See that doc for the full flow, the self-image special case, beta-primitive risk, and its own §10 open decisions.

9. One-paragraph summary

We already own the hard part: K8sExecSandbox is an isolated, tokenless, homelab-native, Mastra-conformant arbitrary-code executor, shipped and load-bearing — wearing a web-app-shaped harness. The right move is to generalize what we own, not adopt a Mastra built-in (every built-in is less isolated, off-fleet, or crippled). Extract a broad verify(files, commands, policy?) primitive; web-app-build becomes one caller; add a single broad evaluate_code tool (P1) that runs any code (P2) in the isolated pod driven from a workflow (P3) with minimal new code (P4). Then take the write-wall down by widening WRITE_ROOTS to images/ only behind a mandatory sandbox verification (typecheck + test) that runs before any code commit, with self-deploy staying through git→CI→Flux and the Recreate rollout’s revert-to-last-good-SHA as the crashloop backstop. Code mode stays SKIPPED (it’s an agent exec surface — P3 violation); the sandbox “reversal” already happened at turn 105 and this plan builds on it. The one decision that genuinely needs Augustin: whether the agent may autonomously redeploy its own brain (images/mastra) after verification, or whether that specific tag bump stays human-triggered.