> ## Documentation Index > Fetch the complete documentation index at: https://docs.augustin.ai/llms.txt > Use this file to discover all available pages before exploring further. # Concurrent workstreams design # Concurrent Self-Editing Workstreams + Conflict Resolution **Status:** PLAN / design doc for review. No code changed, nothing deployed. **Author:** architecture pass (Opus 4.8), 2026-07-06. **Extends:** [`self-capable-arc-design.md`](./self-capable-arc-design.md) (commit `b51e05e8`) — this is the concurrency layer over the self-capable arc. Read that first: the `verify()` Layer-1 primitive, `evaluate_code`, `WRITE_ROOTS`, and the fully-autonomous self-deploy decision all live there. This doc slots into its Phases 4–5 and adds Phase 6. **Scope:** the agent is becoming self-editing AND self-deploying (fully autonomous, all 9 images incl. its own brain `images/mastra` — REWRITE-NOTES turn 109). Augustin runs MULTIPLE workstreams at once. This doc designs what happens when two self-editing workstreams collide: **branch-per-workstream isolation, git-does-the-mechanical-merge, and an LLM resolver thread that re-verifies the merge and CONVERSES with the originating workstreams to preserve intent.** *** ## 0. TL;DR (the recommended architecture) * **A workstream = a registered federation thread + a git branch + (when it needs to build/verify) a `K8sExecSandbox` worktree.** No new registry — extend the turn-86 thread registry (`metadata.kind: "workstream"`) that `register-thread`/`list-threads`/`tell-thread` already ride on. * **Git-layer isolation comes first.** Today `git-commit.ts` commits straight to `main` — there are no "two workstreams", just a push race. The fix is **branch-per-workstream**: each self-editing thread commits to its own branch via the GitHub Data API; `main` is only ever reached by an auto-merge on workstream-done. * **Git does \~95% of the merge.** On workstream-done, a **merge attempt** (GitHub merge API / three-way merge) resolves every mechanically non-conflicting change. The LLM resolver spawns **only** for the true conflict git can't auto-merge. Do NOT build a merge engine — git already is one. The resolver is for **intent**, not diff3. * **Merge → re-verify → deploy-once.** A clean git merge is NOT a green light. Two workstreams can merge cleanly at the text layer and produce a broken brain. So every merge (auto OR resolver-produced) runs through `verify()` (self-capable arc Layer 1); for a deployable image the unit is **merged source → rebuild → re-verify → ONE deploy.** Resolution is authoritative: other in-flight workstreams rebase onto the resolved `main`. * **The resolver CONVERSES, on beta Signals, now.** Per Augustin's redirect ("we are bleeding edge… we will adapt as needed"), the resolver does not merely notify. It reaches BACK to the originating workstreams — "does this merge preserve your intent? I need you to move your helper into `lib/`" — and gets a reply. This is built on the **beta** `subscribeToThread` / `sendMessage.accepted.output` reply-correlation + `sendStateSignal` shared-state lanes, **all isolated behind `lib/autonomy.ts`** (the existing anti-corruption boundary). Beta risk is real and called out in §7; the stance is "isolated, adapt on churn," not "wait for GA." * **The self-image special case is the sharp edge.** Two workstreams both editing `images/mastra` → you cannot "merge two image tags." You merge SOURCE, rebuild, re-verify, deploy ONCE. And the brain is `replicas:1 + Recreate`, so a bad self-image can't roll gracefully — the resolver's re-verify is the load-bearing backstop, revert-to-last-good-SHA is the parachute. *** ## 1. Grounding: what this system actually is (so we design onto it, not beside it) Five facts from the codebase constrain every choice below. Cited so the design stays honest. 1. **Federation model (LOCKED turn 76, LIVE turn 86).** "ONE broad agent, MANY threads, ONE shared memory; coordinate via SIGNALS; no hierarchy." A new automation is a new *thread* (data), not a new *class* (code) — "runtime spawning is free" (REWRITE-NOTES:106). The mesh today: `registerThread`/`listThreads`/`tellThread` in `lib/autonomy.ts`, exposed as `register-thread`/`list-threads`/`tell-thread` tools (`tools/autonomy.ts`). **A conflict-resolver is therefore just another thread with a goal.** Reaching back to a workstream is `sendMessage`/`subscribeToThread` through that same boundary. 2. **`tellThread` is fire-and-wake today, and it DELIBERATELY drops the reply.** `lib/autonomy.ts:276` awaits `result.accepted` and returns only `accepted.action` — it explicitly does NOT await `accepted.output` ("that would block on the peer finishing"). But `accepted.output` **exists**: it's a `MastraModelOutput` promise on the `wake`/`deliver` members of the union (verified `node_modules/@mastra/core/dist/agent/types.d.ts:120,137`). **So the conversational round-trip is not a different API — it is awaiting the field the current code drops.** That is the single most important beta finding in this doc. 3. **The write path commits straight to `main`, one repo, via the GitHub API — no branch, no clone, no shell.** `git-commit.ts:19` `WRITE_ROOTS = ["cluster/", "apps-src/"]`; `lib/github.ts` `commitFile` (Contents API) and `commitFiles` (Data API: blobs→tree→commit→ref) both hardcode `branch: BRANCH` (=`main`) and PATCH `refs/heads/main` with `force:false`. **There is no branching machinery yet — but the Data API `commitFiles` already builds a commit object against a base tree, which is 90% of what branch-per-workstream needs.** A `force:false` ref PATCH is exactly the push-race today: whichever workstream PATCHes second gets a 422 (non-fast-forward). That is the "collision" in the current system — a lost write, not a merge conflict. 4. **The isolated executor is a real `WorkspaceSandbox`, tokenless, and the self-capable arc's `verify()` generalizes it.** `durable/executor.ts` `K8sExecSandbox`: bare pod on i5, zero-privilege SA (`automountServiceAccountToken:false`, zero role bindings), cap-dropped, `activeDeadlineSeconds`. It has **git inside it** — `durable/workspace.ts:95` runs `git init`, and `gitCommitAll`/`gitLog` drive git in-pod. **This is where a three-way merge and a rebuild physically happen.** The pod can `git fetch`, `git merge`, `git rebase`, `git worktree` — git is a binary in the image, driven by fixed argv, never a model shell (P3 held: the sandbox is workflow-internal, never agent-attached). 5. **Self-deploy is fully autonomous through git→CI→Flux, and the brain can't roll gracefully.** REWRITE-NOTES turn 109: YES, the agent may bump its own tag + redeploy all 9 images incl. `images/mastra`. AGENTS.md: self-deploy goes through git, never the agent's own `kubectl apply`. `images/mastra` is `replicas:1 + Recreate` (libsql single-writer), so a bad self-image kills the old pod then crashloops the new one with nothing serving. Backstop = revert-tag-to-last-good-SHA. *** ## 2. What is a "workstream", concretely A **workstream** is the unit that can collide. It is three things, bound by one id: | Facet | What it is | Where it lives | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **A thread** | A registered federation thread, the workstream's identity + conversation + goal. Peers address it by name. | The turn-86 registry — extend `ThreadKind` with `"workstream"` (`lib/autonomy.ts:60`). `listThreads` already filters by `metadata.kind`, so workstreams show up in `list-threads` for free. | | **A branch** | `ws/` in the repo. ALL of the workstream's commits go here, never to `main`. Derived deterministically from the thread name (same slug scheme as `stableThreadId`). | GitHub, via a branch-aware widening of `lib/github.ts` (§3.1). | | **A worktree** | ONLY when the workstream needs to build/verify/merge: a `K8sExecSandbox` pod with the repo cloned and `git worktree`/branch checked out. Ephemeral — created for a verify or a merge, torn down after. | `durable/executor.ts`, driven from a workflow step. | **Registration + tracking.** `registerWorkstream(name, {targetImages?})` = `registerThread({name, kind:"workstream"})` + record the branch name and, in thread metadata, the **declared write scope** (which `images//**` paths this workstream intends to touch). The write scope is what lets the merge trigger cheaply detect *potential* overlap before doing any git work (two workstreams that touch disjoint images can never semantically conflict even if git flags a textual one in a shared file — and two touching the same image are the ones to watch). Metadata, not a new table — it rides `StorageThreadType.metadata`, same as `kind`/`displayName` today. **"Done, ready to merge" signal.** A workstream, when its own `verify()` passes on its branch, calls **`tellThread({targetName: "merge-coordinator", from: "", text: "ready: branch ws/ @ , verified"})`** — fire-and-wake into the coordinator (§4). It does NOT merge itself. This keeps the merge decision in one place (no two workstreams racing to merge each other). > **Design note — why extend the registry, not build a new one.** The brief's non-negotiable #1 is "map onto Mastra's existing federation, not new machinery." A workstream registry that isn't the thread registry would be a second source of truth that drifts. `metadata.kind` + a branch string + a write-scope array is the whole model. This is the "runtime spawning is free" property (REWRITE-NOTES:106) applied to workstreams: a new workstream is a new thread + a new branch, no rebuild. *** ## 3. Git-layer isolation FIRST (the part the agent doesn't have yet) Today there is no "two workstreams." There's one write path to `main` and a push race (fact #3). Before any resolver makes sense, the write layer has to produce *branches that can diverge and be merged*. This is the foundation; the resolver is worthless without it. ### 3.1 Branch-per-workstream write path Widen `lib/github.ts` so its commit functions take an optional `branch` (defaulting to `main`, so every existing caller is byte-identical): * `commitFiles(files, message, branch = BRANCH)` — the Data-API path already builds `blobs→tree→commit→ref`; the only change is which ref it reads as the base (`git/ref/heads/`) and PATCHes at the end. Creating the branch is one extra call (`POST git/refs` with `refs/heads/ws/` pointing at `main`'s head) done idempotently on first write. * A workstream's code writes go to `ws/`; **`main` is never touched by a workstream directly** — only by the merge step (§4). `cluster/` manifest writes and `apps-src/` writes stay on `main` unchanged (they're not code, they're not part of a self-editing workstream; keeping them on `main` avoids a merge-queue for ops changes that were never colliding). This is deliberately small: it's a parameter, not a rewrite. The GitHub Data API is already a git server; we're just stopping the agent from all committing to the same branch. ### 3.2 Where the merge and rebuild physically happen Git's three-way merge runs **inside the executor pod** (fact #4), not against the GitHub API. The GitHub "merge" REST endpoint can do a server-side merge but gives you no place to *rebuild and re-verify* the result before it lands — and re-verify is non-negotiable (§5). So the merge sequence is: 1. Executor pod clones the repo (or fetches; the pod has git + egress), checks out `main`. 2. `git merge ws/` (or merges multiple ready branches in turn). 3. **Clean merge** → git exits 0, working tree has the merged source → go to re-verify (§5). 4. **Conflict** → git exits non-zero, leaves conflict markers / `git status` shows unmerged paths → **this is the ONLY trigger for the LLM resolver** (§6). Git has already resolved everything it can; what's left is genuine overlap. The pod is the merge engine's host; git is the merge engine. We add zero merge logic. *** ## 4. The merge trigger + who runs it **Three genuine options for "who attempts the merge."** This is a real fork, not a foregone conclusion. | Option | Shape | Pros | Cons | | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **A. Merge-on-done heartbeat** | A watcher (`spawnWatcher`, cron e.g. `*/2 * * * *`) that scans for ready workstreams and merges them. | Uses shipped machinery (heartbeats are LIVE). No new workflow. Naturally serializes (one tick at a time) → no merge-of-merges race. | Polling latency (up to the cron interval). A heartbeat fire is an LLM run — cost per tick even when idle (mitigable: cheap "any ready?" check first, only escalate to a merge run if yes). | | **B. Merge-coordinator thread (event-driven)** | A single long-lived registered thread `merge-coordinator`; workstreams `tellThread` it on done; it merges reactively. | Event-driven (no poll latency). Single serialization point (one thread = one merge at a time by construction). Pure federation — the coordinator is just a thread with a goal, exactly non-negotiable #1. Converses natively (it's already a thread). | Relies on `tellThread` wake being reliable (it is, at replicas:1, single-process). If the coordinator thread is mid-run when a second "ready" arrives, it `deliver`s into the active run — must handle a queue of ready branches, not just one. | | **C. Merge as a durable workflow step** | Extend the durable-build workflow family: a "merge+verify+deploy" workflow kicked per ready-set. | Durable (survives restart at phase boundaries — the exact property turn 107 made load-bearing). Clean 3-phase decomposition (merge → verify → deploy) mirrors durable-build. | Heaviest. A workflow isn't a *thread*, so the "converse back to originating workstreams" step still has to hop into the federation mesh anyway — you get workflow durability but the conversation lives in threads regardless. Risks two coordination substrates. | **RECOMMENDATION: B (merge-coordinator thread) as the front door, invoking C (a durable merge+verify+deploy workflow) for the heavy tail, with the resolver as a spawned thread.** Concretely: * The **merge-coordinator** is a seeded, long-lived registered thread (like `concierge`/`self-healer`). Workstreams `tellThread` it when verified-and-ready. It holds a queue of ready branches in its thread state. * For each ready branch (or batch), the coordinator **kicks the durable merge+verify+deploy workflow** (option C, `run_durable_build`-style tool → `runDurableMerge`). The workflow is where the pod clone → `git merge` → `verify()` → rebuild → deploy happens, with durability so a mastra restart mid-merge resumes. * On a **clean merge that passes re-verify**, the workflow deploys once (§5) and reports back; the coordinator tells any *other* in-flight workstreams to rebase onto the new `main` (§6.4). * On a **git conflict**, the workflow does NOT try to be clever — it **spawns a resolver thread** (§6) with the conflict context and suspends that branch's merge until the resolver reports a resolution. The resolver is where the conversation happens. Why this hybrid and not pure-B or pure-C: the *coordination* (who's ready, who rebases, the conversation) is inherently thread-shaped — it's federation, it's non-negotiable #1, keep it in threads. The *mechanical merge + build + deploy* is inherently workflow-shaped — it's long, it must survive restart, it's the exact durable-execution case turn-23/107 reserved for workflows. Forcing all of it into a heartbeat (A) loses durability on the heavy leg; forcing all of it into a workflow (C) fights the federation model on the coordination leg. Split on that seam. **True-conflict detection is git's exit code, full stop.** No heuristic "will these conflict" pre-check decides resolution — the declared write-scope overlap (§2) is only a *cheap hint* to batch likely-related branches together; the *authority* on "is there a conflict" is `git merge` exiting non-zero. This keeps us out of the "build a bespoke merge engine" trap (non-negotiable #2). *** ## 5. Merge → re-verify → deploy-once (the coupling that must be explicit) **A clean git merge is a text fact, not a correctness fact.** Non-negotiable #3, made concrete: workstream A adds a new tool `foo` and registers it in `main.ts`'s tools object; workstream B adds a new tool `bar` and registers it in the same object. If they touched different lines, git merges cleanly. But maybe both added an MCP client keyed `search`, or both bumped `@mastra/core` to incompatible carets, or A's tool imports a helper B moved. **Textually clean, semantically broken.** At `images/mastra` that ships a crashlooping brain. So the merge output is never "commit the merged text." It is: ``` git merge (clean OR resolver-produced) → merged source tree in the pod → verify(mergedFiles, ) ← self-capable arc Layer 1 for a deployable image: the command list ENDS with a real build (pnpm install --frozen-lockfile && pnpm typecheck && pnpm test) → verify PASSES → commit merged tree to main (one commit) → bump image tag in cluster/ (one commit) ← triggers CI→Flux → verify FAILS → do NOT commit to main; hand the failure back to the resolver (or, if this was a clean auto-merge, spawn a resolver FOR the verify failure — a clean merge that fails verify is a semantic conflict git couldn't see) ``` **Merge → re-verify → deploy-once.** The rebuild + re-verify + deploy of a deployable image is **one unit**: you never deploy a merge you haven't re-verified as a whole. This closes the "A's tool + B's tool merge clean, brain breaks" hole that a git-only merge queue would leave wide open. **A clean merge that fails re-verify is itself a semantic conflict** — the resolver spawns for it too, even though git never flagged a hunk. This is the subtle case the brief's non-negotiable #3 is really about, and the design treats "git conflict" and "clean-but-fails-verify" as the *same* trigger for the resolver: both mean "human-level intent reconciliation needed." *** ## 6. The resolver thread — context, authority, and the conversation The resolver is **a spawned federation thread with a goal** (non-negotiable #1). It is not a workflow, not a hierarchy, not a privileged supervisor. It spawns for exactly one reason: a merge git couldn't auto-resolve, OR a clean merge that failed re-verify (§5). At homelab scale that is rare — most concurrent workstreams touch different images and merge clean — so the resolver is a cold path, and every spawn is an LLM session (cost honesty in §8). ### 6.1 What it's handed (its context) Spawned via a `spawnResolver(...)` helper (a `spawnWatcher` sibling in `lib/autonomy.ts`, but a one-shot goal thread not a cron), given: * **Both diffs** — `ws/A` vs merge-base, `ws/B` vs merge-base (the pod can `git diff` these). * **The conflict hunks** (or, for a clean-but-failed-verify, the `verify()` report — the failing command + stderr tail). * **Both workstreams' stated intent** — pulled from each workstream thread's goal/recent messages via the memory the mesh already shares (`searchHistory`/thread messages). This is *why* the workstream registry is a thread: the intent is already captured as conversation. * **The names of the originating workstream threads**, so it can address them. ### 6.2 How it produces a resolution The resolver works in an executor-pod worktree: it proposes a merged source (edit the conflict, or restructure to satisfy both intents), then **runs `verify()` on its proposal**. It iterates in-pod until verify passes or it gives up (bounded — §7). Its output is a resolved tree that has *already passed re-verify*, which then follows the §5 deploy-once path. Resolution is authoritative. ### 6.3 The conversation back to the workstreams (the full vision, on beta, NOW) Per Augustin's redirect, the resolver does not just notify — it **reaches BACK for modifications based on the overlap**. The mechanism, all behind `lib/autonomy.ts`: * **`askThread(target, question) → reply`** — a NEW anti-corruption-boundary function, the conversational sibling of `tellThread`. Where `tellThread` awaits only `accepted.action` and drops `accepted.output` (`lib/autonomy.ts:276`), `askThread` **awaits `accepted.output`** — the woken run's `MastraModelOutput` (verified present, types.d.ts:120,137) — and returns the peer's reply text. This is reply-correlation with **zero new Mastra surface**: it's the same `sendMessage` call, awaiting the field we currently throw away. * **`subscribeToThread(options)`** (agent.d.ts:1145) backs the durable/streamed variant when the resolver wants to *watch* a workstream's progress rather than block on one reply (e.g. "A, restructure your helper; I'll watch your branch and re-merge when you push"). Wrapped as `watchThread(...)` behind the boundary. * **`sendStateSignal(state, target)`** (agent.d.ts:1183, snapshot/delta modes) backs a **shared merge-state lane** — the resolver publishes "here's the merged plan and which hunks each of you owns" as state the workstreams read, instead of re-sending it in every message. Wrapped as `publishSharedState(...)`. So the loop the brief describes — *"resolver reaches back to the other agents for any needed modifications based on the overlap"* — is: ``` resolver detects overlap → askThread("A", "your helper foo() moved to lib/ in the merge; does your caller still assume it's local? reply with the corrected import or 'fine as-is'") → A wakes, answers (accepted.output) → resolver folds A's answer into the merged tree → verify() the folded tree → if A + B's constraints now both hold, deploy-once → if still broken, publishSharedState(mergedPlan) and watchThread(A), watchThread(B) while they push corrections onto their branches → re-merge → verify → deploy-once ``` ### 6.4 Authority + rebase The resolved `main` is authoritative. Any workstream still in flight (branched off the *old* `main`) is told to **rebase onto the resolved `main`** — `tellThread({target: "", text: "main advanced to via resolution R; rebase ws/ and re-verify before you signal ready"})`. The workstream rebases in its own pod, re-verifies its branch, and re-signals ready. Losing/other workstreams never silently keep a stale base. > **Why this is not a hierarchy.** The resolver has no standing authority over the workstreams — it's spawned per-conflict, converses as a peer, and dies when the conflict is resolved. "Authoritative" applies to the *resolved commit*, not the *thread*. This preserves non-negotiable #1's "no supervisor agent." *** ## 7. Beta-primitive dependency + honest risk (the "adapt as needed" stance) Augustin's redirect: we build on beta now, isolate it, adapt on churn. Here is exactly what's beta and what breaks if it moves — **all behind the single `lib/autonomy.ts` boundary**, so a churn is a one-file edit (the turn-75/77 beta-risk mitigation; Dapr-Agents is the named fallback substrate if Mastra Signals is abandoned). | Primitive | Used for | Beta status | What breaks if it changes | Blast radius | | ---------------------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `sendMessage(...).accepted.output` | `askThread` reply-correlation — the resolver's round-trip | `@experimental` (`agent/signals.d.ts`, used already by `tellThread`) | If the `SendAgentSignalAccepted` union drops/renames `output`, the round-trip loses its reply and `askThread` degrades to fire-and-wake (notify-only). The resolver still merges + re-verifies; it just can't *ask* — it falls back to resolving from the diffs + stated intent alone. | `lib/autonomy.ts` `askThread` only. `tellThread` already maps this union onto our own `{ok,action}` shape — we extend that mapping, one place. | | `subscribeToThread(options)` | `watchThread` — resolver watches a workstream's branch progress | beta (agent.d.ts:1145) | If subscription shape churns, `watchThread` breaks → resolver falls back to polling the branch head in its pod (`git ls-remote`) instead of a push subscription. Slower, not broken. | `lib/autonomy.ts` `watchThread` only. | | `sendStateSignal(state, target)` | `publishSharedState` — shared merge-plan lane | beta (agent.d.ts:1183; snapshot/delta) | If state-signal semantics churn, `publishSharedState` breaks → resolver re-sends the plan as plain messages via `askThread`/`tellThread` (more tokens, same outcome). | `lib/autonomy.ts` `publishSharedState` only. | | `heartbeats.create` | merge-coordinator seeding (if option A elements are used) | beta, already LIVE (self-healer, backup-watcher) | Already load-bearing; churn here already hits the whole autonomy layer, not new. | `lib/autonomy.ts` (existing). | **The honest engineering posture:** every conversational primitive has a *degrade path to a still-correct, less-conversational behavior* (notify-only / poll / re-send). The merge + re-verify + deploy-once core does NOT depend on any of them — it's git + `verify()` + `commitFiles`, all stable. **So even a total Signals-beta meltdown leaves us with a working "branch → auto-merge → re-verify → deploy, resolver resolves conflicts from diffs+intent and NOTIFIES workstreams to rebase" system.** The conversation is the enhancement that beta buys us now; its loss is a graceful degrade, not an outage. That is what "we will adapt as needed" means in code: the boundary is the adaptation point, and the fallbacks are pre-designed. **What is NOT deferred:** the conversational round-trip ships now, on beta. There is no "notify-now, converse-at-GA" split. The degrade paths above are failure handling, not a phase gate. *** ## 8. Failure / thrash modes and how the design bounds them | Mode | Scenario | Bound | | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Resolver deadlock** | Resolver `askThread`s A and B; both keep editing; resolver waits forever. | `askThread` awaits `accepted.output` with a **hard timeout** (mirror the 10s pattern in `reachOut`'s ntfy leg / the pod command timeout). On timeout the resolver proceeds from diffs+intent alone and `reachOut`s Augustin that it resolved without a reply. Never an unbounded wait. | | **Livelock (resolve → rebase → conflict → resolve…)** | Resolution advances `main`; in-flight workstreams rebase; a rebase creates a new conflict; repeat. | **Bounded resolution attempts per conflict-set** (e.g. 3), tracked in the coordinator's thread state. On exceeding, the coordinator **pauses merging** for that set and `reachOut`s Augustin with the branches + last verify report. Livelock converts to a human escalation, not an infinite loop. | | **Resolution fails its own re-verify** | Resolver's proposed merge doesn't pass `verify()`. | The resolver iterates in-pod up to a bounded number of proposals; if none pass, it `reachOut`s Augustin and **nothing lands on `main`** (fail-closed — same discipline as the write-wall: an unverified change is structurally uncommittable). The branches stay intact for a human. | | **Workstream abandoned mid-flight** | A workstream thread goes idle with an un-merged branch. | Branches are cheap and isolated; an abandoned `ws/` simply never signals ready, so it never merges and never blocks others. A janitor heartbeat can list branches with no recent commits + no ready signal and `reachOut` (or auto-delete after N days). No impact on other workstreams (that's the point of branch isolation). | | **Merge-of-merges race** | Two ready-sets try to merge into `main` at once. | Single serialization point by construction: the merge-coordinator is ONE thread; the durable merge workflow it kicks holds `main` for the duration of one merge+verify+deploy unit. A second ready-set queues in the coordinator's state. At replicas:1 this is airtight (fact #5, in-process). | | **Self-image mid-op brick** | The merge deploys a bad `images/mastra`; the resolving agent is the one being redeployed. | §9. | | **Push race regression** | Two workstreams somehow both target `main`. | The `force:false` ref PATCH (`lib/github.ts:111`) already rejects a non-fast-forward — a second concurrent write to the *same* branch 422s. With branch-per-workstream, same-branch concurrent writes shouldn't happen; if they do, the 422 is a safe loud failure, not silent corruption. | *** ## 9. The self-image special case (`images/mastra`) — highest stakes Two workstreams both bump `images/mastra`. **You cannot merge two image tags** — an image tag is a build output, not source. The only correct move: 1. **Merge the SOURCE** (`images/mastra/**`) on a branch, in the pod (§3.2). 2. **Rebuild + re-verify the merged source as ONE unit** — `verify(mergedMastraSource, ["pnpm install --frozen-lockfile", "pnpm typecheck", "pnpm test"])`. This is the self-capable arc's mastra self-verification (self-capable-arc §5.1), now applied to the *merged* tree, not a single workstream's tree. (Open question inherited from that doc: how much of the package to inject + pod RAM sizing — §10.) 3. **If re-verify passes: ONE deploy.** Commit merged source to `main` (→ CI builds ONE new `ghcr.io/amarcin/mastra` image) → bump the tag in `cluster/apps/agent/mastra/deployment.yaml` ONCE. Never two tag bumps, never a tag "merge." 4. **If re-verify fails: nothing lands, resolver escalates** (§8). The brain keeps running its current image. **The `replicas:1 + Recreate` sharp edge (fact #5).** When the merged self-image tag lands and Flux reconciles, Recreate kills the running mastra pod (the very agent that may have been coordinating) and starts the new one. If the merged image crashloops despite passing `verify()` (a runtime failure tests didn't catch — bad env assumption, startup ordering), there is nothing serving and no agent to fix it. Three layers of defense, in order: 1. **Re-verify the MERGE, not the branches** (§5) — catches the "A's change + B's change together break the build/tests" class, which is exactly the class a per-branch verify misses. This is the whole reason merge→re-verify is coupled. 2. **Revert-to-last-good-SHA is a pure tag change** (self-capable-arc §5.3): the previous mastra image is already in GHCR, so rollback = revert the one tag-bump commit → Flux rolls back → known-good brain returns. Fast because no rebuild. 3. **A watcher outside the brain's own liveness.** The self-healer heartbeat runs *in* the mastra process — useless if that process is crashlooping. **Open question (§10): does the self-image deploy need an external liveness→auto-revert** (a tiny k8s-native probe or a CronJob on a different node that reverts the tag if mastra is unready for N minutes)? This is the one place the "agent can't recover a pod it just killed" risk isn't fully closed by the design above. Verify lowers the odds; it does not eliminate them. Honest: "verified" means "builds + tests pass," not "cannot crashloop." > **Serialization for the brain specifically.** Even though branch isolation lets many workstreams edit `images/mastra` concurrently, the *deploy* of the brain should be serialized to one-at-a-time (the coordinator already serializes merges, so this falls out — but state it: never two `images/mastra` deploys in flight, because the second would race the Recreate of the first). Augustin explicitly rejected serializing *workstreams* with a lock; this is not that — workstreams stay concurrent, only the final brain *deploy* is one-at-a-time, which is a physical constraint of replicas:1+Recreate, not a concurrency cop-out. *** ## 10. Open decisions that need Augustin 1. **External liveness→auto-revert for the brain (§9).** The design closes the crashloop risk with re-verify + fast revert, but the *trigger* for the revert, if the new brain never becomes ready, can't live inside the brain. Options: (a) accept manual revert (Augustin reverts the tag when ntfy goes silent — but a crashlooped brain can't ntfy); (b) a tiny external CronJob/Deployment on i3/pentium that watches mastra readiness and reverts the tag commit via the GitHub API after N minutes unready; (c) a Flux `HealthCheck` + a manual gate. **Recommend (b)** — it's the only option that survives the brain being fully down, and it's small. Needs Augustin's go (it's new machinery outside the agent, which the design otherwise avoids). 2. **Merge granularity: per-branch or batched ready-sets?** Merge each ready workstream as it arrives (simpler, more merges, each small) vs. batch all-currently-ready and merge together (fewer deploys, but a bigger conflict surface per merge). **Recommend per-branch by default**, batching only when the write-scope hint (§2) shows two ready workstreams touch the same image (merge those two together so their overlap is resolved once, not twice). 3. **Inherited from self-capable-arc §7 (unchanged, still open):** how much of `images/mastra` to inject for a merged-source verify + whether to bump the eval pod's RAM; per-image verify command declaration (`images//verify.json`). The merge path reuses `verify()`, so these ride along. 4. **Resolver model + budget.** Every true conflict spawns an LLM resolution session (§8). At homelab scale (rare conflicts, one operator) this is fine — but say so: the resolver should run the *same* Opus-4.8-effort-high the main agent uses, and its cost is bounded by the attempt caps (§8), not open-ended. If conflicts ever become frequent, that's the signal that workstreams are badly scoped, not that the resolver needs optimizing. *** ## 11. Phased build sequence (slots into the self-capable arc) The self-capable arc ends at its Phase 5 (verified code-write path + self-deploy). This concurrency layer is **Phase 6**, and its first sub-phases are prerequisites that the arc's Phase 4 should absorb. * **Phase 6a — Branch-per-workstream write path (prereq; fold into arc Phase 4).** Add the optional `branch` param to `lib/github.ts` `commitFiles`/`commitFile` (default `main` = no behavior change). Route `images/**` code writes for a named workstream to `ws/`. *No resolver, no merge yet* — just isolation. Independently shippable; makes the push-race impossible before any concurrency exists. * **Phase 6b — Workstream registry + ready signal.** Extend `ThreadKind` with `"workstream"`; `registerWorkstream` + write-scope metadata; the `tellThread`-to-coordinator "ready" signal. Seed the `merge-coordinator` thread (like `concierge`). Still no merge — just the bookkeeping. * **Phase 6c — Merge + re-verify + deploy-once (the durable merge workflow).** The `runDurableMerge` workflow (option C): pod clone → `git merge` → `verify()` → rebuild → deploy-once, driven by the coordinator (option B). Handles the CLEAN-merge path fully. On conflict OR failed-verify, for now: `reachOut` to Augustin (resolver stubbed). This alone is a huge capability — concurrent workstreams that don't truly conflict now merge + re-verify + self-deploy autonomously. * **Phase 6d — The conversational resolver (the full vision, on beta).** `spawnResolver` + `askThread`/`watchThread`/`publishSharedState` behind `lib/autonomy.ts`. The resolver spawns on conflict/failed-verify, converses back to the workstreams, re-verifies, deploys-once, and drives rebases. This is where the beta Signals primitives (§7) come in — isolated behind the boundary, with the degrade paths as failure handling. * **Phase 6e — Self-image hardening (§9).** External liveness→auto-revert (open decision #1) + explicit one-at-a-time brain-deploy serialization. Gated on Augustin's §10.1 call. 6a–6c deliver autonomous concurrent self-editing for the common (non-conflicting) case with full re-verify safety and NO conversational-beta dependency — a complete, shippable system on stable primitives. 6d adds the conversational resolver on beta. 6e closes the self-image crashloop gap. Each phase is independently useful and testable (the durable-build e2e-test pattern applies: local sandbox backend, real `git merge`, a deliberately-conflicting fixture pair). *** ## 12. One-paragraph summary Branch-per-workstream comes first — the agent commits straight to `main` today, so there's no conflict, just a push race; give each self-editing thread its own `ws/` branch via a one-parameter widening of the existing GitHub Data-API write path. A workstream is a registered federation thread + that branch + (on demand) an isolated executor-pod worktree — no new machinery, it rides the turn-86 thread registry. On done, a workstream signals a single **merge-coordinator** thread, which kicks a **durable merge+verify+deploy workflow**: git does \~95% of the merge (three-way, in-pod), and a clean merge is NOT a green light — it runs through the self-capable arc's `verify()` (rebuild + re-verify for a deployable image) so two textually-clean-but-semantically-broken changes can't ship a crashlooping brain; then it deploys ONCE. The **LLM resolver is a spawned peer thread** that fires only for a true git conflict OR a clean-merge-that-failed-verify, and — on **beta** Signals, isolated behind `lib/autonomy.ts` — it CONVERSES back to the originating workstreams (`askThread` = the reply-correlation we get by awaiting the `accepted.output` that `tellThread` currently drops; `subscribeToThread`/`sendStateSignal` for watch + shared-state), folding their corrections into a re-verified, authoritative `main` that other workstreams rebase onto. The self-image case is the sharp edge: you merge source, rebuild, re-verify, deploy once, and because the brain is `replicas:1+Recreate`, the one gap the design can't fully close from *inside* the agent is an external liveness→auto-revert (open decision for Augustin). Beta risk is real and named per-primitive — but every conversational primitive degrades to a still-correct notify/poll/re-send fallback, so a Signals meltdown costs us the conversation, not the system. We're bleeding edge; the boundary is where we adapt.