Mentat 2.0: plan, execute, review — an agent-driven development loop

The annoying part of agent-driven development is not writing the agent loop. It is making the loop trustworthy. An agent that can push code needs a branch, a pull request, a reviewer, and a way for a human to see the whole story later. Most demos stop at docker run and a wall of logs. I wanted the boring middle: an issue gets labeled, the work happens somewhere, a real PR appears, CI runs, someone reviews it, and the outcome lands back on the issue.

There is also a physical constraint that shapes everything: an agent run should not share a process, a filesystem, or a checkout with the next one. Each phase should be a fresh, disposable container. Which means every phase must reconstruct its world from outside state — the issue, the PR, the branch, the labels. Once you accept that, the design writes itself: GitHub is the database.

The pipeline has three tiers. n8n is the trigger and the router. A small Go service — the runner — is the only component allowed to touch Docker. Everything that runs a model runs in a one-shot agent container. The runner's API is deliberately tiny: create a run, list runs, cancel, follow logs, stream completion callbacks.

GitHub issue labeled agent:ready
        │  webhook (HMAC-verified)
        ▼
n8n ── planning flow ──► runner ──► agent container (one per phase)
        ▲                 │              │
        └── callback ─────┘              └── stdout → Dozzle / Loki
        └── phase router: plan → execute → review ∥ CI → done

The whole loop on one canvas: GitHub holds the state, n8n routes the phases, the runner owns Docker, and the observability stack watches every container.

Each container gets a stack-matched image — a JDK for the Kotlin backend, Node and pnpm for the frontend — resolved from the repository in the request. It gets its own clone, its own branch name, its own secrets injected at create time, and a hard CPU and memory cap. When it exits, the runner emits a completion callback to n8n with the exit code and duration. Nothing is shared between phases except what is written back to GitHub.

The agent itself is a headless docker-agent run: --exec for no TUI, --safety restricted with a narrow allow-list, and a team file that pins the model, toolsets, and instructions. The instructions live in versioned skills that are baked into the image at a pinned commit, so an agent can only use the skills its phase exposes.

Plan. The agent reads the issue, searches the internal knowledge base over MCP, inspects the repository read-only, and posts one plan comment. The comment starts with a marker: <!-- agent-plan:v1 -->. That marker is the handoff. n8n does not parse prose or scrape logs; it looks for the marker.

Execute. A second container reads the plan from the prompt, creates agent/issue-<n>, implements the change, runs the repository's verification commands, commits, pushes, and opens the pull request itself. It posts no comments. The diff is the artifact. If the agent finishes without opening the PR, the orchestrator opens it as a fallback — the pipeline never depends on the agent remembering the housekeeping.

Review. A third container reviews the diff only — no checkout, no builds — using GitHub MCP to read the PR. It posts one comment starting with <!-- agent-review:v1 verdict=approved|changes -->. The router reads that verdict and either finishes the loop or sends the feedback back to execution. The reviewer is under one hard contract: approved means every finding from earlier rounds is resolved in the current diff — it reads its own prior review comments before deciding, and anything unresolved comes back as changes.

The phase router is a single n8n workflow with one job: look at the run's task id and the markers on GitHub, then decide the next phase. Task ids encode everything it needs — issue-42-plan, issue-42-exec-2, issue-42-review-1 — and every transition is a label change plus a dispatch. Because the phase is encoded in the task id, the runner needs to know nothing about the workflow: it just runs containers.

Feedback rounds push the same branch, so the PR updates in place. The loop is capped at three rounds — review feedback or failing CI, same counter; past that the issue is labeled failed with an escalation comment. The human still merges. Agents never merge.

The gate between "code exists" and "code is good" went through three designs, and the third one only came from admitting what the first two had in common: both made review and CI take turns when they could have run at the same time.

Generation one had a comfortable lie in it: the reviewer approved based on the diff, and verification was somebody else's problem. In practice the reviewer could not see CI, could not run anything (diff-only by design), and kept filing "verification evidence absent" as a major finding. The loop spun in place.

Generation two made CI the orchestrator's job, sequentially: execution finishes, n8n finds the PR, then polls the Actions API every 45 seconds for up to 45 minutes, and only when every run is green does it dispatch the reviewer. It worked — and then it failed in the most instructive way: a perfectly good PR "timed out after ~22 minutes" while its checks were still running. The gate had silently given up after a single poll, seven seconds in, because one IF node used a legacy operator name that n8n no longer recognizes — more on that in the war stories — and the failure message cheerfully reported a wait that never happened.

Generation three stopped sequencing. The reviewer is dispatched the moment the PR exists, and CI just runs — GitHub's problem, on GitHub's clock. When the verdict lands, the orchestrator evaluates the PR head's Actions runs exactly once and routes by result:

• All green — the review bot submits its official APPROVE, the PR is labeled agent:approved, and the issue gets agent:approved + agent:done. The approval, the label, and the checks all agree.
• Red — the failing check names become the feedback, and the next execution round starts on the same branch. Same PR, updated in place, round cap still applies.
• Still running (or unreadable) — the PR is labeled agent:approved with a warning comment: review passed, CI hasn't finished — verify the checks and merge. The workflow stops and hands the last mile to a human instead of burning forty minutes polling.

The ordering trick that makes this safe: the official APPROVE is deferred until the join. The reviewer's verdict alone is only a label; the formal GitHub approval — the thing branch protection counts — is submitted exclusively on the green path. A red CI after an approval can't happen, and a fix push auto-dismisses any stale review through ordinary branch protection. Merge stays blocked by required checks no matter what the agents do.

The verdict is posted on the PR, where the conversation belongs, and n8n then drops a one-line link comment on the issue. The issue stays a timeline of pointers; the PR stays the place where the work is discussed.

Everything above works with a personal access token, and that is exactly what I had. It also means the reviewer is the same identity as the PR author, which GitHub refuses to approve. A bot cannot approve its own work — for good reasons — so the loop could never produce an official approval, only a comment that said "approved".

The clean fix is three GitHub Apps, each with its own identity and the minimum permissions:

• dev bot — contents write, pull requests write, issues write. It is the PR author and the plan author.
• review bot — pull requests write, contents read, issues write, actions read. It reviews and submits the official APPROVE or REQUEST_CHANGES — but only on the green path.
• orchestrator bot — issues write, pull requests write, actions read. It owns labels, comments, the CI evaluation, and the fallback PR.

The runner became a small token broker: it signs an RS256 JWT with each app's private key, exchanges it for an installation token, caches that token for about 55 minutes, and injects the right identity into each run by team. n8n fetches the orchestrator and review tokens from the runner at execution start. No long-lived token is baked into an image or handed to an agent that does not need it.

Two operational lessons: when you change an app's permissions, the installation does not pick them up until an org owner accepts the update — and a cached installation token keeps the old permissions until it expires or the broker restarts. Both of those cost me a debugging round.

Almost every bug in this project was an integration bug, not a model bug. A few that are worth writing down:

• An unrecognized filter operator is silently false. The CI gate once "timed out after ~22 minutes" on a healthy PR — it had actually quit after one poll, seven seconds in. An IF node carried the legacy operator name smaller; modern n8n only knows lt/gt/… and evaluates anything else to false, logging one warning nobody reads. The loop guard took its "give up" branch on attempt zero, and a hardcoded failure message invented a 22-minute wait. Worse: a sibling Wait node had silently lost its unit and defaulted to hours. Both bugs hid behind a green execution status.
• Hand-authored workflow JSON can look right and route wrong. Rewriting the router by script, I flattened a Switch's four branches into one output group — valid JSON, renders fine in the editor, and at runtime every item takes branch zero while the other outputs go nowhere. The symptom was a label appearing "too early", which made no sense until the execution trace showed the dispatch item also flowing into the failure chain. If a graph can be wrong without looking wrong, walk every branch and assert the wiring.
• n8n 2.x delivers the raw webhook body as binary. Signature verification worked on the old shape (json.rawBody) and silently dropped every event on the new one. The handler now accepts both: binary payload decoded from base64, or the legacy string.
• A 204 response carries zero items. Deleting a label returns 204 No Content, and n8n emits no items for an empty body — so the next node never ran. Four downstream steps vanished, with green checkmarks everywhere. The fix is asking HTTP nodes for the full response, so even an empty DELETE yields an item.
• Array responses fan out. Adding a label returns the issue's whole label array. n8n turns each element into an item, so the "dispatch review" node ran once per label. The first time this happened it created two review runs for one PR. Same fix — full response on the label calls — plus consuming arrays deliberately with .all().
• MCP transports move. The remote GitHub MCP endpoint dropped SSE; the toolset now says streamable. The old transport returned a 405 and the agent started without its GitHub tools, which is a quiet kind of broken.
• Restricted safety mode denies everything you did not allow. That includes MCP tools and filesystem writes. My first allow-list covered shell commands only, so the planner could read the world but not post its plan. The verifier could edit code but not create a directory. Allow-list the identities you actually use: github_*, nuclino_*, and the file-writing tools.
• A Dockerfile rewrite silently dropped the entrypoint. Containers started, ran bash, and exited zero with empty logs. That one is invisible unless you inspect the image config — the pipeline looked perfectly healthy while doing nothing.
• SQLite pragmas are per-connection. A busy_timeout set once at startup only applied to the first pooled connection; concurrent writes from the event watcher and the API occasionally failed with "database is locked". Moving the pragmas into the DSN fixed it.

Labeling an issue now produces, without anyone at the keyboard: a plan comment written by the dev identity, a PR authored by the dev bot, review and CI running side by side, an official GitHub verdict posted on the PR by the review bot, and an issue timeline that links to every review round. If the review asks for changes — or CI flags a failing check — the next round starts from that feedback on the same branch. When the verdict and the checks both say yes, the approval lands and a human merges. When CI simply hasn't finished by review time, the pipeline says so out loud, labels the PR approved-with-a-caveat, and leaves the last mile to a person instead of pretending.

The observability side is boring in the best way: every phase is a container with a name that encodes issue, phase, and round; its stdout is one clean transcript (no JSON firehose); Dozzle streams it live; Loki keeps it for a week and Grafana slices it by run, team, or task. When something goes wrong, the question is almost always "which phase, and what did the orchestrator see" — and both answers are a click away.

• Put state where humans already look. Labels, markers, branches, and PRs beat a bespoke database nobody can inspect. The agent can be ephemeral because GitHub is not.
• Separate identities, separate powers. The author and the reviewer must not be the same actor. GitHub Apps give you that with short-lived tokens and honest audit trails.
• Make handoffs machine-checkable. A marker on the first line of a comment is a contract. If the marker is missing, fail loudly — never loop silently. "Approved" is also a contract: it must mean all feedback resolved, or the word decays.
• Run slow things in parallel; join at the decision. The worst version of a CI gate is a poll loop in front of the review; the best is no gate at all — start both, and make the approval wait for the verdict and the checks to agree. If they can't agree in time, say so and hand a human the last mile instead of stalling.
• A silent false is worse than an error. The costliest bug in the project was an operator name n8n didn't recognize: no exception, one log line, and a loop guard that took its give-up branch on attempt zero. Anywhere a framework can coerce your intent into a default, assert the default is not a decision.
• Every integration layer will lie to you once. Empty bodies, split arrays, moved transports, silently dropped entrypoints, flattened branch graphs that render fine and route wrong. Test the seams, not just the happy path — and when a graph matters, walk every branch.

The pipeline is not trying to replace the engineer. It is trying to make the boring middle of a small change — plan it, type it, open the PR, wait for CI, take the first review pass — happen without a human in the loop. The human still gets the interesting parts: the plan review, the merge, and the taste.