Cracks in the Agent Loop: How to Make Coding Agents Predictable, Auditable, and Recoverable

This is a follow-up to the write-up on the pipeline itself — a small loop that plans, implements, and reviews issues in disposable containers. The lessons below apply to any agent loop, not that specific stack.

A normal-looking issue landed in a real codebase. The planner wrote a sensible plan and estimated the change at 550–700 lines. The executor picked it up, and 13 minutes and 28 seconds later the run was dead.

It had used 100 model calls, changed 1,951 lines across 21 files, written 942 lines of tests, and opened zero pull requests. The run record said agent exited with code 1. The real reason was max_iterations — the hard cap on model calls before the run is killed.

The agent was not idle. It explored broadly — a dozen broad file reads, 268 KB of tool output — then wrote the feature, then spent its last twenty calls fighting the same lint and static-analysis errors. It never reached a commit. I recovered the work by splitting it into two smaller pull requests, stacked one on the other, and both passed CI. But the recovery was manual. The loop had failed in a way that needed a human to diagnose.

The tempting response is to raise max_iterations and move on. It buys more of the same, not a better outcome: more model spend, more tool calls, and a larger diff for a human to review. A 2,000-line pull request is not a delivery win, even if it eventually compiles.

The plan was not wrong because the model was weak. It was wrong because it was treated as a suggestion. It had no size arithmetic, no file contract, and no checkpoint. The executor could spend a hundred calls and still have nothing to show. The loop had a circuit breaker, but no budget.

The operating question is not “did the agent finish?” It is: what did it cost, what did it produce, and can we recover it? The goal is not an agent that never fails. It is an agent system that fails small, visibly, and recoverably.

I found seven recurring cracks. Each is an operating risk, not a prompt bug.

The fix is four moves, in this order. None of them requires a smarter model.

Contract

Every plan carries acceptance criteria, a size estimate, an exact file list (we call ours plan-files), and an end marker. The executor stops if the plan is missing, truncated, or does not name the slice it was asked to implement. A plan that cannot be checked is not a plan; it is a prompt.

What a plan contract looks like

budget: 780 lines / 12 files
scope: src/upload/session.ts, tests/upload/session.test.ts
done: explicit end marker

Ours is a small comment marker; the syntax matters less than the fact that a machine can check it. For larger work, the same contract also carries a split: a small machine-readable list of slices, each with its own files, estimate, and dependency.

Bound

Put a budget on calls, wall-clock time, changed lines, changed files, and full verification attempts. Budgets are policy, not suggestions. One owner sets them; exceptions require a human decision. The circuit breaker is the last resort, not the plan.

Split

When the estimate crosses roughly 600 lines, more than eight files, or the issue target with a safety buffer, split the work. One child issue per slice, one pull request per slice. Slices are disjoint, linear, and stacked; the final pull request closes the parent issue. Each run stays small enough to review and cheap enough to retry.

Recover

Checkpoint commits and pushes at meaningful milestones, so a dead run leaves evidence rather than a blank workspace. Resume is idempotent: retry the failed slice, do not re-plan the parent. A failed slice blocks its dependents and marks the parent failed. The operator should never have to reconstruct what happened from a container that no longer exists.

Those four moves harden the loop. The next section hardens the blast radius around it.

The loop bounds what the agent does. It does not bound what the agent can reach. A coding agent runs with a shell, a filesystem, network access, and tool servers. If any of those are over-privileged, a bad instruction or a confused model can do damage that no iteration budget will catch.

The design principle is simple: no component gets a credential or a control surface it does not need. In practice that means three boundaries.

No direct container-runtime access

Agents run in disposable containers and never get access to the container runtime's control socket. The service that launches containers is the only component allowed to talk to the runtime, and it goes through a narrow read-write proxy: containers, images, and events, plus the specific create, start, and stop operations it needs. No exec, volume, network, plugin, or administrative surfaces. The log viewer and log shipper use a separate read-only proxy with POST disabled. No application code mounts the raw socket.

Tool servers behind a credential proxy

External tool servers — the Model Context Protocol (MCP) servers an agent can call — are another trust boundary. The agent does not hold the upstream API key. It gets a short-lived, per-run token bound to the run ID; a small proxy validates that token and adds the real upstream credential on the internal network. The long-lived key exists only in the proxy, not in the agent container.

Short-lived identities and narrow keys

Code-host access is split across separate short-lived bot identities for development, review, and orchestration (we use three GitHub Apps). Each run gets a short-lived installation token for the role it needs — not a personal token, and not the orchestrator's token. The container receives only the model API key for the model it is actually using; unused provider keys are never injected. The safety mode is restricted, OS capabilities are dropped, and the command allow-list is explicit.

Treat monitoring as an operating dashboard, not a metrics dump. Four panels cover the loop:

Useful starting SLOs, not claims:

• At least 95% of plans carry a valid budget, file list, and end marker.
• Fewer than 10% of runs exceed 80% of their call or line budget.
• Fewer than 5% of terminal runs end without a pull request.
• p95 verification attempts stay at two or fewer.
• Every run exposes a termination reason and its effective model.
• Cost per merged PR is reviewed weekly, not just token spend.

If you cannot answer these, the loop is not hardened yet.

The goal is not an agent that never fails. It is a system that fails small, fails visibly, and can be retried without losing the work — or the budget. The next test is not a better benchmark. It is the next oversized issue.