Bharat Bhavnasi San Francisco, CA, USA

Compiling the Agent Loop Away: Late May's Anti-Orchestration Turn

• 6 min read
#agents #inference #distillation #latency #research

For two years the agentic playbook was additive. Want better results? Add a step. Add a planner, a critic, a researcher sub-agent, another tool, another reflection pass. Capability went up and so did the bill: every step is a round trip, and a multi-step research agent that costs $0.50 and takes 30 seconds is normal now. A cluster of papers that landed in the last two weeks of May 2026 does the opposite. Instead of optimizing the prompts inside the loop, they go after the loop itself — and they each pay down the orchestration tax in a different currency.

This is a meaningful shift in where the engineering happens. Caching, routing, and tool-result distillation — the moves I wrote about in cost-optimized architectures — all keep the runtime loop intact and trim its edges. The new work treats the loop as the thing to be eliminated, hidden, or moved out of runtime entirely.

Compile it into the weights

The most aggressive move is Compiling Agentic Workflows into LLM Weights. The idea is exactly what it says: take a working multi-agent pipeline — planner, researcher, writer, reviewer — run it thousands of times, collect the input/output pairs, and fine-tune a smaller model on the trace. The trained model reproduces the pipeline’s output in a single forward pass. The paper’s headline numbers are the ones that make you sit up: a 0.50four−callworkflowcollapsestoa0.50 four-call workflow collapses to a 0.005 inference call, and 30 seconds of latency becomes 2. Two orders of magnitude on cost, an order of magnitude on latency. The companion line of work, AgentArk, frames the same trick as distilling “explicit test-time interactions into implicit model capabilities” — the multi-agent dance becomes a learned reflex.

The catch is structural, not incidental. When you compile a loop into weights, you freeze it. The runtime loop’s whole value was that it was inspectable and editable: you could read the planner’s plan, swap a tool, add a guardrail between two steps. A compiled workflow is a black box that happens to be cheap. You lose per-step observability, you lose the ability to inject a human-in-the-loop checkpoint, and — most importantly — you lose adaptability. The compiled model is a snapshot of the pipeline’s behavior on the distribution you sampled. Drift the task distribution and you are back to fine-tuning. This is a genuinely good trade for high-volume, stable workflows (classification, extraction, structured research over a fixed corpus) and a bad one for open-ended work. Treat compilation as a deployment optimization for a workflow you have already validated, not as a way to build one.

Hide it behind speculation

IdleSpec (Choi et al.) attacks a different fact: in most agent runs, the model is not the bottleneck. The wall-clock time is dominated by tool execution and environment latency — the API call, the file read, the database query. While the agent waits for an observation, the GPU sits idle. IdleSpec uses that idle window to speculatively pre-plan the next action, generating candidate continuations so that when the tool result lands, the agent can respond instantly. The paper reports the speculation is correct 60–80% of the time and cuts perceived latency by more than half.

The honest framing matters here. IdleSpec does not make the agent cheaper — it makes it feel faster. The speculative drafts burn extra tokens during the idle window, so on a metered API your per-task cost goes up, not down. And when a tool returns faster than a single reasoning step, the technique degrades to the vanilla baseline. This is borrowed straight from speculative decoding’s playbook: spend spare compute to mask latency. It is the right move for interactive agents where a human is watching a cursor blink, and the wrong move for a batch pipeline where nobody cares whether step 7 took 200ms or 2s.

Move adaptation below the prompt

The third paper is the one I keep thinking about. MOSS (Self-Evolution through Source-Level Rewriting, May 21) observes that deployed agents are static: they do not learn from their own failures, and a recurring bug persists until a human ships a fix. MOSS closes that loop by letting the agent rewrite its own source code. Each evolution is anchored to a batch of curated production-failure traces; a pluggable coding-agent CLI proposes the code change; candidates are verified by replaying the failure batch against an ephemeral trial container; and only a passing candidate gets promoted via a consent-gated, health-probe-guarded in-place swap. On OpenClaw — this year’s breakout open-source agent — MOSS lifted a four-task mean grader score from 0.25 to 0.61 in a single cycle with no human in the loop.

What makes this different from “self-evolving agents” hype is the medium. Most self-improvement work mutates prompts or scratchpad memory. MOSS argues source code is the better surface: it is Turing-complete, deterministic, and does not erode under long-context drift the way an ever-growing prompt does. A fix written into a function stays fixed. The risk is equally obvious — an agent with commit access to its own substrate is a powerful failure mode — which is why the verify-replay-and-rollback scaffolding is the actual contribution, not the rewriting itself.

These three only make sense together, and they line up with where the loop is physically running. As orchestration moves server-side into managed-agent platforms (Google’s I/O 2026 managed agents, Anthropic’s enterprise agent services), the provider owns the loop and can compile, speculate, and self-evolve on your behalf — none of which you can do from a thin client. The pattern under all of it: the agent loop is becoming a compile-time and platform-time artifact, not just something you assemble at runtime. The next year of “agent frameworks” is going to be as much about what you can bake out of the loop as what you can add to it.

Worth bookmarking

Share:
Suggest changes