The Environment Became the Curriculum: Agent RL's Synthesis Turn

The most expensive thing in agent RL is no longer the GPUs, the base model, or even where you put the reward. It’s the environment — the simulated world the agent acts in while it learns. For most of the past year, building those environments meant hiring domain experts to hand-author tasks, and the going rate was brutal: 15 to 25 hours of a PhD’s, lawyer’s, or tax accountant’s time per data point. Anthropic leadership has reportedly discussed spending over a billion dollars on environments in a single year. That is the real bottleneck, and the last few weeks of research are an organized attempt to make it disappear.

The unit of value keeps migrating

Watch where agent RL teams have spent their scarce attention over three successive waves. First it was data — collect more trajectories. Then it was reward — and as I wrote about the reward-design split, serious groups still disagree on whether to score every turn or only the outcome. Now the frontier has moved one level up the stack again, to the environment: the set of tools, states, and verifiable tasks the policy is allowed to practice against.

This is not a fashion cycle. Each migration happened because the previous layer got commoditized and the next one became the constraint. You can buy trajectories. You can copy a reward recipe from a paper. But a good environment — one with tools that actually execute, states that persist coherently, and tasks whose success is machine-checkable — is still mostly handcrafted, and handcraft does not scale to the breadth that general agents need. The industry response was a new class of well-funded vendors: Mechanize paying engineers half a million dollars to build a handful of robust coding grounds, Prime Intellect launching what it pitches as a Hugging Face for RL environments. Those are marketplaces for human-built environments. The research response is more radical: generate the environments themselves.

Synthesis: mining environments instead of writing them

The clearest statement of the new direction is Agent-World (April 2026). Instead of authoring sandboxes, it autonomously mines thousands of real-world themes, uses deep-research agents to pull data and synthesize executable tools, and ends up with 1,978 environments spanning 19,822 tools — all generated, not hand-built. Tasks come with controllable difficulty and programmatic verification. The reported payoff is the part worth noting: 8B and 14B agents trained this way beat proprietary models across 23 benchmarks, and performance scales with environment diversity rather than just parameter count. The environment, not the model, became the thing you scale.

AutoForge attacks the same problem from the tool side. Its pipeline starts from nothing but tool-description documentation, builds a state database and Python implementations, constructs a dependency graph over the tools, and takes random walks through that graph to yield diverse, multi-step tool sequences — which become tasks. It pairs this with ERPO, an algorithm doing advantage estimation at the environment level to tame the user-simulator instability that wrecks naive setups. Results land on tau-bench, tau2-bench, and VitaBench with an emphasis on out-of-domain generalization. CuES rounds out the cluster by generating tasks directly from an environment’s structure and affordances, driven by a curiosity signal so the synthesized tasks stay meaningful rather than degenerate.

The connective tissue across all three is co-evolution. Agent-World makes it explicit: train, diagnose the agent’s weaknesses in a held-out arena, then automatically generate new environments targeting those weaknesses, and repeat. The environment stops being a fixed dataset and becomes a curriculum that adapts to the student. A late-April survey on rethinking agentic RL frames the same shift in the abstract: away from “static objectives and episodic interactions” toward agents whose learning loop includes self-reflection and dynamic strategy adaptation. You cannot have an adaptive learner without an adaptive environment to learn in.

Where this breaks

I am bullish on the direction and skeptical of the easy version of it. Three failure modes are already visible.

Verifiability is the whole game, and it does not synthesize for free. The reason these papers lean on tool-use and coding domains is that success there is programmatically checkable. Generate environments for legal reasoning or medical triage and you are back to needing the expert — not to build the sandbox, but to certify that the auto-generated reward signal means anything. Synthesis moves the human cost; it does not delete it.

Auto-generated tasks drift toward the trivial or the gameable. A random walk over a tool graph produces a lot of tasks that are either too easy or solvable by a degenerate policy. CuES’s curiosity objective and AutoForge’s difficulty filtering exist precisely because unfiltered synthesis collapses. Reward hacking gets worse, not better, when the thing being hacked was also machine-written.

Co-evolution can chase its own tail. An environment generator tuned to the current policy’s weaknesses can overfit the curriculum to quirks of one checkpoint, producing agents that ace the synthetic arena and stall on real distribution. The held-out “fresh task” arenas in these papers are load-bearing; without an environment source the generator cannot see, the loop measures its own reflection.

If you train agents, the practical takeaway is to treat environment construction as a first-class part of your stack, not a one-time data-collection chore. Tooling is arriving: NVIDIA’s open-source NeMo Gym standardizes building and scaling environments and now ships with agent harnesses and dozens of environments out of the box, and Unsloth’s write-up on RL environments is a decent practitioner on-ramp. The teams that win the next round will be the ones who can manufacture verifiable, hard, diverse environments cheaply — and who are honest about which domains still need a human to sign off on the reward.

Worth bookmarking

• Agent-World — scaling real-world environment synthesis with co-evolving agents and environments (arXiv, April 2026).
• AutoForge — automated environment synthesis from tool docs plus the ERPO algorithm.
• CuES — curiosity-driven, environment-grounded task synthesis.
• Rethinking Agentic RL in LLMs — the paradigm-shift survey framing static-vs-adaptive learning.
• NVIDIA NeMo Gym — open-source library for building and scaling RL environments.
• Silicon Valley bets on environments — the market context behind Mechanize, Prime Intellect, and the environment land grab.