For a coding Agent evolving a Policy, this creates two intertwined challenges:

1. Perception: recover nearby terrain, entities, inventory, and player status from the observation.
2. Long-horizon control: turn that state into a coherent strategy for survival, exploration, combat, and development.

In this experiment, we separate these two sources of difficulty. We evaluate GPT-5.6 Sol, Terra, and Luna under the same Crafter task, but give their evolved Policies either the original RGB observation or a local symbolic representation of the same visible state.

The difference is substantial for Sol and Terra, but much smaller for Luna. Removing most visual recognition work therefore exposes different bottlenecks in the Policies evolved by the three coding Agents.

Crafter as a Policy-Evolution Environment​

Crafter procedurally generates a fresh world for every Episode. The player begins without tools or resources and must balance immediate survival with longer-term development.

A capable Policy needs to coordinate several behaviors:

• maintain food, drink, and health;
• explore an initially unknown world;
• collect increasingly advanced resources;
• craft and place tools and structures;
• avoid or fight hostile creatures;
• retain enough local information to revisit useful areas.

These objectives compete with one another. Exploration creates opportunities for development but also exposes the player to danger. Crafting requires resources that may be far from safety. A Policy that focuses only on immediate survival can stagnate, while aggressive progression can quickly collapse if basic needs are neglected.

Crafter therefore tests whether a coding Agent can evolve a coordinated long-horizon program, rather than merely discover a locally useful action rule.

RGB vs. Local Symbolic Observations​

The two conditions use the same simulator, procedural worlds, action space, reward metric, Episode pools, and evaluation setup. Only the observation exposed to the Policy changes.

RGB​

The RGB Policy receives the rendered 64 × 64 × 3 Crafter frame.

It must infer:

• terrain from colors and textures;
• creatures and objects from sprites;
• inventory and vitals from the HUD;
• player position and orientation from rendering;
• useful state under changing illumination, including nighttime.

Local symbolic​

The symbolic Policy instead receives structured information for the same local region:

• local terrain and entity IDs;
• health, food, drink, energy, resources, and tools;
• facing direction, sleeping state, and daylight.

This representation removes most object recognition, HUD reading, and nighttime visual ambiguity.

However, it does not expose privileged global state. The Policy still receives no global semantic map, absolute position, Environment seed, hidden creature state, or other information outside the local observation.

It must still explore, remember useful locations, handle collisions, sequence resources, time interactions, fight enemies, and coordinate survival with development.

The paired experiment therefore simplifies state recognition while preserving most of the long-horizon decision-making problem.

Long-Horizon Survival Score​

Crafter's canonical score is primarily achievement-oriented. For our policy-evolution setting, we additionally want to distinguish a Policy that occasionally reaches advanced achievements from one that can survive reliably while continuing to make progress.

We therefore use the Long-Horizon Survival Score (LHS Score) as the primary Benchmark metric.

At each step, its survival component contains two signals:

• an alive reward for remaining alive;
• a vital-quality reward determined by the weakest of health, food, and drink.

Using the weakest vital is intentional: high food and health should not compensate for a Policy that is about to die from thirst.

LHS also retains bounded secondary incentives for useful development:

• the first unlock of a new achievement;
• actual restoration of food or drink;
• productive repeated behaviors such as resource collection, combat, and planting.

Repeated actions are capped within rolling windows, preventing a simple farming or maintenance loop from dominating the score.

Across Episodes, LHS further emphasizes robustness. The aggregate score combines:

• average healthy-survival return;
• additional weight on the weakest quarter of Episodes;
• bounded development and maintenance return.

Conceptually,

LHS = average survival + lower-tail robustness + bounded progression

rather than rewarding only the best trajectories.

This makes short achievement-rich Episodes insufficient to compensate for a fragile survival Policy, while still encouraging the Agent to progress beyond passive survival.

The canonical Crafter score is reported separately as a diagnostic of technology-tree progression.

Experiment​

We evaluate GPT-5.6 Sol, Terra, and Luna.

For each coding Agent, we run one policy-evolution trajectory with RGB observations and one with local symbolic observations. All six Runs use the same long-horizon survival objective.

The evolved candidate selected at the end of each Run is evaluated on 64 held-out Episodes.

In addition to LHS, we report:

• mean and maximum effective survival;
• the fraction of Episodes surviving at least 300 steps;
• canonical Crafter score;
• achievement coverage.

These diagnostics allow us to distinguish average robustness, exceptionally long trajectories, and technology progression.

Results​

The effect of simplifying perception differs sharply across the three Agents.

Each cell compares RGB → local symbolic observations.

For Sol, LHS increases from 5.70 to 11.53, while mean survival rises from 195 to 316 steps. Its longest held-out trajectory grows from 401 to 1,043 steps.

Terra shows an equally pronounced shift. LHS more than doubles, mean survival increases by 127 steps, and its longest trajectory reaches 871 steps.

The change for Luna is much smaller. Symbolic observations increase achievement coverage substantially, from 5 to 11 achievements, but mean survival rises by only 13 steps and LHS improves by only 8%.

The difference is especially visible in robust long-horizon survival. Under symbolic observations, 39.1% of Sol Episodes and 31.2% of Terra Episodes survive at least 300 steps, compared with only 3.1% for Luna.

For Sol and Terra, simplifying perception therefore changes not only their best-case behavior, but the broader survival distribution.

Terra's symbolic Run includes one validation protocol failure and one protocol error during the held-out Assessment. The failed held-out Episode receives zero under the Benchmark definition; among successfully completed Episodes, its shortest survival is 156 steps.

One World, Six Policies​

Aggregate results measure robustness across procedural worlds. To make the behavioral differences easier to inspect, we also replay the six selected Policies in the same showcase world.

The earlier fixed world happened to favor Luna. We replaced it using a deterministic, separate 128-Episode showcase pool: eligible Episodes had to complete normally, place Sol above the shared RGB baseline, and order the symbolic Policies as Sol > Terra > Luna by effective survival, with every adjacent gap at least 50 steps. Among eligible Episodes, we selected the one closest to the corresponding held-out mean survival values, rather than the one with the largest gap. Because this selection uses Policy outcomes, the replay remains a qualitative illustration, not evaluation evidence.

For the symbolic conditions, each Policy still receives only its structured local observation. The RGB GIF is a human-facing deterministic replay of that Policy's recorded Actions in the identical world; those RGB frames are not Policy inputs. All six GIFs share the same timeline. A Policy that ends early holds on its terminal frame.

Sol	Terra	Luna
RGB · 244 steps	RGB · 162 steps	RGB · 162 steps
Symbolic · 391 steps	Symbolic · 261 steps	Symbolic · 194 steps

The symbolic row now reflects the aggregate ordering clearly: Sol continues longest, Terra occupies the middle, and Luna ends much earlier. Terra RGB and Luna RGB cannot be separated on a common Episode because both Runs selected the byte-identical packaged baseline; their matching replays and 162-step outcomes are intentional.

Three Agents, Different Bottlenecks​

The paired results expose a different limiting factor for each Agent.

Sol: strong RGB progress, then another large gain​

Sol already beats the packaged baseline with RGB, then improves again with symbolic input: LHS rises from 5.70 to 11.53, mean survival from 195 to 316 steps, and coverage from 10/22 to 18/22. It can jointly improve perception and control, but more reliable state recognition still unlocks a large gain.

Terra: perception was a major bottleneck​

Terra's RGB Run falls back to the unmodified baseline, while symbolic input raises LHS from 4.58 to 9.88, mean survival from 164 to 291 steps, and coverage from 5/22 to 14/22. Visual state extraction was therefore its dominant bottleneck in this Run.

Luna: better recognition does not solve coordination​

Luna also falls back to the baseline under RGB. Symbolic input expands coverage from 5/22 to 11/22, but LHS only moves from 4.58 to 4.97 and mean survival from 164 to 178 steps. Its remaining bottleneck is long-horizon coordination rather than recognition alone.

Changing the observation contract affects their policy evolution in fundamentally different ways.

What Does the Ablation Tell Us?​

At first glance, Crafter poses a single challenge: evolve a Policy that can survive and develop in an open world.

The paired experiment shows that the difficulty can originate at different stages.

For Sol and Terra, local visual state extraction is a major part of the challenge. Removing object recognition, HUD reading, and nighttime visual ambiguity leads to much stronger survival and technology progression.

For Luna, perception is only part of the problem. Cleaner state information enables broader development, but the resulting Policy still struggles to turn those capabilities into reliable long-horizon survival.

This is the distinction we want environments such as Crafter to expose.

A final scalar score tells us which Program performs better. Controlled changes to the observation interface can additionally reveal where policy evolution stops improving.

Crafter therefore evaluates more than whether a coding Agent can write a game-playing Policy. It gives us a way to separate failures of perception from failures of long-horizon planning, control, and program coordination.

These six Runs represent individual policy-evolution trajectories rather than repeated statistical trials, and their training Episode consumption is not identical. We therefore do not interpret them as a general ranking of Sol, Terra, and Luna.

The narrower observation is more informative:

Removing local visual recognition transforms the Policies evolved by Sol and Terra, but only modestly improves Luna's survival—revealing different bottlenecks behind long-horizon policy evolution.

NetHack is a turn-based roguelike set in a procedurally generated dungeon. The full game asks the player to descend, obtain the Amulet of Yendor, return to the surface, and complete an ascension. Reaching that goal requires much more than winning individual fights: the player must explore unknown layouts, interpret messages, manage resources, remember useful locations, and survive permanent death.

A simplified progression loop looks like this:

explore a dungeon level
        ↓
handle creatures, obstacles, and resources
        ↓
find a downward staircase
        ↓
descend and repeat

The details change in every Episode. Rooms and corridors are rearranged, objects and creatures appear in different places, and only part of the current level is visible. An Action that looks reasonable locally may waste hundreds of turns, consume scarce food, or lead the character away from the route it was trying to follow.

This makes NetHack a useful Environment for studying executable strategy. A Policy must combine immediate reactions with memory and longer-term goals, and it must recognize when the world did not respond as expected.

Bringing NetHack into EvoPolicyGym​

The Benchmark integrates NLE 1.3.0 NetHackScore-v0, backed by NetHack 3.6.7. The Policy receives a semantic view of the terminal map together with status values, the current message, public inventory entries, and the current input mode. It chooses from 23 Actions covering movement, running, stairs, waiting, kicking, eating, searching, and message prompts.

The Action profile is intentionally narrower than the complete NetHack command set. Within a 5,000-step Episode, it concentrates the experiment on early-game exploration, obstacle handling, survival, and descent rather than full-game ascension.

The score follows progress recognized by NetHack and penalizes repeated steps that leave the character frozen in place. This gives the Agent a primary optimization signal while dungeon depth, game score, and frozen-step rate help explain what kind of behavior produced it.

Learning from complete trajectories​

Many weak NetHack Policies do not crash. They continue running while pushing against a boulder, trying to cross iron bars, alternating between two tiles, or standing on a staircase without descending. An aggregate score says that the Policy performed poorly, but not where its behavior broke down.

For each training submission, the Environment therefore returns the complete Policy-visible trajectory for every evaluated Episode. The Agent can inspect positions, Actions, messages, status changes, and repeated states, then connect those patterns back to the source code.

submit an executable Policy
        ↓
inspect complete training trajectories
        ↓
identify a behavioral failure
        ↓
rewrite memory, routing, or interaction rules
        ↓
submit a new Program

The Environment supplies evidence, not a diagnosis. Deciding which Episodes to inspect and which patterns matter remains part of the coding agent's work. Lasting improvement is stored in the executable Program rather than in model weights or hidden state carried between Episodes.

Experiment​

We ran three GPT-5.6 model variants—Luna, Terra, and Sol—through Codex. Each Agent began with the same packaged baseline and could improve it using training scores and trajectories. The optional NetHack optimization Skill was disabled, so the Agents had to develop their own analysis and revision process.

The baseline already performs simple local exploration. It remembers visit counts, usually prefers a less-visited neighboring tile, avoids immediate reversal when alternatives exist, searches periodically, kicks visible closed doors, and eats recognized food when hungry. It does not build an explicit map, plan routes to distant targets, remember failed edges, or treat descending as a persistent objective.

The training allowance was a ceiling rather than forced consumption. Sol used all 128 Episodes, Terra used 68, and Luna used 40 before finishing.

Results​

Sol produced the strongest selected Policy in this experiment. It reached a mean Assessment return of 204.026, a mean game score of 208.230, and a mean dungeon depth of 2.867. Its deepest held-out Episode reached depth 11.

All three selected Policies completed the held-out Assessment without a Policy execution failure. None ascended. The results measure early-game exploration, survival, and dungeon progress—not complete NetHack mastery.

Submission 000008, training Episode 16. This replay shows an Agent-written Policy acting autonomously from the first observation to the end of the Episode. It is a representative training trajectory, not part of the held-out Assessment reported in the table.

The baseline's capability boundary​

The baseline can answer a useful local question:

Which visible neighboring tile has been visited least?

It cannot yet answer the larger navigation question:

How should I build and maintain a route through an uncertain, multi-level dungeon?

That gap became the main opportunity for Policy evolution. Moving beyond the baseline required a Policy that could preserve knowledge after a location left the screen, pursue a target across several rooms, and revise its plan after a failed Action.

What strategy did Sol build?​

Sol turned the local exploration baseline into a more structured navigation system. Its final Policy combined three ideas that reinforce one another.

Persistent spatial memory​

The Policy records discovered terrain, routes, targets, and movement outcomes. Instead of repeatedly choosing only among adjacent tiles, it can use earlier observations to navigate through known rooms and corridors. Information remains useful after the character moves elsewhere on the level.

Goal-directed exploration​

Sol made dungeon progress an explicit objective. When the Policy knows about a downward staircase, it can preserve that target, route back to it, and use it. When it does not know a staircase, it seeks unexplored space rather than merely choosing the least-visited visible neighbor.

This turns depth from a metric observed after the Episode into a goal expressed inside the executable strategy.

Failure detection and recovery​

Movement in NetHack can fail because of walls, boulders, iron bars, creatures, doors, or a stale interpretation of the map. Sol compares the intended move with the next observation. If the expected transition did not occur, the Policy can mark the route as blocked, discard an invalid target, and choose another path instead of repeating the same Action.

The result is more general than a list of special cases: observe the outcome, update the internal map, and replan.

How agents used Environment Feedback​

Luna — remembering obstacles. Luna found trajectories dominated by repeated attempts to move through a boulder or iron bars. It added memory for failed directions and reduced immediate retries against static obstacles.

Terra — escaping loops and using stairs. Terra introduced anti-loop behavior and explicit stair descent. Its strongest selected candidate came from an earlier revision, illustrating that continued editing does not always produce a better Policy.

Sol — organizing fixes into a navigation system. Sol combined remembered routes, stair-oriented progress, blocked-edge handling, and target recovery. Rather than treating each failure as an isolated patch, it connected the lessons through a shared model of position, routes, targets, and outcomes.

In every case, the durable result was not the Agent's explanation of the trajectory. The lesson had to survive as code that independently received observations and returned Actions during held-out evaluation.

Findings and boundaries​

Complete semantic trajectories were enough for the Agents to locate behavioral failures and make useful Policy changes without an Environment-authored diagnosis. The experiment also shows why NetHack needs more than one metric: return ranks candidates, while depth, game score, and frozen-step rate reveal different aspects of exploration and progress.

These results are still an initial study. The Agents did not consume equal training budgets, and each model lane is represented by one primary Run. With the short 128-Episode training ceiling, coding-agent search can vary because of randomness: a separate Terra Run under the same Environment configuration scored 49.464, compared with 80.237 here.

The table should therefore be read as evidence that the Agents built better NetHack Policies in these Runs, not as a general ranking of Luna, Terra, and Sol.

Code and notes​

• NLE NetHack Benchmark
• Evaluation and Runs
• Policy boundary

The EvoPolicyGym adapter is MIT licensed. NLE and NetHack remain separate dependencies governed by their respective licenses.

Balatro is a roguelike deck-building game centered on scoring poker hands. A player begins each Run with a standard deck, scores points by playing hands, strengthens the build through the shop, and wins by defeating the Boss Blind in Ante 8. Draws, shops, and rewards change from Run to Run; losing starts a new Run.

Each Run repeatedly follows the same loop:

choose a Blind → play or discard → reach the target score → earn money → build in the shop → next Blind

• Blinds: A Run has eight Antes, each containing a Small Blind, Big Blind, and Boss Blind. The Small and Big Blinds can be skipped in exchange for a Tag, while the Boss Blind adds a rule that changes how the round plays.
• Playing a hand: The player selects one to five cards. Poker hands such as pairs, two pair, straights, and flushes determine the base Chips and Mult. Scoring cards and other effects modify both values, producing a final score of Chips × Mult.
• Clearing a Blind: Scores from multiple hands accumulate within a Blind. Reaching the target Chips clears it; running out of hands ends the Run.
• Discarding: A discard replaces unwanted cards, using a limited resource to improve later hands.
• Building: Clearing a Blind awards money and opens the shop. Jokers change scoring, Planets level up poker hands, Tarot and Spectral cards modify the deck, Vouchers and Boosters provide longer-term upgrades, and rerolls spend money to refresh the shop.

The core challenge is allocating resources between clearing the current Blind and growing the build over time: which hand to play, when to discard, which Jokers to buy and how to order them, how much cash to keep, whether to skip a Blind, and how to handle Boss rules. Together, these decisions form a complete Balatro strategy.

Bringing Balatro into EvoPolicyGym​

In EvoPolicyGym v0.3.0, we integrated the unofficial Balatro engine Jackdaw as a vendored dependency and used it to implement a Balatro evaluation Environment.

At every step, the Policy receives a public observation that includes:

• the Ante, Blind, target score, and remaining hands and discards;
• the hand, Jokers, Consumables, public deck statistics, and current poker-hand levels;
• the current shop, Boosters, Vouchers, Tags, and cash;
• a strict enumeration of legal Actions for the current phase;
• rule descriptions for visible objects in the pinned engine version.

The Policy returns a semantic Action such as playing, discarding, buying, selling, rerolling, opening a pack, or reordering Jokers. The Environment does not repair an invalid Action; it records a Policy failure. State may persist within one Episode, while every new Episode starts with a fresh Policy instance.

The final Policy score is:

number of Blinds cleared + 1000 × whether the Run was completed

Each cleared Blind contributes one point, and completing the Run adds 1,000 points. This preserves completion as the final objective while giving Policies that have not yet won continuous Feedback through their average progress.

A brief overview of the EvoPolicyGym evaluation process​

During an EvoPolicyGym Run, a Coding Agent starts from an initial Program, repeatedly submits strategies, evaluates them on training Episodes, and continues optimizing from scores and replays. After the Agent finishes, Validation selects the final Program, and Assessment measures it on held-out test Episodes. Every submitted Program, its Feedback and replays, and the optimization record are retained in the Run data.

Experiment​

We compared the packaged baseline with the Programs handed off by optimization Runs from three Coding Agents: Luna, Terra, and Sol. All three Agents started from the same baseline and used the same training Episode pool and Environment interaction budget.

The baseline is a deterministic poker-hand strategy. It enumerates every one-to-five-card combination in the current hand and chooses according to standard poker-hand rank and card values. In the shop, it buys the first affordable Joker; when opening a pack, it selects the first Joker. It does not use discards, Consumables, or rerolls, nor does it manage Joker combinations or the economy.

Luna, Terra, and Sol each received the same training Episode pool and a total Episode budget of 1,024. After optimization, we froze the Program selected by each Agent and evaluated all four Programs on the same 128 held-out test Episodes:

The experiment used Red Deck, White Stake, a Run seed of 20260729, and a 60-second timeout for each Episode.

Results​

Sol was the only model to produce a strategy capable of completing a Run. It cleared 10.45 Blinds on average, 2.83 times the baseline result, and completed 5 of 128 Runs. Luna and Terra also more than doubled the baseline's average progress, but neither completed a Run. Reducing early mistakes can extend a Run; completing one requires a coherent long-term build strategy.

Held-out case index 41 from the same 128-Episode test set. The geometric rendering was generated from the Benchmark's semantic replay and contains no official Balatro game assets.

The baseline's capability boundary​

The packaged baseline enumerates every one-to-five-card combination in the current hand, preferring combinations with a higher standard poker-hand rank and card values. In the shop and when opening packs, it selects the first Joker that meets its conditions.

It answers “which hand ranks higher right now,” but does not connect projected score, discards, Joker combinations, economy, and Boss rules.

What strategy did Sol build?​

Sol built an approximate model of the game state and applied a coordinated set of heuristics for each phase. The final Policy changed in three main ways.

Score estimation and hand planning​

Sol still enumerates one-to-five-card combinations, but evaluates their expected Chips × Mult. The estimate accounts for poker-hand levels, scoring cards, Enhancements, Editions, held-in-hand effects, and Joker combinations.

The Policy combines that estimate with the target score and the remaining hands and discards to decide whether to play immediately or keep searching. It protects cards with value while held in hand and, when no discards remain, can play low-value cards to draw replacements.

Joker construction and economy management​

Sol estimates Joker value, seeks stable combinations of Chips, Mult, and X Mult, and reorders Jokers according to effect dependencies. When all slots are full, it replaces an existing component only if the candidate Joker is clearly stronger.

Shop decisions are also tied to the build. Based on the Ante, available cash, and missing components, the Policy decides whether to buy, preserve interest, or reroll. Cash is used to strengthen later rounds, not just the current shop.

Cross-round state management​

The Policy records shop visits, pack openings, poker hands, and skipped Blinds, then adapts its behavior to Boss rules. Score estimates influence discards, discards affect the probability of clearing the Blind, and the resulting economy changes the next round's build, forming a complete decision loop.

How models use Environment Feedback​

Given the same baseline, training data, and budget, Sol was the only model to complete a Run. It used replay and score Feedback more effectively, organizing local lessons into a complete strategy spanning hands, construction, and economy—more like an experienced player.

Sol's Policy also reveals an engineering problem: about 1,860 lines of logic are concentrated in a single file, creating tight coupling. The next step is to separate the strategy into modules that are easier to test, calibrate, and continue improving.

Next steps​

The first direction is Skills. An effective Skill can provide more than domain knowledge: it can also provide methods for modularization, replay analysis, and testing. Comparing the same Agent and budget with and without a Skill would measure both the final score and the engineering structure of the Policy.

The second direction is cross-Environment RL. The goal is not to memorize one Environment's rules, but to learn how to locate failures, form hypotheses, design experiments, and update strategies—and transfer that process to an unseen Environment.

While integrating different Environments, we also encountered another valuable but underexplored question: can an Agent observe an external system and build an Environment suitable for training and evaluation? For example, can an Agent understand a game's core rules and implement a behaviorally equivalent engine that strips away strategy-irrelevant details such as art and audio, retaining only states, Actions, and Feedback?

Answering this requires measuring whether an Agent can correctly abstract states, Actions, Feedback, and evaluation rules. Today we primarily evaluate how Agents use Environments to optimize strategies; Environment construction itself still lacks systematic measurement. If an Agent can do this well, its Environment can be integrated into EvoPolicyGym and used for further strategy optimization. This complete loop—from observation to modeling to optimization—could help future Agents adapt to new Environments faster and build effective decision systems.

Code and notes​

• EvoPolicyGym Balatro Benchmark
• EvoPolicyGym
• Balatro experiment data

This Benchmark is unaffiliated with LocalThunk, Playstack, or the official Balatro project and contains no official card faces, artwork, music, fonts, or other game assets.

EvoPolicyGym was directly inspired by Jiayi Weng's Learning Beyond Gradients. The article describes Heuristic Learning: a coding agent absorbs rewards, failures, tests, logs, and replays, then improves a programmatic policy by editing the software system itself. It showed us that agentic coding can serve as a learning process whose evolving state is explicit in code.

EvoPolicyGym begins from that insight and asks how to make the process bounded, reproducible, and comparable across interactive Environments.

An EvoPolicyGym Run places a coding agent in the role of a policy-system expert. The agent studies the Environment and its interface, inspects an initial Program, forms hypotheses about successful behavior, and writes those ideas into a complete executable Policy system.

That system may combine domain knowledge, state estimation, rules, planning, search, memory, algorithms, or tuned parameters. The coding agent is free to change its internal design as evidence accumulates. Its responsibility is to turn what it learns into source code that can make decisions on its own.

Authoring and execution occupy two distinct phases. During Evaluation, the submitted Policy independently receives observations and produces Actions. The result is a separable strategy artifact that can be frozen, inspected, rerun, and compared.

Environment Feedback closes the engineering loop:

study the Environment
    ↓
author an executable Policy system
    ↓
submit and evaluate it
    ↓
inspect scores, traces, and artifacts
    ↓
diagnose, redesign, and submit again

This is the motivation for Autonomous Policy Evolution in EvoPolicyGym. The coding agent contributes expertise and software engineering; the Environment contributes empirical evidence; the evolving Program records the resulting strategy. The central question is how effectively an agent can transform limited Environment Feedback into a better executable decision system.

Separate the expert from the Policy​

EvoPolicyGym gives the coding agent and the Policy different roles.

The coding agent is the outer policy engineer and optimizer. It reads instructions and public Feedback, edits workspace/program/, and decides when to submit another candidate. The Policy is the inner decision system. It receives observations through a small ABI and returns Actions while an Episode is running.

A Policy may retain state between act() calls inside one Episode. Each new Episode receives a fresh process and Policy instance, while cross-Episode improvement is represented by a new Program.

EvoPolicyGym locates learning at the Program level between Episodes. Episode-local state supports temporal behavior; Program revision captures lasting improvement. Each change therefore has a visible source snapshot and a clear relationship to its Evaluation evidence.

Make the artifact first-class​

The workspace supports live authoring. Every accepted submission turns its current source tree into an immutable, content-addressed Program. The Evaluation, Feedback, and artifacts belong to that exact snapshot, and the final result returns the retained Program selected from submitted candidates.

This choice makes a Run understandable as a sequence of authored artifacts:

The Program is the durable result. The agent transcript and process logs provide supporting diagnostics.

Let each Benchmark define useful Feedback​

Different Environments expose different kinds of evidence. A control task may benefit from state trajectories and termination causes. A card game may need round summaries, economy decisions, or compact replays.

EvoPolicyGym standardizes the Feedback carrier while each Benchmark defines its useful domain content. Feedback always has a scalar score, and the Benchmark may add bounded public values and artifacts. The Benchmark also owns Episode planning, Environment construction, Action validation, and scoring.

The Kernel owns what must remain consistent across Benchmarks: budgets, immutable submissions, lifecycle ordering, publication, selection, records, and the Policy ABI. Environment packages remain independently installable and depend only on the public authoring interface.

This division lets the project grow like a Gym-style ecosystem while the Kernel remains stable and domain-independent.

Treat evidence access as part of the experiment​

A Run's submission limit, total Episode budget, fixed training-pool size, and optional per-Submission cap define its experimental condition. For example, sixteen submissions, forty-eight Episode units, and a pool of ninety-six Episode identities grant forty-eight total observations selected from a wider set; the larger pool does not increase the interaction budget.

The Host constructs this indexed pool before the agent starts. Each Submission names a non-empty set of public Run-local indices. Reusing an index keeps its hidden Episode specification and Policy seed fixed, so two immutable Programs can be compared on matched evidence. Every use still creates a fresh Environment and Policy runtime and consumes budget again. Actual seeds, scenarios, and pool construction remain Host-owned.

Once an Evaluation begins, its reserved Episode allocation is consumed. Policy failures and invalid Actions are reported as observed behavior, preserving the exact semantics of the submitted Program. Feedback maps every sanitized Episode outcome back to its public index, while comparisons over different selectors remain unmatched evidence.

The agent uses public search Feedback to decide what to try next. When it finishes, authority returns to the Host. Private Validation selects among the handed-off candidates, and held-out Assessment measures the selected Program. Optimization Feedback closes at finish; selection and final measurement remain Host-side.

Keeping search, selection, and final measurement distinct makes the reported result easier to interpret.

Keep the Kernel focused​

EvoPolicyGym is infrastructure with deliberately focused ownership.

• Agent integrations translate a Host-owned task into a provider invocation.
• Benchmark distributions own domain semantics, dependencies, baselines, Feedback, and tests.
• The Kernel owns the shared evaluation and Program-evolution lifecycle.
• The Policy boundary carries bounded public values.

Today Codex is the first supported coding-agent integration, and local process execution is the active backend. The contracts remain provider- and backend-independent, preserving the meaning of Program, Submission, Evaluation, and Run as more integrations arrive.

What EvoPolicyGym enables​

EvoPolicyGym connects scalable interactive Environments, coding agents, versioned Programs, and verifiable Benchmark evidence. The Agent is the subject of study and training; the Program is the executable evidence it leaves behind.

Study agents that evolve strategy systems​

With the Environment, initial Program, interaction budget, Feedback visibility, and selection rules held constant, repeated Runs can study:

• Agent capability: which coding agent authors the strongest Policy system under the same conditions?
• Improvement efficiency: how much Environment interaction produces stable Program improvement?
• Feedback value: which traces, diagnostics, replays, and aggregate signals lead to effective revisions?
• Evolution dynamics: how does Program structure change across Submissions, and which changes produce durable gains?
• Selection validity: does the candidate chosen by Validation retain its advantage in held-out Assessment?
• Policy-system design: which state representations, rules, planners, memories, and controllers do different agents encode?
• Scaling and generalization: how do these results change across budgets, profiles, seeds, task complexity, and Environment families?

The final score measures the Agent's selected artifact. The sequence of immutable Programs, Feedback, artifacts, and outcomes explains how the Agent reached it.

Train coding agents with interactive Environments​

An Environment and Benchmark together form a task generator, evidence generator, and verifier. Profiles, scenarios, and seeds create task variation; Program evaluations, public Feedback, diagnostics, and held-out outcomes provide training signals.

The same Environment ecosystem can support:

• coding-agent RL and RLVR using Program evaluation and held-out performance as verifiable outcomes;
• SFT from successful long-horizon Agent trajectories;
• agent distillation from observable evolution records: task context, public Feedback, Program changes, Submissions, and outcomes;
• rejection sampling of high-quality Agent trajectories based on their final artifacts and results;
• curriculum learning across task profiles and difficulty;
• process supervision from intermediate failures, revisions, and evaluations.

An Agent evolution trajectory spans the full task: understanding the Environment, authoring a Program, reading Feedback, diagnosing behavior, revising the strategy system, submitting candidates, and completing the final handoff. These long-horizon records provide training material for coding agents and policy-engineering agents.

Environment + Benchmark
        │
        ▼
Agent authors and revises a Program
        │
        ├── evolution trajectory ─────▶ Agent SFT / distillation
        └── evaluation outcomes ──────▶ Agent RL / RLVR

The Kernel provides the common task, Evaluation, Run, and evidence contracts. Dataset exporters and training systems can turn retained Agent trajectories into SFT, RL, and distillation data, then return trained agents for held-out measurement. In this way, the Environment catalog is both an Agent Benchmark surface and a scalable source of verifiable long-horizon experience.

Continue reading​

• Core concepts →
• Evaluation and Runs →
• Environment catalog →
• Core16 results →
• Paper ↗