From forward-deployed work to compounding capability

ByteDance Seed and TokenWave · September 1, 2026

Self-Developing Agents

Today’s agents can work. They do not yet reliably compound. The next frontier is an agent that can turn each deployment into validated capability for the next: discovering what matters, learning the right lesson from real work, and carrying forward only the changes that deserve to persist.

Why forward-deployed work compounds

A great deployment does more than solve today’s problem. It makes tomorrow’s deployment better.

Every organization defines good work differently. Forward-deployed engineers close the gap between general capability and local productivity by discovering the real problem, delivering a working solution, learning what was reusable, and folding validated lessons back into the platform. Their leverage comes not only from local delivery, but from turning local work into reusable capability.

Outcome valueAre deployments solving increasingly important problems for the customer?
Product leverageDoes the core system make each new deployment faster, easier, and more reusable?
01
PlatformStart with general-purpose capability
02
DiscoverFind the problem the customer actually needs solved
03
DeliverBuild a working solution in the field
04
LearnExtract the recurring pattern from local experience
05
CompoundFold validated capability back into the system

The loop compounds only when a local fix becomes a validated capability that the next deployment can reuse.

From FDE to self-developing agent

What if an agent could learn from deployments the way a great FDE does?

A self-developing agent should turn deployment experience into validated capability that improves future deployments. Our work suggests this requires solving three distinct problems: deciding what is worth improving, extracting the right lesson from experience, and integrating the right change into the persistent system. Aspire, S³Gym, and HarnessDev study these problems from complementary angles.

01 / TARGET

Find the right thing to improve

Aspire

Operationalize a broad capability goal into sub-goals, data, supervision, and an update strategy—without seeing the final hidden evaluation.

Core questionIs this the right target to optimize?
02 / EXPERIENCE

Learn the right lesson

S³Gym

Test behavior, judge outcomes, and turn raw trajectories into history, compressed memory, or parameter updates that alter future decisions.

Core questionDoes this experience improve future decisions?
03 / SYSTEM

Make the right change persist

HarnessDev

Encode lessons in the persistent execution substrate: tools, context, control flow, recovery, state, and verification.

Core questionDoes this system change generalize?
Experimental evidence · three failure modes

Changing an agent is easy. Knowing what actually improved is hard.

None of these studies runs inside a live customer deployment. Each isolates one step of the loop above in a setting where the ground truth is knowable—because that is the only way to tell a real gain from a local one. Across all three, the same pattern appears in different forms: a system can execute an update without improving the intended capability, accumulate experience without learning a better policy, or make promising local edits that fail to transfer. The panels below show where apparent self-improvement breaks down.

Aspire · five decision-agent configurations × six capability goals

Choose what to learn

1 retained gain / 30 cellsOnly one cell produced a gain that cleared the retention threshold
Almost every run produced a checkpoint. Almost none produced a better model.Each run starts from a broad capability goal; later stages apply completion and baseline checks
0102030 30 28 21 1 AllcellsEvaluatedcheckpointEligiblecheckpointRetainedimprovement
Three improving checkpoints—all worse than the baseQwen3.5-4B Self · hidden-evaluation mathematics score by checkpoint
Self-trained checkpointsUnevolved baseline
05101520CK1CK2CK3 Unevolved base 17.86 0.791.752.38 Best CK: −15.48 vs base Local gain, global regression
Agents can optimize a trajectory without improving the capability. Across 30 configuration–goal cells, 28 produced an evaluated checkpoint. Two found a checkpoint above the base score, but only one cleared the retention threshold and survived rollback.Adaptive-feedback protocol · hidden evaluation set
Hover, click, or use ← → to move between studies.
Experimental evidence · what actually changes?

The hard part is not producing change. It is deciding what should carry forward.

The evidence above shows which candidate changes survive evaluation. The case studies below explain why so many fail. Across all three, the same question recurs: which local signal deserves to shape the agent’s next starting point?

01 / TARGET FORMATIONAspire

A vague goal changes the search before it changes the model.

Vague goals move the bottleneck upstream. Agents spend more of the fixed budget interpreting the target and choosing a proxy—and less on active training and evaluation. The result is more operationalization, not evidence of a better objective: an agent can execute an update without knowing whether it is optimizing the intended capability.

Evidence boundary. These are descriptive differences across 48 matched Opus 4.8 run pairs, not an isolated prompt-only causal effect; other run metadata may differ. Behavioral densities are normalized per 10,000 trace characters and show where attention shifts, not which choice causes the final score gap.

More deliberation, less active GPU work

Vague-goal − explicit-task · minutes per matched pair Decision thinking GPU idle Active training / eval +35 +37 −76 −600+30

Vague goals intensify target operationalization

Trace-density ratio vs explicit-task · per 10,000 characters Read task definitions Inspect eval scripts LoRA / PEFT activity 2.98×2.39×3.09× 01×2×3× LoRA use: 24.1% → 89.8%
02 / EXPERIENCE INTEGRATIONS³Gym

Experience helps only when the update pathway preserves useful behavior.

No experience representation wins universally—and which one wins depends on how the score is normalized. Under baseline-normalized NABA, Summary Memory leads in Nullify, Tetris, and Trust; History ICL leads in Minesweeper, Snake, and PvZ; Chess splits 3–3–1. Under the raw AUC+ shown in the panel above, the same runs place the tie on Tetris instead of Chess. That a single normalization choice reorders the result is itself the finding: at this effect size, no pathway is separated cleanly enough to promote. Parameter updates are less predictable still: Trust improves across most checkpoints, while PvZ drops from 23 to 6 and stays there. Self-judgment barely predicts the next gain (ρ = −0.010 and −0.018).

Evidence boundary. The left chart uses NABA = AUC+ / max(y₀, ε) across 49 matched model–game pairs, with ε set by the game’s first non-zero score scale when y₀ = 0; the leaderboard uses raw AUC+, so its win counts can differ. Parameter-training results come from a separate 20-checkpoint Qwen3-8B study and are not matched against the context-level methods; persistent degradation is observed, but its cause is not identified.

Which memory pathway wins more model comparisons?

Model-level ΔNABA wins · seven paired models per game ChessMinesNullifyTetrisSnakePvZTrust 33 34 42 51 34 34 43 111 SummaryHistoryTie

The same training pathway produces opposite trajectories

Qwen3-8B · strict-mode score across 20 checkpoints Trust18 / 19 above baseline · max 30 PvZ19 / 19 updates remain at 6 300 300 236AUC+ 163.5 05101519
03 / SYSTEM INTEGRATIONHarnessDev

Editing a harness is easier than integrating a working mechanism.

All 18 Code harnesses can execute tasks, but state rarely becomes an operating mechanism. Of 108 component instances, 18 were never observed in formal runs—and all 18 were state or memory. Eleven artifacts define a State class, yet only one exposes state saving, only one implements periodic checkpointing, and no checkpoint event occurs across 26,679 task trajectories. In a separate evolution study, agents again modify execution and tools far more often than state.

Evidence boundary. Creation evidence covers 18 Code artifacts; observed triggering establishes execution, not causal benefit. Evolution categories overlap and describe 64 official switches across nine single trajectories. Post-freeze held-out evaluation covers SWE-Pro only.

Agents edit execution far more than state or verification

64 official switches · overlapping edit categories Execution / control Tools Lifecycle recovery Context State Standalone verifier 583717 1640 016324864

State appears in code—but disappears at runtime

Artifact design → observed execution ARTIFACT DESIGN · 18 CODE HARNESSES State class defined State-save interface Periodic checkpointing 11 / 181 / 181 / 18 OBSERVED RUNTIME 0 checkpoint eventsacross 26,679 task trajectories 18 / 108 components never observed · all state or memory

Training is not improvement, remembering is not learning, and editing the system is not improving it. What separates each pair is a check the agent cannot pass on its own: an evaluation it did not choose, a decision it has not yet made, and a runtime it does not control.

What deserves to compound?

Our results do not say that self-development fails. They say that the checks most agents apply to themselves are too weak to tell a real gain from a local one. Three checks did the separating work in every study, and each one is something the agent cannot supply for itself.

01

Anchor the target to an evaluation the agent cannot see.

Given a vague goal, agents reliably produce something: 21 of 30 cells reached an eligible checkpoint. Only one produced a retained gain under the protocol. Vague goals also change how effort is spent: decision time rises while active training and evaluation fall, even as LoRA use jumps from 24.1% to 89.8%. That is evidence of a different search process, not yet a better target. Treat any gain measured on a target the agent chose as unproven.

02

Validate a lesson on the next decision, not on the agent’s own verdict.

Self-judgment carries almost no signal about the next gain (ρ = −0.010 and −0.018), and no experience representation wins across environments—a single change of normalization is enough to reorder which one leads. So there is no default to ship: the pathway has to be selected per environment and confirmed on held-out decisions, not on the trace that produced it.

03

Require a frozen held-out and an executor swap before a change persists.

Visible feedback and held-out scores agree in only 34 of 64 version switches, and 2 of 9 declared final versions are held-out optimal—roughly what a coin flip would give. Swapping in a fixed Gemini executor turns three of four lineages negative. And a mechanism that never fires is not a mechanism: 26,679 trajectories produced zero checkpoint events. Persistence therefore needs integration, versioning, and rollback—not another edit.

A change deserves to compound when all three hold at once: it was aimed at a target the agent could not game, it improved the next decision rather than the current trace, and it survived a runtime the agent did not control. Almost nothing we measured cleared all three. Closing that gap—not expanding the capacity for self-modification—is the work.

Deployment is where self-improvement becomes useful.

Abstract self-improvement asks whether an AI system can make another AI system stronger. Deployment-grounded improvement asks a more consequential question: can an agent become more useful to the organization it serves, using evidence from real work—and can that learning improve the next deployment rather than remain a one-off fix?

The target is not simply a higher public benchmark score. It is a better fit to the organization's changing goals, workflows, data, and quality standards—without turning every local workaround into permanent complexity.

The goal is not an agent that changes constantly. It is an agent that knows which changes should compound.

Our research goal

We want to study agents that can grow into a role, not only complete a task.

That does not mean replacing an FDE end to end. It means giving an agent a role-level objective and asking whether it can discover what the organization means by good work, build tests and learning signals from imperfect feedback, and reorganize models, memory, and harness as evidence accumulates. In that sense, the long-term goal is FDE-like: not merely solving local problems, but turning repeated field work into compounding organizational capability.

Give the agent a platform, a role, and a customer environment. Each deployment should solve something useful now, create evidence about what mattered, and leave the agent—and the platform—better prepared for what comes next.

What this program leaves behind

A negative result matters when the apparatus outlives it. Each study leaves a reusable test of self-improvement: Aspire pairs sealed evaluation with auditable trajectories; S³Gym separates permissive exploration from strict held-out evaluation and grounds self-judgment in executable verifiers; HarnessDev freezes runnable artifacts, then tests post-freeze held-out generalization and transfer across executors.

Aspire

A benchmark for vague-goal-driven self-evolution: six capability goals, a sealed expert-authored evaluation of 520 items, and a minimal interactive environment supporting both weight and harness updates, plus 48 matched vague-goal/explicit-task run pairs and auditable trajectories.

S³Gym

An interactive benchmark that decomposes experience-driven self-improvement into Self-Testing, Self-Judging, and Self-Improvement across seven verifier-backed games. It compares raw-history ICL, summary memory, and parameter training, with seven complete context-level models and a separate 20-checkpoint Qwen3-8B training study.

HarnessDev

A two-stage benchmark for creating and evolving runnable agent harnesses. Creation covers six creator LLMs, four domains, five downstream benchmarks, and 2,207 unique instances, including 18 Code artifacts and 26,679 recorded task trajectories; Evolution records 73 official versions across nine trajectories, with a post-freeze SWE-Pro-630 held-out split and fixed-executor ablations.