MindGames Arena Generalization Track:
In2AI Solution with Delayed Per-Step Reward Attribution

Consider the following Colonel Blotto exchange:

Beta’s Round 8 allocation, [A0 B10 C10], identical to Round 7, lost that round. Yet this move was strategically brilliant: it established a predictable pattern that Alpha learned to exploit. When Beta broke the pattern in Round 9, Alpha’s counter-strategy backfired.

Problem: Locally suboptimal actions may be globally optimal. A reward function that evaluates each step on its immediate outcome would penalize setup actions that sacrifice short-term performance for long-term success, discouraging strategies such as deception, baiting, or delayed gratification.

Implication: Rewards must incorporate final outcome and propagate backward to preceding actions. Intermediate results alone are insufficient to evaluate action quality.

Consider this Codenames episode where the team words include “ocean” and “fish”:

Spymaster gives clue: [water 2]

Operative reasons: “Options: ocean, shower, blue, fish. With 2 guesses, I’ll try ocean and fish…”

Operative guesses: [ocean] $\checkmark$ Correct! Team word revealed.

Operative reasons: “Ocean was right. Remaining: shower, blue, fish. Since ocean is blue, maybe ‘blue’ is the connection…”

Operative guesses: [blue] $\times$ Assassin! Game over.

Despite being correct, the first guess [ocean] should not receive maximum reward: the turn ended in catastrophe, and all guesses and the clue share responsibility.

Problem: Actions within a logical sequence are interdependent. Early actions depend on future actions for their true value; later actions inherit context from past actions; initiating actions depend on all subsequent responses they trigger. Per-step rewards that evaluate actions independently reinforce locally correct actions even when they contribute to global failure.

Implication: All actions within a logical group must be evaluated together, with rewards computed after observing the complete sequence and attributed based on collective outcome.

Consider the Three-Player Iterated Prisoner’s Dilemma conversation phase, where players speak in fixed order (1 $\rightarrow$ 2 $\rightarrow$ 3) before submitting simultaneous decisions:

• •

  Player 1 (early position): Speaks first, can propose coalitions (e.g., “Let’s both defect against Player 3”), and observes reactions before the decision phase.

• •

  Player 2 (middle position): Can respond to prior proposals and add their own, receiving partial feedback.

• •

  Player 3 (late position): Speaks last. Any proposals they make are never discussed before decisions are locked in.

Problem: Turn order confers asymmetric strategic affordances. Early actors set agendas and observe responses; late actors must interpret prior commitments without any chance to clarify. An agent trained mostly from one position learns position-specific strategies that do not transfer to other positions.

Implication: Training must expose the agent to all positions uniformly to learn position-agnostic strategies. Skewed sampling produces brittle policies that break under position reassignment.

A model that has not yet learned game rules and action formats will produce invalid outputs. If this untrained model immediately faces frontier opponents like GPT-5 (OpenAI, 2025a), every episode ends in rapid defeat before the model can observe what correct play looks like. The training signal becomes dominated by format penalties rather than strategic learning.

Conversely, training exclusively against a narrow set of opponents or teammates produces a model that exploits their specific patterns but fails against diverse real-world agents. In cooperative games like Codenames, a spymaster trained only with one type of operative learns to give clues tailored to that partner’s interpretation style, failing when paired with different teammates at evaluation time.

Problem: The training distribution of opponents and teammates must be designed along two axes. First, skill progression: opponents that are too strong yield no positive signal for a novice agent, while opponents that are too weak fail to challenge an improving one. Second, diversity: training against a homogeneous set of agents produces brittle policies that overfit to specific play styles instead of learning strategies that generalize.

Implication: Agent sampling must provide both curriculum learning and diversity. Training should start with opponents that let the model learn basic rules and formats, then gradually introduce stronger and more varied opponents as skill improves. Earlier opponents must remain in the sampling pool throughout training, both to prevent catastrophic forgetting and to ensure the model continues to beat weaker opponents while learning to compete with stronger ones.

Consider two failure scenarios:

Agentic episodes exhibit high variance in length. A game may terminate early through decisive victory, end abruptly due to an invalid action, extend through prolonged negotiation toward a draw, or reach a predefined step limit. In Codenames, hitting the assassin terminates immediately, while revealing all words through careful play may take substantially longer. Table 1 illustrates this variability across our training data.

This variability compounds with eligibility filtering: not all steps in an episode qualify for training. An episode with 15 steps may yield only 8 eligible training samples after removing invalid actions and steps that lack signal. Rollout generation must collect episodes into an episodes bank, accumulating eligible steps until enough samples are available for a training batch.

Problem: Standard reinforcement learning pipelines assume fixed or predictable episode lengths for batch construction. Variable-length episodes with dynamic eligibility create uneven data flow: some rollouts produce many training samples, others produce few or none. Synchronous collection idles compute while waiting for long episodes; fixed batch sizes waste capacity or train on stale data.

Implication: Rollout generation must run asynchronously across parallel workers and respect the actual number of trainable steps each episode provides when assembling training batches.

As Table 1 illustrates, different games produce episodes of vastly different lengths. When training a single model across multiple games, unbalanced sampling causes the model to see disproportionately more steps from longer-episode games, biasing learned policies toward those environments.

Even within a single game, episode length varies substantially (note the standard deviations in Table 1). An episodes bank containing a mix of long and short episodes may have most of its steps concentrated in a few long trajectories. Sampling uniformly by step would draw disproportionately from these long episodes, reducing diversity: the model repeatedly sees correlated steps from the same trajectory rather than learning from varied situations across many episodes.

Reward distributions are also skewed. Most steps cluster around average rewards, while highly positive (excellent moves) and highly negative (critical errors) steps are rarer. Uniform sampling underrepresents these extremes, yet the model must see failures to avoid them and successes to reinforce them.

Problem: Uniform step sampling produces imbalanced coverage across three dimensions: games, episodes, and reward outcomes. Long-episode games dominate gradients; long individual episodes reduce batch diversity; skewed reward distributions cause the model to underlearn from rare but important successes and failures.

Implication: Batch construction must balance across games, episodes, and reward bins. Steps should be sampled for even game representation, broad episode coverage, and stratified reward sampling that covers bad, average, and good actions proportionally.

Different game situations demand vastly different amounts of reasoning. A spymaster crafting a clue that connects multiple words while avoiding the assassin requires extensive chain-of-thought deliberation, while an operative selecting a single word from a short list needs far less reasoning. Similarly, negotiating a coalition in the Prisoner’s Dilemma requires more deliberation than submitting a binary cooperate/defect decision. These differences appear both across games and across roles within the same game.

When multiple workers run episodes concurrently (as required by Challenge 4.6), these heterogeneous demands create synchronization bottlenecks. Standard inference pipelines process requests in synchronized batches: all requests in a batch must complete before any results are returned. A worker generating a short response cannot proceed while another worker in the same batch deliberates extensively.

Problem: Synchronous batch inference creates idle time proportional to the variance in generation length. If $N$ workers submit requests with generation times $t_{1},\ldots,t_{N}$, synchronous batching forces all workers to wait $\max_{i}(t_{i})$ before any can proceed, wasting $\sum_{i}(\max_{j}(t_{j})-t_{i})$ worker-seconds per inference round. With high variance across games and roles, this overhead compounds multiplicatively across episodes, dominating training time and reducing throughput.

Implication: The inference engine must support asynchronous request handling with continuous batching: requests should be processed as they arrive, with results returned upon individual completion rather than held for batch synchronization. Each worker must be able to proceed independently so that episodes advance at their natural pace.

These eight challenges are not independent obstacles to be addressed in isolation; they form a coupled problem stemming from the mismatch between traditional RL assumptions and the realities of agentic interaction.

The temporal entanglement challenges (Lose to Win, Interdependent Rewards) show that reward computation must be delayed: we cannot assign meaningful rewards until we observe how actions interact with future events. The structural asymmetry challenges (Positional Bias, Opponent/Teammate Diversity, Missing Training Signal) show that not all training configurations are equal: some lack valid signal entirely, others encode position-specific strategies that do not generalize, and still others arise from homogeneous opponent distributions that produce brittle policies. The logistical challenges (Variable Episode Length, Multi-Dimensional Balancing, Heterogeneous Inference) show that episode structure and inference demands are unpredictable: training pipelines must tolerate high variance in episode length, step eligibility, reward distribution, and per-step generation time.

Attempting to solve any single challenge in isolation creates new problems. Delaying rewards without filtering steps assigns arbitrary values to undefined outcomes. Filtering steps without balanced sampling biases the model toward easy cases. Balanced sampling without proper reward attribution trains on noisy gradients. Asynchronous rollout generation without continuous-batching inference introduces synchronization bottlenecks. The challenges compound, and no piecemeal fix suffices; the system must address all three themes jointly.

The TextArena framework (Guertler et al., 2025) does not uniformly terminate episodes on invalid actions: some games continue with default behaviors, others ignore malformed outputs, and error handling varies across environments. There is also no unified interface for determining whether an action was incorrect: each environment reports errors differently, if at all. This inconsistency complicates reward attribution because we cannot reliably determine which steps failed or why.

To address this, we introduce an Action Validator abstraction for each environment. During episode execution, every action passes through the validator, which performs three checks:

• •

  Reasoning template compliance: Does the output follow the expected reasoning structure? (e.g., thinking inside designated tags before the final answer)

• •

  Format compliance: Does the output match the expected action pattern? (e.g., [word N] for spymaster clues, [A5 B10 C5] for Blotto allocations)

• •

  Game-rule validity: Does the action satisfy game constraints? (e.g., allocation sum $\leq 20$ in Blotto, clue not being a substring of board words in Codenames, guess being an existing word on the board)

Invalid actions terminate the episode immediately with a recorded error type. This uniform validation serves several downstream purposes: it provides the metadata needed to identify Missing Training Signal (Challenge 4.5) by detecting when valid actions lack outcomes due to external failures; it establishes clear episode boundaries for delayed reward attribution (Challenges 4.1 and 4.2); and it lets us reliably determine who won, lost, or caused premature termination. The Players Builder (Section 5.2.1) relies on this metadata to extract episode-level results, the Steps Filter uses it to gate steps that depend on failed actions, and the Reward Assigner applies appropriate penalties to invalid outputs.

After episode termination, each completed episode passes through a three-stage processing pipeline (Figure 1) that transforms raw trajectories into training-ready data with per-step reward attribution. The Players Builder extracts episode outcomes for delayed evaluation; the Steps Filter gates steps that lack sufficient signal (Challenge 4.5); and the Reward Assigner computes per-step rewards with backward propagation (Challenges 4.1 and 4.2).

The postprocessing pipeline requires high-throughput episode generation to collect enough training data. We built a training system comprising four subsystems (Figure 2).

We initialized from Qwen3-8B (Team, 2025), a strong open-source base model that fits the 8B-parameter constraint of the Efficient track. The opponent curriculum (Challenge 4.4) proceeded in two phases.

Post-training analysis showed that win rates could be improved substantially by tuning generation parameters (temperature, top-$p$, top-$k$, min-$p$) per environment. We ran a hyperparameter sweep of 300 games per environment for each parameter combination against a mixed pool of opponents. Table 3 shows the configurations evaluated and their win rates.

No single configuration dominated across all environments, confirming the need for environment-specific tuning. Based on these results:

• •

  Codenames: We selected^⋆ temp=0.70, top-$p$=0.80, top-$k$=30, min-$p$=0.05 which achieved the best win rate excluding draws (54%).

• •

  Colonel Blotto: We selected^‡ the least restrictive settings (temp=1.0, top-$p$=1.0, no truncation) which achieved the best win rate including draws (53%) and the second best win rate excluding draws (68%).

• •

  Three-Player Iterated Prisoner’s Dilemma: We selected^† temp=0.60, top-$p$=0.85, top-$k$=30 which achieved the best win rate excluding draws (74%).

Table 4 summarizes the final per-environment generation parameters used for evaluation. Default vLLM values are: temperature $=1.0$, top-$p$ $=1.0$, top-$k$ $=-1$ (consider all tokens), and min-$p$ $=0.0$.