\project

https://tencent.github.io/ContextPilot \sourcecodehttps://github.com/Tencent/ContextPilot \modelhttps://huggingface.co/collections/panzs19/contextpilot \correspondence $\clubsuit$ Equal Contribution; $\heartsuit$ Corresponding Authors.

ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL

Zhuoshi Pan^{1,2 $\clubsuit$} Qizhi Pei^{3 $\clubsuit$} Junru Lu² Honglin Lin³ H. Vicky Zhao^{1 $~\heartsuit$} Di Yin² Xing Sun^{2 $~\heartsuit$} Affiliation: ¹Tsinghua University ²Tencent Youtu Lab ³Shanghai AI Lab

Abstract

Long-horizon agentic tasks require large language models (LLMs) to iteratively retrieve, integrate, and maintain dispersed information across multi-turn interactions, but preserving all interaction histories leads to a continuously growing working context. Recent proactive context management methods allow models to edit their own working context with specialized tools, yet they still face three key limitations: (1) a limited toolset restricted to search, deletion, and summarization, with no support for global planning, long-term memory, and adaptive compression; (2) inefficient exploration that treats context management actions uniformly despite their heterogeneous impacts on final outcomes; and (3) coarse-grained credit assignment that assigns the final trajectory-level reward to all intermediate context editing actions during RL. To bridge these gaps, we introduce ContextPilot, a proactive context management framework for long-horizon agentic reasoning. Our approach systematically augments the toolset with planning, long-term memory, and soft context offloading tools. We further propose an RL method tailored for context management, which uses context and entropy variation to identify critical editing decisions for branch sampling and estimates action-level advantages from all branched trajectories that pass through the corresponding context editing action. Experiments on long-context QA and deep search tasks show that ContextPilot achieves stronger performance with a more compact working context, consistently outperforming existing baselines across various base models and benchmarks.

1 Introduction

Refer to caption — Figure 1: Overview of ContextPilot: (a) Extending the context management tools to support planning, long-term memory, and soft context offloading; (b) Improved RL training algorithm with context-aware partial rollout and (c) fine-grained snapshot-level credit assignment.

Large language models (LLMs) have shown strong reasoning capabilities on information-intensive tasks, including long-context QA [42, 5] and deep search [32, 30]. These tasks require models to identify key evidence from long contexts, integrate complex relations, and, when necessary, seek information through multi-turn tool interactions [7]. Prevailing agentic paradigms like ReAct [38] typically append prior reasoning, tool calls and tool responses to the context as the interaction proceeds, resulting in a rapidly growing context. Prior studies [25, 34, 18, 28] address context growth through human-designed workflows, where truncation or summarization is triggered by fixed rules. Although straightforward, they give the model no control over its own context, and are difficult to adapt to different scenarios [16]. To alleviate this, recent works [13, 39, 16] propose proactive context management, allowing models to spontaneously manage their working context through context editing tools. Despite its flexibility, this paradigm still faces three key limitations.

First, existing context management toolsets, which typically consist of search, deletion, and summarization, are insufficient for managing the context of long-horizon tasks. Effective context management requires more than simply removing or summarizing content: models also need to build long-term memory across scattered fragments [3, 36], maintain structured entity-event episodes [19], and plan globally before taking subsequent actions [38, 15]. Second, although recent studies [41, 16] have applied RL fine-tuning to help models further familiarize with context editing tools, their training procedures are not specifically adapted for context management. Unlike common tools, context editing tools can substantially overwrite the interaction history and exert a larger influence on subsequent steps [25]. As shown in Figure 2, trajectory branches originating from different context management operations exhibit substantially different variance in their final success rates, indicating that certain operations are more sensitive than others. Unfortunately, traditional RL relies on trajectory-level rollouts [4], limiting adaptive exploration for various context management decisions. Third, existing training procedures directly assign the final trajectory-level reward to all intermediate context management actions, neglecting fine-grained credit assignment [11].

To address these challenges, we propose ContextPilot, a proactive context management framework for long-horizon agentic reasoning. Our contributions are threefold: (1) On the tool side, our framework extends the existing context management toolset with planning, long-term memory, and soft context offloading tools, enabling the agent to better control its context across long-horizon interactions. (2) On the training side, we design an RL training method tailored to context management. We propose context-aware partial rollout, which uses context and entropy variation to identify critical context management actions for branch sampling. For credit assignment, we incorporate rewards from all subsequent branches to estimate the advantage of an intermediate context management action, yielding a more fine-grained reward signal. (3) On the evaluation side, whereas prior work mostly focuses on long-context QA, we extend the evaluation to deep search tasks. Experiments on both tasks show that ContextPilot maintains a more compact context while achieving superior performance, bringing consistent gains and surpassing existing baselines.

2 Related Work

2.1 Passive Context Management

To handle growing interaction histories, existing methods typically manage historical context through predefined rules, such as truncating or summarizing messages once the context exceeds a length threshold. Some methods [17, 23, 34, 28] are training-free and rely on fixed workflows or external modules to summarize, compress, or extract key information from historical context. Another line of work [18, 25] incorporates context folding into RL training, enabling models to summarize context and resume reasoning from the compressed state. Although these methods reduce context length, the model remains a passive recipient of rule-managed context rather than an active manager that can decide when and how to manage its working context.

2.2 Agentic Proactive Context Management

Beyond the passive paradigm, recent work equips agents with context management tools, allowing agents to actively decide when and how to edit their working context. MemGPT [22] is an early framework that treats an agent’s memory as virtual memory and manages it through tool calls. Yu et al. [41] and Xu et al. [36] propose memory editing tools, training the agent to control its memory without an external controller. Sculptor [13], StateLM [16], and MemAct [43] provide models with context editing tools such as fragmentation, search, summarization, and deletion, and fine-tune models to use these tools. AgentFold [39] uses SFT to learn multi-scale folding operations that can condense the history. Despite demonstrating the promise of proactive context management, existing methods are still constrained by a narrow set of context management tools, and their training procedures fail to consider the varying impact of these tools. In contrast, ContextPilot improves proactive context management by enriching tool design and refining training paradigms.

3 Preliminary

Before introducing ContextPilot, we first revisit the concepts of context management in agentic reasoning and conduct a pilot study.

3.1 From Passive to Proactive: Context Management in Agentic Reasoning

In prevailing agentic reasoning paradigms like ReAct [38], the agent can perform reasoning and invoke tools to interact with an environment $\mathcal{E}$ . Formally, given the user query $c_{0}=q$ , at the $i$ -th step, the LLM $\pi_{\theta}$ generates thought $t_{i}$ and tool call $a_{i}$ based on the current context $c_{i}$ :

(t_{i},a_{i})\sim\pi_{\theta}(\cdot\mid c_{i}).

(1)

In the next step, the generated thought $t_{i}$ and tool call $a_{i}$ , together with the environment feedback $o_{i}\sim\mathcal{E}(\cdot\mid a_{i})$ , are appended to the context $c_{i+1}=c_{i}\oplus(t_{i},a_{i},o_{i})$ . The model then iteratively continues this cycle until accomplishing the task. As agentic reasoning proceeds, the context grows monotonically. To mitigate the context overload, proactive context management enables models to manipulate their context through specific tools, such as search, retrieval, deletion, and summarization. Specifically, let $\mathcal{A}$ denote the set of available actions, and $\mathcal{A}_{\mathrm{cm}}\subseteq\mathcal{A}$ denote the subset of context management actions. Among them, we distinguish context editing actions as $\mathcal{A}_{\mathrm{ce}}\subseteq\mathcal{A}_{\mathrm{cm}}$ , which directly modify the interaction history. At step $i$ , when the model invokes a context management action $a_{i}^{\mathrm{cm}}\in\mathcal{A}_{\mathrm{cm}}$ , its context is updated by a context transition function $\mathcal{F}$ :

c_{i+1}=\mathcal{F}(c_{i},a_{i}^{\mathrm{cm}})\oplus(t_{i},a_{i},o_{i}).

(2)

Under this paradigm, training solely on the final trajectory is inadequate, as context editing operations (i.e., $a_{i}^{\mathrm{ce}}\in\mathcal{A}_{\mathrm{ce}}$ ) will modify historical messages. To address this, existing studies [13, 16] adopt trajectory snapshots and token-level loss masking. Suppose a trajectory $T$ contains $K$ tool invocations $\{a_{i}\}_{i=1}^{K}$ , of which $M$ are context editing operations. These context editing operations segment the multi-step trajectory into $M+1$ independent trajectory snapshots $\mathcal{S}=\{S_{1},S_{2},\dots,S_{M+1}\}$ . Each snapshot is then treated as an independent training instance, ensuring that every intermediate context state is incorporated in training. Furthermore, token-level loss masking is applied to exclude the loss of outputs that have already appeared in previous trajectory snapshots. This prevents redundant optimization over earlier tool calls and improves training stability and efficiency.

Figure 2: Impact of different tools on final outcomes: for each tool, we branch from it, sample

10

continuation rollouts, and report the standard deviation of their task success rates. Results are evaluated with Qwen3-8B on NovelQA [31].

Correct Answer
Weak Context Management Q: Name the reviews editor of a mid-20th-century university journal, given an article they wrote and their Dec-2023 role. Gold: Robert D. McBain Answer: Robert D. McBain ✓ Tool chain: analyze

\rightarrow

index

\rightarrow

\times 3

\rightarrow

readChunk

\rightarrow

\times 2

\rightarrow

readChunk

\rightarrow\cdots\rightarrow

readChunk

\rightarrow

\times 2

\rightarrow

readChunk

\rightarrow

finish

Wrong Answer
Strong Context Management Q: Find the guitarist who, in 70s high school, founded a band known for conscious multilingual music, and died in 2003. Gold: Frank Leepa Answer: Ty Longley ✗ Tool chain: analyze

\rightarrow

index

\rightarrow

\rightarrow

readChunk

\rightarrow

note

\rightarrow

delete

\times 2

\rightarrow

\times 2

\rightarrow

readChunk

\rightarrow

updateNote

\rightarrow

delete

\times 2

\rightarrow

readNote

\rightarrow

finish

Figure 3: Case study on the mismatch between task correctness and context management quality. Cases are sampled from the traces of StateLM-8B [16] on BrowseComp+ [2]. Questions are condensed to save space.

3.2 Pilot Study

To better understand the impact of context management tools, we branch from context management actions in existing trajectories, sample continuation trajectories, and compute the standard deviation of their final success rates. As shown in Figure 2, this variance differs substantially across actions, suggesting that certain context management actions have a larger impact on the final outcomes and should receive more exploration budget. For credit assignment, Figure 3 shows that final correctness can be misaligned with context management quality: a correct trajectory may rely on repeated retrieval, whereas an incorrect one may still perform reasonable management. Therefore, directly assigning the final trajectory-level reward to all intermediate snapshots may reinforce inefficient context management behaviors and penalize reasonable decisions, motivating fine-grained action-level rewards.

4 Methodology

In this section, we elaborate on the methodology of ContextPilot. We first describe the design of the extended toolset and SFT data construction, followed by the reinforcement learning recipe.

4.1 Context Management Toolset Design

ContextPilot builds upon the basic toolset of StateLM [16] and introduces additional tools for planning, long-term memory and soft context offloading, as detailed in Table 1. To support long-term memory, we introduce memorize, which extracts structured information about entities, timestamps, and event episodes, and builds edges among related memory items. The model can later call readMemory to retrieve a target memory together with its neighbors. For context offloading, we introduce soft-deletion tools, including summarizeContext, compressContext, and foldHistory. foldHistory can condense historical messages into keywords and a summary, which can be recovered by calling searchContext with keyword queries. We treat context offloading, memory-writing, and memory-updating tools as context editing tools, and segment trajectories into snapshots at these operations because they modify the interaction history. Through these extensions, ContextPilot provides a comprehensive toolset for context management, empowering models to effectively manage their working context in long-horizon tasks.

Category	Tool Name	Function Description
Perception & Planning	analyzeText	Calculate context length.
	checkBudget	Check remaining token budget.
	plan	Propose a concise plan.
Information Retrieval	buildIndex	Build a searchable index.
	searchContext	Search for relevant content.
	readChunk	Load a specific context chunk.
	readMultiChunks	Batch load multiple context chunks.
Memory Management	note	Record key information into a note.
	updateNote	Update the content of a note.
	readNote	Load the content of a note.
	memorize	Extract key information and cross-chunk relations into event memory.
	updateMemory	Update a memory item.
	readMemory	Load a specific memory item.
Context Offloading	deleteContext	Replace a message with a placeholder.
	summarizeContext	Replace a message with a summary.
	compressContext	Compress a message via a lightweight compression model, such as llmlingua-2 [24].
	foldHistory	Discard all historical messages and build a searchable index.

Table 1: Context management tools of ContextPilot. Tools highlighted in purple are newly introduced.

4.2 Supervised Fine-Tuning Data Synthesis

Before constructing SFT data, we design a context management harness based on the developed toolset. Through carefully orchestrated rules, the harness provides the teacher model with hints and constraints at appropriate steps. For example, it guides the teacher to inspect retrieved content with readChunk after search. When the context length exceeds a predefined threshold, it restricts the model to context offloading tools for cleaning up. These hints and constraints are used only as generation-time scaffolding and are excluded from the final SFT trajectories. Additional details of SFT data construction are in Appendix C.

4.3 RL Training with Partial Rollout and Fine-Grained Credit Assignment

Context-Aware Partial Rollout.

Inspired by ARPO [4], we introduce partial rollout, which allocates more exploration budget to those critical context management decisions. To identify these critical decisions, we first calculate the context variation:

\Delta C_{t}^{\mathrm{cm}}=\left|\frac{\text{len}(c_{t+1}^{\mathrm{cm}})-\text{len}(c_{t}^{\mathrm{cm}})}{\text{len}(c_{t}^{\mathrm{cm}})}\right|,

(3)

and the entropy variation:

\Delta H_{t}^{\mathrm{cm}}=H_{t}^{\mathrm{cm}}-H_{\mathrm{initial}},

(4)

for each context management operation $a_{t}^{\mathrm{cm}}\in\mathcal{A}_{\mathrm{cm}}$ . $H_{t}$ is the average entropy of $k$ generated tokens after receiving the observation at step $t$ :

\displaystyle H_{t}^{\mathrm{cm}}=\frac{1}{k}\sum_{i=0}^{k-1}\left(-\sum_{j=1}^{V}p_{t+i,j}\log p_{t+i,j}\right),

(5)

where $V$ is the vocabulary size, $p_{t+i,j}$ denotes the probability after softmax and $H_{\mathrm{initial}}$ is the average entropy of the first $k$ tokens at the beginning of the trajectory. We use the initial entropy $H_{\mathrm{initial}}$ instead of the entropy of the preceding step $H_{t-1}^{\mathrm{cm}}$ as the reference, because partial rollout aims to identify critical tool calls that induce significant uncertainty changes relative to the initial query state, instead of local fluctuations between adjacent steps. Based on these metrics, the sensitivity score $\mathcal{S}(a_{t}^{\mathrm{cm}})$ can be defined as:

\mathcal{S}(a_{t}^{\mathrm{cm}})=\alpha\cdot\Delta C_{t}^{\mathrm{cm}}+\beta\cdot\Delta H_{t}^{\mathrm{cm}},

(6)

where $\alpha$ and $\beta$ balance the weights of the two metrics.

During the rollout phase, given a global rollout budget of $N$ snapshots per query, we first perform trajectory-level rollouts and segment the resulting trajectories into $M$ snapshots at context management actions. If $M<N$ , the remaining snapshot budget is allocated to partial rollouts: we rank all context management actions by their sensitivity score $\mathcal{S}(a_{t}^{\mathrm{cm}})$ in descending order and select the top $N-M$ actions as branching points. Starting from the parent node of each selected action, we sample additional sub-trajectories to enrich the snapshot pool. This adaptive partial rollout mechanism facilitates more exploration at critical context management actions that have a larger impact.

Fine-Grained Credit Assignment.

Benefiting from the partial rollout mechanism, we can assign credit to intermediate trajectory snapshots in a more fine-grained manner. Specifically, suppose a complete terminal trajectory $T$ is segmented into $M$ trajectory snapshots $\{S_{1},S_{2},\dots,S_{M}\}$ at context editing operations. Unlike existing works [13, 16] that directly assign the final sparse reward $R(T)$ to all intermediate snapshots, we estimate the value of each snapshot using the reward of all its subsequent branches. For a terminal snapshot $S_{M}=T$ , the reward includes an outcome reward $R_{\mathrm{out}}$ , a format reward $R_{\mathrm{fmt}}$ , and a penalty item $R_{\mathrm{pen}}$ :

R(S_{M})=R_{\mathrm{out}}+R_{\mathrm{fmt}}+R_{\mathrm{pen}}.

(7)

Here, $R_{\mathrm{out}}$ is computed by comparing the model’s predicted answer with the ground truth, while $R_{\mathrm{fmt}}$ checks whether the final output can be successfully parsed. The penalty term $R_{\mathrm{pen}}$ penalizes invalid tool invocations, such as calling readMemory before any memory has been constructed, as well as context-length violations. For an intermediate trajectory snapshot $S_{i}$ , its reward is determined by all terminal trajectories that take $S_{i}$ as a prefix:

R(S_{i})=\frac{1}{|\mathcal{T}(S_{i})|}\sum_{T\in\mathcal{T}(S_{i})}R(T)

(8)

where $\mathcal{T}(S_{i})$ denotes the set of terminal trajectories with $S_{i}$ as a prefix.

After obtaining snapshot-level rewards, we group all trajectory snapshots generated under the same query $q$ as an advantage calculation group $\mathcal{G}=\{S_{t}^{(j)}\mid\forall j,t\}$ . We then compute the advantage of each snapshot sample $S_{t}^{(j)}$ using the group mean and standard deviation:

\hat{A}_{t}^{(j)}=\frac{R(S_{t}^{(j)})-\mathrm{mean}(\{R(S)\mid S\in\mathcal{G}\})}{\mathrm{std}(\{R(S)\mid S\in\mathcal{G}\})}

(9)

Finally, we optimize $\pi_{\theta}$ with the GRPO objective [26], treating each trajectory snapshot as an independent sample. This shifts credit assignment from trajectories to snapshots, enabling more precise reward estimation for intermediate context management actions. Appendix A provides a theoretical discussion of its variance reduction effect.

5 Experiments

5.1 Experiment Setup

Base Models and Training Data.

To evaluate the effectiveness of ContextPilot, we conduct extensive experiments on two long-horizon tasks: long-context QA and deep search. For long-context QA, we follow the data construction setup of StateLM [16]. Specifically, we construct SFT data from the PublicDomain split of NovelQA [31] and the training split of NarrativeQA [9], and perform reinforcement learning on the training set of LongBench-v2 [1]. For model selection, we use Qwen3-8B, Qwen3-14B [37] and Gemma4-E4B-it [8] as base models. For deep search, base models are WebSailor-7B [12] and WebExplorer-8B [14]. Since they already possess basic search capabilities, we skip SFT and directly perform RL training on $1$ K samples drawn from OpenSeeker [6]. Detailed statistics of training data are provided in Appendix Table 6.

Implementation Details.

We conduct SFT and RL training with the verl library [27]. In RL, we limit each complete trajectory to be segmented into at most $8$ trajectory snapshots. For each query, we collect $N=128$ trajectory snapshots, starting from $8$ trajectory-level rollouts, producing at most $M=64$ trajectory snapshots, and completing the remaining budget with partial rollout. During inference, we set the maximum input and generation lengths to $30$ K and $2$ K tokens, respectively. More details are in Appendix D.

Evaluation Benchmarks and Metrics.

We evaluate long-context QA on four benchmarks: the Copyright split of NovelQA [31], the En.MC split of $\infty$ Bench [42], LongMemEval-S [33], and BrowseComp+ [2]. Note that BrowseComp+ is built on a fixed corpus and does not require searching on the Internet. We therefore include it in the long-context QA tasks. The evaluation on deep search tasks includes GAIA (the text-only subset with $103$ examples) [20], BrowseComp [32], BrowseComp-ZH [44], and xBench-DeepSearch [35]. We adopt exact-match evaluation for multiple-choice benchmarks, NovelQA and $\infty$ Bench. Other benchmarks are evaluated by LLM-as-a-Judge.

Baselines.

For long-context QA, we compare ContextPilot with (1) a training-free method: ReadAgent [10], (2) an RL-trained memory agent: MemAgent [40], (3) a proactive context management agent: StateLM [16], and (4) a prompt-only baseline that uses the same tools as ContextPilot but without any fine-tuning, denoted as “w/ tools”. Regarding deep search, we include the following baselines: (1) ReAct (w/ truncation), which truncates early messages when the context exceeds $28$ K tokens, (2) ReSum [34], an inference-time summarization method, (3) SUPO [18], which jointly trains summarization and agentic ability through RL, and (4) OpenSeeker, which performs RL training on the same $1$ K samples as ours but without context management tools. Baseline replication details are in Appendix E.

Table 2: Performance comparison of ContextPilot against baseline methods on long-context QA tasks. We run each method three times and report the mean and standard deviation. Results with ^$\dagger$ are from Liu et al. [16].

Model	Length	NovelQA	$\infty$ Bench	LongMemEval-S	BrowseComp+	Avg.
Qwen3.5-397B-A17B (w/o tools)	256K	88.77	90.39	81.00	62.05	80.55
Qwen3.5-397B-A17B (w/ tools)	32K	91.94	92.13	83.60	80.96	87.16
RL-MemoryAgent-7B^$\dagger$	32K	60.24	62.45	40.60	-	-
RL-MemoryAgent-14B^$\dagger$	32K	78.86	74.24	59.00	-	-
ReadAgent-8B^$\dagger$	32K	16.38	24.02	0.00	-	-
ReadAgent-14B^$\dagger$	32K	23.12	34.06	14.60	-	-
StateLM-8B-RL^$\dagger$	32K	84.15 $\pm$ 1.00	73.07 $\pm$ 1.33	59.73 $\pm$ 2.20	46.44 $\pm$ 0.77	65.85
StateLM-14B-RL^$\dagger$	32K	84.85 $\pm$ 0.42	78.46 $\pm$ 0.67	64.47 $\pm$ 0.50	52.67 $\pm$ 4.00	70.11
Qwen3-8B (w/o tools)	128K	65.74 $\pm$ 0.55	66.96 $\pm$ 1.09	45.20 $\pm$ 1.02	5.82 $\pm$ 0.86	45.93
Qwen3-8B (w/ tools)	32K	38.09 $\pm$ 0.88	39.59 $\pm$ 1.03	24.47 $\pm$ 0.57	8.28 $\pm$ 0.11	27.61
ContextPilot-8B	32K	82.56 $\pm$ 0.49	71.03 $\pm$ 1.44	60.67 $\pm$ 1.91	48.84 $\pm$ 1.48	65.78
ContextPilot-8B-RL	32K	83.88 $\pm$ 0.67	75.25 $\pm$ 0.82	64.27 $\pm$ 1.15	54.18 $\pm$ 1.47	69.40
Qwen3-14B (w/o tools)	128K	78.03 $\pm$ 0.44	74.53 $\pm$ 0.41	54.20 $\pm$ 0.75	6.27 $\pm$ 0.87	53.26
Qwen3-14B (w/ tools)	32K	68.43 $\pm$ 1.22	54.59 $\pm$ 0.94	40.33 $\pm$ 0.50	15.78 $\pm$ 0.10	44.78
ContextPilot-14B	32K	84.28 $\pm$ 0.76	79.04 $\pm$ 0.94	65.93 $\pm$ 1.20	53.13 $\pm$ 1.37	70.60
ContextPilot-14B-RL	32K	84.81 $\pm$ 0.86	81.08 $\pm$ 1.15	67.40 $\pm$ 0.91	55.50 $\pm$ 1.71	72.20
Gemma4-E4B-it (w/o tools)	128K	48.75 $\pm$ 0.53	39.74 $\pm$ 0.87	28.50 $\pm$ 1.90	7.03 $\pm$ 0.30	31.01
Gemma4-E4B-it (w/ tools)	32K	36.90 $\pm$ 2.06	32.75 $\pm$ 2.17	21.80 $\pm$ 1.30	2.73 $\pm$ 0.30	23.55
ContextPilot-E4B	32K	66.80 $\pm$ 0.81	55.02 $\pm$ 1.07	55.07 $\pm$ 1.20	42.05 $\pm$ 1.04	54.74
ContextPilot-E4B-RL	32K	72.92 $\pm$ 0.86	60.99 $\pm$ 1.25	62.47 $\pm$ 1.62	47.47 $\pm$ 1.08	60.96

Table 3: Performance comparison of ContextPilot against baseline methods on deep search tasks. We run each method three times and report the mean and standard deviation.

Backbone	Method	BrowseComp	BrowseComp-ZH	GAIA	xBench-DS	Avg.
		pass@3	pass@3	pass@1	pass@1
WebSailor-7B	ReAct	11.33 $\pm$ 1.03	25.47 $\pm$ 0.98	31.07 $\pm$ 0.79	34.00 $\pm$ 0.82	25.47
	ReAct (w/ truncation)	12.67 $\pm$ 0.62	27.91 $\pm$ 0.91	33.66 $\pm$ 0.92	35.00 $\pm$ 0.82	27.31
	ReSum	15.83 $\pm$ 0.85	38.99 $\pm$ 1.56	38.19 $\pm$ 1.21	35.33 $\pm$ 1.25	32.09
	SUPO	18.50 $\pm$ 1.08	42.68 $\pm$ 0.99	42.07 $\pm$ 1.65	42.00 $\pm$ 1.41	36.31
	OpenSeeker	16.50 $\pm$ 2.12	41.87 $\pm$ 1.02	41.42 $\pm$ 1.21	43.33 $\pm$ 0.94	35.78
	ContextPilot	21.17 $\pm$ 1.31	43.14 $\pm$ 1.07	45.31 $\pm$ 0.92	43.67 $\pm$ 0.94	38.32
WebExplorer-8B	ReAct	23.83 $\pm$ 1.55	46.57 $\pm$ 0.74	48.87 $\pm$ 0.92	52.33 $\pm$ 0.94	42.90
	ReAct (w/ truncation)	24.33 $\pm$ 1.18	45.91 $\pm$ 1.07	49.84 $\pm$ 0.46	52.33 $\pm$ 0.47	43.10
	ReSum	28.83 $\pm$ 1.03	47.87 $\pm$ 1.39	52.75 $\pm$ 1.21	51.00 $\pm$ 1.41	45.11
	SUPO	31.00 $\pm$ 1.78	50.40 $\pm$ 1.07	56.96 $\pm$ 1.21	58.00 $\pm$ 0.82	49.09
	OpenSeeker	29.17 $\pm$ 1.31	48.56 $\pm$ 1.39	57.28 $\pm$ 1.37	56.67 $\pm$ 0.47	47.92
	ContextPilot	32.17 $\pm$ 1.18	53.63 $\pm$ 1.23	57.93 $\pm$ 0.92	56.67 $\pm$ 0.94	50.10

5.2 Main Results

Tables 2 and 3 compare ContextPilot with various baselines on long-context QA and deep search tasks. We highlight three key observations.

(1) ContextPilot achieves the best average performance among comparable-size models. Shown by Table 2, ContextPilot uses only a $32$ K context window, yet outperforms the $128$ K backbone. ContextPilot also surpasses prior RL-trained context management agents. For example, ContextPilot-8B-RL outperforms StateLM-8B-RL by an average of $3.55$ points across four benchmarks; on deep search tasks, ContextPilot also surpasses SUPO by $1.51$ average points across both backbones.

(2) RL training brings more pronounced gains in more complex long-context settings. Starting from the SFT model, RL further improves ContextPilot. For instance, ContextPilot-8B-RL improves over ContextPilot-8B by an average of $3.62$ points across the four long-context QA tasks. The improvement on NovelQA is relatively modest, because the SFT data already includes another split of NovelQA. By contrast, RL yields a substantial $5.34$ point increase on the more challenging BrowseComp+ benchmark. Given that the average input length of NovelQA is about $119$ K tokens whereas BrowseComp+ reaches $552$ K tokens, this contrast suggests that the benefit of RL is larger on more challenging tasks.

(3) The advantages of ContextPilot generalize across tasks and base models. As shown in Tables 2 and 3, ContextPilot brings consistent gains on both long-context QA and deep search. The gains also hold across different backbones, including Qwen3-8B, Qwen3-14B and Gemma4-E4B on long-context QA, as well as WebSailor-7B and WebExplorer-8B on deep search.

5.3 Further Analysis

Figure 4: Token usage per turn on BrowseComp.

Token efficiency analysis.

Beyond performance, we examine whether ContextPilot can maintain a more compact working context. We consider trajectories with at least $15$ turns and compute the average number of input tokens per turn. As shown in Figure 4, the input length of WebExplorer-8B grows almost linearly on BrowseComp, reaching around $30$ K tokens. By contrast, ContextPilot-8B stabilizes its input length each turn at roughly $8$ K– $10$ K tokens.

(a)

\infty

Bench.

(b) NovelQA.

Figure 5: Tool category breakdown during Qwen3-8B RL training on

\infty

Bench and NovelQA.

RL reshapes tool use strategy.

To better understand how RL changes the context management behavior, we track the distribution over tool call categories of ContextPilot-8B-RL in RL training. As shown in Figure 5, the model relies heavily on information retrieval tools in the early stage of RL training, with retrieval accounting for roughly half of all tool calls. As RL training proceeds, the share of information retrieval tools gradually decreases, whereas planning and perception, long-term memory, and context offloading tools exhibit an upward trend. This shift suggests that RL encourages the model to move beyond naive information retrieval toward more coordinated context management.

(a)

\infty

Bench.

(b) NovelQA.

Figure 6: Tool invocation failure rates and task success rate on

\infty

Bench and NovelQA during RL training.

RL synergistically boosts tool-use correctness and task success.

Beyond the tool call frequency breakdown, we further examine whether RL improves the correctness of context management tool use. To quantify this effect, we define a tool invocation as failed when it triggers an environment-side error—such as a malformed call, invalid arguments, or a violation of tool-specific preconditions—and count such failures for each intermediate checkpoint of ContextPilot-8B-RL. Figure 6 reports the failure rates of four representative tool categories together with the task success rate throughout RL training. At the early stage of training, memory and context offloading tools exhibit substantially higher failure rates than information retrieval tools. This pattern suggests that although the SFT model can invoke these tools, it lacks a genuine understanding of their usage. During the RL process, the model’s proficiency with context management tools contributes to the improvement in task success rate.

5.4 Ablation Studies

Table 4: Ablation results of tool design using Qwen3.5-397B-A17B. LME-S and BC+ denote LongMemEval-S and BrowseComp+, respectively.

Tool Design	NovelQA	$\infty$ Bench	LME-S	BC+	Avg.
Original tools	88.90	85.15	74.00	63.49	77.89
+ Planning	89.76	87.23	78.50	65.66	80.29
+ Soft offloading	91.28	89.63	80.20	71.20	83.08
+ Long-term memory	91.94	92.13	83.60	80.96	87.16

Tool design.

To isolate the effect of tool design, we evaluate Qwen3.5-397B-A17B with cumulative tool configurations, progressively adding planning, soft context offloading, and long-term memory tools. As presented in Table 4, the performance improves as more context management tools are introduced, with the full toolset achieving the highest score. The gains are especially clear on BrowseComp+, where accuracy increases from $63.49\%$ to $80.96\%$ with the full toolset, highlighting the effectiveness of the newly introduced tools.

Table 5: Ablation results of the RL algorithm using Qwen3-8B. Results for Gemma4-E4B-it are in Table 8 (Appendix). “+ Entropy.”, “+ Context.”, and “+ Fine-grained.” denote RL training with entropy-based partial rollout, context-aware partial rollout, and fine-grained credit assignment. Numbers in parentheses indicate changes relative to the previous line.

Training method	NovelQA	$\infty$ Bench	LME-S	BC+
SFT	82.56	71.03	60.67	48.84
GRPO	83.53 (+0.97)	72.78 (+1.75)	60.07 (-0.60)	50.96 (+2.12)
+ Entropy.	82.52 (-1.01)	73.07 (+0.29)	62.13 (+2.06)	49.64 (-1.32)
+ Context.	83.05 (+0.53)	73.94 (+0.87)	61.40 (-0.73)	51.08 (+1.44)
+ Fine-grained.	83.88 (+0.83)	75.25 (+1.31)	64.27 (+2.87)	54.18 (+3.10)

RL training design.

We further ablate context-aware partial rollout and fine-grained credit assignment in RL training. As shown in Table 5, entropy-based partial rollout improves several tasks but remains unstable, decreasing BrowseComp+ accuracy by $1.32$ points on Qwen3-8B. This suggests that entropy alone is insufficient for identifying critical context editing operations, while adding context variation yields more stable gains. Fine-grained credit assignment further improves over context-aware partial rollout across all four benchmarks, highlighting the importance of assigning more granular action-level credit rather than relying only on trajectory-level rewards.

6 Conclusion

We present ContextPilot, a proactive context management agent system for agentic reasoning. It extends the context management toolset with planning, long-term memory, and soft context offloading tools, and further improves RL training with context-aware partial rollout and fine-grained credit assignment. Experiments show that ContextPilot maintains a more compact working context while achieving stronger performance, bringing consistent gains on both long-context QA and deep search tasks. These results highlight the importance of proactive context management and point to a promising direction for building stronger agent systems that can self-manage their context throughout long-horizon tasks.

Limitations

While ContextPilot demonstrates strong performance and improved token efficiency, this work has several limitations. First, although our approach extends existing context management tools, the toolset may still not cover all forms of context editing demands. Future work can explore richer operations for organizing, compressing, and retrieving context under different task requirements. Second, due to computational constraints, we do not conduct extensive search over some training hyperparameters. These settings may affect training efficiency and final performance, especially for partial rollout and credit assignment. Finally, our experiments focus mainly on long-context QA and deep search tasks. Extending proactive context management to broader agentic reasoning scenarios, such as agentic coding and GUI agents, remains an important direction for future work.

References

[1] Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. (2025) Longbench v2: towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3639–3664. Cited by: §5.1.
[2] Z. Chen, X. Ma, S. Zhuang, P. Nie, K. Zou, A. Liu, J. Green, K. Patel, R. Meng, M. Su, et al. (2025) Browsecomp-plus: a more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600. Cited by: Figure 3, Figure 3, §5.1.
[3] P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav (2025) Mem0: building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Cited by: §1.
[4] G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, et al. (2025) Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849. Cited by: §1, §4.3.
[5] B. Du, Z. Ye, Z. Wu, M. A. Jankowska, S. Zhu, Q. Ai, Y. Zhou, and Y. Liu (2025) SimVBG: simulating individual values by backstory generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 13104–13133. Cited by: §1.
[6] Y. Du, R. Ye, S. Tang, X. Zhu, Y. Lu, Y. Cai, and S. Chen (2026) Openseeker: democratizing frontier search agents by fully open-sourcing training data. arXiv preprint arXiv:2603.15594. Cited by: Appendix B, §5.1.
[7] J. Feng, S. Huang, X. Qu, G. Zhang, Y. Qin, B. Zhong, C. Jiang, J. Chi, and W. Zhong (2025) Retool: reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536. Cited by: §1.
[8] Google (2026) Welcome Gemma 4: frontier multimodal intelligence on device. Note: https://huggingface.co/blog/gemma4 Cited by: §5.1.
[9] T. Kočiskỳ, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette (2018) The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics 6, pp. 317–328. Cited by: §5.1.
[10] K. Lee, X. Chen, H. Furuta, J. Canny, and I. Fischer (2024) A human-inspired reading agent with gist memory of very long contexts. In International Conference on Machine Learning, pp. 26396–26415. Cited by: §5.1.
[11] J. Li, Y. Wang, Q. Yan, Y. Tian, Z. Xu, H. Song, P. Xu, and L. L. Cheong (2026) Salt: step-level advantage assignment for long-horizon agents via trajectory graph. In Findings of the Association for Computational Linguistics: EACL 2026, pp. 4709–4725. Cited by: §1.
[12] K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, W. Shen, J. Zhang, D. Zhang, X. Wu, Y. Jiang, M. Yan, P. Xie, F. Huang, and J. Zhou (2025) WebSailor: navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592. Cited by: §5.1.
[13] M. Li, L. Xu, Q. Tan, L. Ma, T. Cao, and Y. Liu (2025) Sculptor: empowering llms with cognitive agency via active context management. arXiv preprint arXiv:2508.04664. Cited by: §1, §2.2, §3.1, §4.3.
[14] J. Liu, Y. Li, C. Zhang, J. Li, A. Chen, K. Ji, W. Cheng, Z. Wu, C. Du, Q. Xu, J. Song, Z. Zhu, W. Chen, P. Zhao, and J. He (2025) WebExplorer: explore and evolve for training long-horizon web agents. ArXiv abs/2509.06501. Cited by: §5.1.
[15] S. Liu, S. Dehghan, J. Ganhotra, M. Hirzel, and R. Jabbarvand (2026) From plan to action: how well do agents follow the plan?. arXiv preprint arXiv:2604.12147. Cited by: §1.
[16] X. Liu, T. Liang, D. Ma, D. Zhou, H. Mi, P. He, and Y. Wang (2026) The pensieve paradigm: stateful language models mastering their own context. In The Fourteenth International Conference on Learning Representations, Cited by: Figure 9, Figure 9, Appendix D, §1, §1, §2.2, Figure 3, Figure 3, §3.1, §4.1, §4.3, §5.1, §5.1, Table 2, Table 2.
[17] J. Lu, S. An, M. Lin, G. Pergola, Y. He, D. Yin, X. Sun, and Y. Wu (2023) Memochat: tuning llms to use memos for consistent long-range open-domain conversation. arXiv preprint arXiv:2308.08239. Cited by: §2.1.
[18] M. Lu, W. Sun, W. Du, Z. Ling, X. Yao, K. Liu, and J. Chen (2025) Scaling llm multi-turn rl with end-to-end summarization-based context management. arXiv preprint arXiv:2510.06727. Cited by: Appendix E, §1, §2.1, §5.1.
[19] Z. Lu, D. Li, Y. Shi, B. Wang, L. Wang, and B. Hu (2026) Structured episodic event memory. arXiv preprint arXiv:2601.06411. Cited by: §1.
[20] G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom (2024) GAIA: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, Cited by: §5.1.
[21] OpenAI (2025) Gpt-oss-120b & gpt-oss-20b model card. External Links: 2508.10925, Link Cited by: Appendix C, Appendix D.
[22] C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez (2024) MemGPT: towards llms as operating systems. In The Twelfth International Conference on Learning Representations, Cited by: §2.2.
[23] Z. Pan, Q. Wu, H. Jiang, X. Luo, H. Cheng, D. Li, Y. Yang, C. Lin, H. V. Zhao, L. Qiu, et al. (2025) SeCom: on memory construction and retrieval for personalized conversational agents. In The Thirteenth International Conference on Learning Representations, Cited by: §2.1.
[24] Z. Pan, Q. Wu, H. Jiang, M. Xia, X. Luo, J. Zhang, Q. Lin, V. Rühle, Y. Yang, C. Lin, et al. (2024) Llmlingua-2: data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 963–981. Cited by: Table 1.
[25] J. Shao, Y. Miao, W. Zhang, and B. Luo (2025) FoldAct: efficient and stable context folding for long-horizon search agents. arXiv preprint arXiv:2512.22733. Cited by: §1, §1, §2.1.
[26] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §4.3.
[27] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279–1297. Cited by: Appendix D, §5.1.
[28] J. Su, R. Fang, Y. Li, X. Wang, S. Cai, P. Xie, N. Zhang, and F. Yuan (2026) U-fold: dynamic intent-aware context folding for user-centric agents. arXiv preprint arXiv:2601.18285. Cited by: §1, §2.1.
[29] Q. Team (2026) Qwen3.5: accelerating productivity with native multimodal agents. External Links: Link Cited by: Appendix C.
[30] T. D. Team, B. Li, B. Zhang, D. Zhang, F. Huang, G. Li, G. Chen, H. Yin, J. Wu, J. Zhou, et al. (2025) Tongyi deepresearch technical report. arXiv preprint arXiv:2510.24701. Cited by: §1.
[31] C. Wang, R. Ning, B. Pan, T. Wu, Q. Guo, C. Deng, G. Bao, X. Hu, Z. Zhang, Q. Wang, et al. (2025) NovelQA: benchmarking question answering on documents exceeding 200k tokens. In The Thirteenth International Conference on Learning Representations, Cited by: Figure 2, Figure 2, §5.1, §5.1.
[32] J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese (2025) Browsecomp: a simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516. Cited by: §1, §5.1.
[33] D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu (2025) LongMemEval: benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations, Cited by: §5.1.
[34] X. Wu, K. Li, Y. Zhao, L. Zhang, L. Ou, H. Yin, Z. Zhang, X. Yu, D. Zhang, Y. Jiang, et al. (2025) Resum: unlocking long-horizon search intelligence via context summarization. arXiv preprint arXiv:2509.13313. Cited by: Appendix D, Appendix E, §1, §2.1, §5.1.
[35] Xbench-Team (2025) Xbench-deepsearch. External Links: Link Cited by: §5.1.
[36] W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang (2026) A-mem: agentic memory for llm agents. Advances in Neural Information Processing Systems 38, pp. 17577–17604. Cited by: §1, §2.2.
[37] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5.1.
[38] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, Cited by: §1, §1, §3.1.
[39] R. Ye, Z. Zhang, K. Li, H. Yin, Z. Tao, Y. Zhao, L. Su, L. Zhang, Z. Qiao, X. Wang, et al. (2025) AgentFold: long-horizon web agents with proactive context management. arXiv preprint arXiv:2510.24699. Cited by: §1, §2.2.
[40] H. Yu, T. Chen, J. Feng, J. Chen, W. Dai, Q. Yu, Y. Zhang, W. Ma, J. Liu, M. Wang, and H. Zhou (2025) MemAgent: reshaping long-context llm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.02259. Cited by: §5.1.
[41] Y. Yu, L. Yao, Y. Xie, Q. Tan, J. Feng, Y. Li, and L. Wu (2026) Agentic memory: learning unified long-term and short-term memory management for large language model agents. arXiv preprint arXiv:2601.01885. Cited by: §1, §2.2.
[42] X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. Hao, X. Han, Z. Thai, S. Wang, Z. Liu, et al. (2024) $\infty$ -Bench: extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262–15277. Cited by: §1, §5.1.
[43] Y. Zhang, J. Shu, Y. Ma, X. Lin, S. Wu, and J. Sang (2025) Memory as action: autonomous context curation for long-horizon agentic tasks. arXiv preprint arXiv:2510.12635. Cited by: §2.2.
[44] P. Zhou, B. Leon, X. Ying, C. Zhang, Y. Shao, Q. Ye, D. Chong, Z. Jin, C. Xie, M. Cao, et al. (2025) Browsecomp-zh: benchmarking web browsing ability of large language models in chinese. arXiv preprint arXiv:2504.19314. Cited by: §5.1.

Appendix A Theoretical Discussion of Fine-Grained Credit Assignment

We justify our fine-grained credit assignment as a lower-variance estimator of the conditional continuation value. For a query $q$ , let $T$ be a complete trajectory with terminal reward $R(T)$ , and let $S$ be an intermediate trajectory snapshot induced by a context editing action. The target credit for $S$ is

Q(S)\triangleq\mathbb{E}\!\left[R(T)\mid S\preceq T\right],

(10)

where $S\preceq T$ denotes that $S$ is a prefix snapshot of $T$ .

Trajectory-level credit assignment uses the reward of the single terminal trajectory containing $S$ , while our method averages all sampled terminal continuations that pass through $S$ :

	$\displaystyle\widehat{Q}_{\mathrm{traj}}(S)$	$\displaystyle\triangleq R(T_{1}),$		(11)
	$\displaystyle\widehat{Q}_{\mathrm{ours}}(S)$	$\displaystyle\triangleq\frac{1}{n_{S}}\sum_{k=1}^{n_{S}}R(T_{k}).$		(11)

Here $n_{S}=|\mathcal{T}(S)|$ , and each $T_{k}$ is sampled from the same continuation distribution $p(\cdot\mid S)$ .

Claim.

Conditioned on $S$ , suppose $\{R(T_{k})\}_{k=1}^{n_{S}}$ are independent samples with finite conditional variance

\sigma^{2}(S)\triangleq\operatorname{Var}\!\left[R(T)\mid S\preceq T\right].

(12)

Then both estimators are unbiased:

	$\displaystyle\mathbb{E}\!\left[\widehat{Q}_{\mathrm{traj}}(S)\mid S\right]$	$\displaystyle=Q(S),$		(13)
	$\displaystyle\mathbb{E}\!\left[\widehat{Q}_{\mathrm{ours}}(S)\mid S\right]$	$\displaystyle=Q(S),$		(13)

and their conditional variances satisfy

	$\displaystyle\operatorname{Var}\!\left[\widehat{Q}_{\mathrm{traj}}(S)\mid S\right]$	$\displaystyle=\sigma^{2}(S),$		(14)
	$\displaystyle\operatorname{Var}\!\left[\widehat{Q}_{\mathrm{ours}}(S)\mid S\right]$	$\displaystyle=\frac{\sigma^{2}(S)}{n_{S}}.$		(14)

Therefore, when $n_{S}>1$ and $\sigma^{2}(S)>0$ , our estimator has strictly smaller mean-squared error than trajectory-level credit assignment.

Proof.

Since each continuation is sampled from $p(\cdot\mid S)$ ,

	$\displaystyle\mathbb{E}\!\left[\widehat{Q}_{\mathrm{traj}}(S)\mid S\right]$	$\displaystyle=\mathbb{E}\!\left[R(T_{1})\mid S\right]$		(15)
		$\displaystyle=Q(S),$		(15)

and

	$\displaystyle\mathbb{E}\!\left[\widehat{Q}_{\mathrm{ours}}(S)\mid S\right]$	$\displaystyle=\frac{1}{n_{S}}\sum_{k=1}^{n_{S}}\mathbb{E}\!\left[R(T_{k})\mid S\right]$		(16)
		$\displaystyle=Q(S).$		(16)

Thus both estimators are unbiased.

The trajectory-level estimator uses one continuation, so

\operatorname{Var}\!\left[\widehat{Q}_{\mathrm{traj}}(S)\mid S\right]=\sigma^{2}(S).

(17)

For our estimator, conditional independence gives

		$\displaystyle\operatorname{Var}\!\left[\widehat{Q}_{\mathrm{ours}}(S)\mid S\right]$		(18)
		$\displaystyle=\operatorname{Var}\!\left[\frac{1}{n_{S}}\sum_{k=1}^{n_{S}}R(T_{k})\,\middle\|\,S\right]$
		$\displaystyle=\frac{1}{n_{S}^{2}}\sum_{k=1}^{n_{S}}\operatorname{Var}\!\left[R(T_{k})\mid S\right]$
		$\displaystyle=\frac{\sigma^{2}(S)}{n_{S}}.$

Since both estimators are unbiased, their mean-squared errors equal their variances. The claim follows. ∎

Thus, our method estimates the same target credit $Q(S)$ as trajectory-level credit assignment with lower variance. Since GRPO computes relative advantages from these reward estimates within each query group, this lower-variance estimator provides a more stable advantage signal for optimizing context editing decisions. Context-aware partial rollout strengthens this effect by increasing $n_{S}$ for sensitive context management actions.

Appendix B Training Dataset Statistics

Table 6 summarizes the training data used by ContextPilot across SFT and RL stages, for both experimental scenarios. For SFT, we construct trajectories from the PublicDomain split of NovelQA and the training split of NarrativeQA, and retain trajectories that satisfy the outcome and context management quality requirements. The retained trajectories are further segmented into trajectory snapshots at context editing operations, producing $51{,}469$ snapshots in total. For RL, we use $488$ LongBench-v2 questions for long-context QA and $1{,}000$ questions randomly sampled from the OpenSeeker [6] dataset for deep search.

Appendix C SFT Data Synthesis Details

We synthesize SFT trajectories for long-context QA only. The teacher is Qwen3.5-397B-A17B [29] in thinking mode, decoded with temperature $0.6$ , top- $p$ of $0.95$ , and a maximum output length of $4$ K tokens. To elicit valid context management behavior from the teacher, we use a context management harness that supplies tool definitions together with procedural guidance for managing long interaction histories. Rather than exposing the full toolset at every step, the harness dynamically presents only tools whose preconditions are satisfied by the current state. For instance, readMemory is unavailable before any memory has been written, while readChunk and readMultiChunks are withheld until searchContext has been invoked. The harness also implements correction and retry mechanisms for recoverable generation errors. When the teacher provides an invalid argument, such as a nonexistent message id or an already deleted message id for deleteContext, summarizeContext or compressContext, the harness returns a targeted hint and asks the teacher to retry the tool call. Similarly, if the teacher attempts to answer in plain text without invoking finish, the harness reminds it to submit the answer through the required tool interface. These auxiliary hints serve only as generation-time scaffolding and are excluded from the final trajectories. If a retry is triggered, we keep only the final successful attempt, preventing failed intermediate calls from being imitated.

Stage	Source	Questions	Trajectories	Snapshots
SFT	NovelQA	3,096	2,987	50,691
SFT	NarrativeQA	100	81	778
RL	LongBench-v2	488	–	–
RL	OpenSeeker-v1 (sampled)	1,000	–	–

Table 6: Statistics of the training data, where NovelQA, NarrativeQA, and LongBench-v2 are used for long-context QA and OpenSeeker-v1 is used for deep search.

\iow_now:Ne¨\iow_now:Ne¨You are an AI assistant specialized in long-context document QA. Use the currently available tools to find evidence in the attached document, save useful findings, keep the context compact, and submit the final answer through ‘finish‘.\iow_now:Ne¨\iow_now:Ne¨## Rules\iow_now:Ne¨- Use only tools that are currently available in the API payload. If a tool is mentioned here but not currently available, do not call it.\iow_now:Ne¨- Keep reasoning before tool calls brief. Put durable facts into ‘memorize‘ / ‘updateMemory‘ or ‘note‘ / ‘updateNote‘ instead of long assistant prose.\iow_now:Ne¨- When cleaning context, use exact ‘[msg_id=N]‘ values from previous messages. Do not target the system message, the original user question, or messages already deleted/truncated/summarized/compressed/folded unless ‘restoreContext‘ is available and needed.\iow_now:Ne¨- Final answers must be submitted with ‘finish‘; do not answer in plain text outside the tool call.\iow_now:Ne¨\iow_now:Ne¨## Workflow\iow_now:Ne¨1. Call ‘analyzeText‘ first. If ‘plan‘ is available before indexing, write a concise initial strategy. Then call ‘buildIndex‘.\iow_now:Ne¨2. Use ‘searchEngine‘ for precise keywords, names, dates, titles, rare phrases. If a search returns no useful chunks, try different keywords or queries.\iow_now:Ne¨3. Make a ‘plan‘ after a successful search.\iow_now:Ne¨4. Read evidence with ‘readChunk‘ or ‘readMultiChunks‘ (respect the tool’s chunk-count limit, usually at most 3 ids).\iow_now:Ne¨5. After reading, save useful findings with ‘memorize‘, ‘updateMemory‘, ‘note‘, or ‘updateNote‘.\iow_now:Ne¨6. After saving notes or memories, clean up bulky context whenever cleanup tools are available:\iow_now:Ne¨   - If there are 2 or more uncleaned successful search-result messages, clean search-result tool responses first. Target only the msg_ids of the tool responses of ‘searchEngine‘.\iow_now:Ne¨   - If there are 3 or more uncleaned ‘plan‘ calls, clean old ‘plan‘ assistant messages. Target the assistant msg_ids that invoked ‘plan‘, not the short tool responses.\iow_now:Ne¨   - Then prune the current ‘readChunk‘ / ‘readMultiChunks‘ tool result, and delete the assistant message that invoked ‘memorize‘ / ‘updateMemory‘ / ‘note‘ / ‘updateNote‘.\iow_now:Ne¨   - Use ‘deleteContext‘ for irrelevant content, ‘truncateContext‘ for useful spans, ‘summarizeContext‘ for manual summaries, and ‘compressContext‘ for automatic compression.\iow_now:Ne¨   - Never apply these cleanup tools to the same message twice. If a msg_id has already been deleted, truncated, summarized, or compressed, do not target that msg_id again.\iow_now:Ne¨7. Before finishing, if ‘readNote‘ or ‘loadMemory‘ is available, review the saved notes or memories that are relevant to the answer.\iow_now:Ne¨8. Continue searching, reading, memorizing, cleaning, and reviewing until the evidence is sufficient; then call ‘finish‘ with a concise answer grounded in the document.

Figure 7: System prompt used by ContextPilot for long-context document QA.

\iow_now:Ne¨\iow_now:Ne¨You are a web-search QA agent. Answer the user’s question with evidence gathered through the currently available tools, keep the conversation compact, and submit the final answer through ‘finish‘.\iow_now:Ne¨\iow_now:Ne¨## Rules\iow_now:Ne¨- In search phase, use ‘search‘ for batched web queries, ‘visit‘ to read promising pages, and ‘finish‘ when you have found the answer.\iow_now:Ne¨- If ‘checkBudget‘ shows that the context length is beyond the threshold, call cleanup tools.\iow_now:Ne¨- When cleaning context, use exact ‘[msg_id=N]‘ values. Do not target the system message, the original user question, or messages already deleted/truncated/summarized/compressed/folded.\iow_now:Ne¨- Final answers must be submitted with ‘finish‘; do not answer in plain text outside the tool call.\iow_now:Ne¨\iow_now:Ne¨## Workflow\iow_now:Ne¨1. Use ‘search‘ with concise, complementary query strings to explore names, translations, aliases, dates, clue interpretations, and likely sources.\iow_now:Ne¨2. Use ‘visit‘ on the most promising URLs. Give each visit a precise goal describing what to extract or verify.\iow_now:Ne¨3. Iterate search and visit until you have enough evidence. Keep pre-tool reasoning brief and avoid copying large retrieved text into assistant messages.\iow_now:Ne¨4. If cleanup tools are available, reduce bulky history before continuing:\iow_now:Ne¨   - Prefer large ‘search‘ or ‘visit‘ results that are no longer needed verbatim.\iow_now:Ne¨   - Preserve facts still needed for the final answer: names, aliases, titles, dates, URLs, clue matches, and intermediate conclusions.\iow_now:Ne¨   - Use ‘deleteContext‘ for irrelevant content, ‘truncateContext‘ for useful spans, ‘summarizeContext‘ for concise faithful summaries, ‘compressContext‘ for automatic compression, and ‘foldHistory‘ when available to fold accumulated history.\iow_now:Ne¨5. Continue researching if evidence is incomplete; otherwise call ‘finish‘ with a concise, evidence-grounded answer.

Figure 8: System prompt used by ContextPilot for deep search tasks.

After trajectory generation, we apply a three-stage filtering pipeline. First, we perform outcome-based filtering with exact match. For samples that are initially answered incorrectly, we allow two additional retries and retain the trajectory if any retry yields the correct answer. Second, we use GPT-OSS-120B [21] for process-based filtering, removing trajectories that exhibit improper context management behavior. Finally, we discard trajectories whose peak context length exceeds $32$ K tokens, ensuring that the retained demonstrations remain compatible with the target context budget. This procedure starts from $3{,}196$ questions and retains $3{,}114$ trajectories after outcome-based filtering. Process and peak-token filtering further remove $46$ trajectories, leaving $3{,}068$ qualified trajectories. These trajectories are then segmented at context editing operations, producing $51{,}469$ SFT trajectory snapshots.

Appendix D Training Details

We conduct SFT and RL training with the verl library [27]. During SFT, we use ZeRO-3 parallelism, a global batch size of $128$ , a learning rate of $5\times 10^{-6}$ , a cosine learning-rate scheduler, and a warmup ratio of $0.03$ . For RL, we train for $128$ steps with a rollout batch size of $16$ and set the KL coefficient to $0.001$ . For each query, we first sample $8$ trajectory-level rollouts. Each complete trajectory is segmented into at most $8$ trajectory snapshots, yielding up to $64$ snapshots from the initial rollouts. We collect $128$ snapshots per query, and use partial rollout to complete the remaining snapshots. For the sensitivity score used to select partial-rollout branching points, we set $\alpha=1$ and $\beta=1$ . The maximum input sequence length is $30$ K tokens, and the maximum output length is $2$ K tokens. During inference, a temperature of $0.7$ and top- $p$ of $0.8$ are used for our model. The maximum interaction budget is $60$ turns for long-context QA and $60$ tool calls for deep search following Wu et al. [34]. The system prompts used by ContextPilot for long-context document QA and deep search are shown in Figures 7 and 8, respectively. For LLM-as-a-Judge grading, we use GPT-OSS-120B [21] with the open-ended judging prompt of StateLM [16], as shown in Figure 9.

Figure 9: Prompt template used by the LLM-as-a-Judge for evaluating open-ended questions, following StateLM [16].

Appendix E Baseline Reproduction Details

ReSum

We reproduce the training-free version of ReSum [34] for deep search, using WebSailor-7B and WebExplorer-8B as backbone models. Following our deep search setting, all runs use a $32$ K context window, with $30$ K tokens allocated to the input and $2$ K tokens reserved for generation; summarization is triggered when the context exceeds $25$ K tokens, and the maximum tool call budget is set to $60$ . Since the official ReSum summarization model is not publicly available, we use Qwen3-30B-A3B as the external summarization model. For the summarizer, we adopt the recommended non-thinking decoding configuration of Qwen3, setting temperature to $0.7$ and top- $p$ to $0.8$ . The context summarization prompt and the summary-conditioned continuation prompt follow the original ReSum prompt.

SUPO

We reproduce SUPO [18] using WebSailor-7B and WebExplorer-8B as backbone models. To isolate the effect of the training data, we train SUPO on the same $1$ K OpenSeeker samples as ContextPilot. Following the design of SUPO, summaries are generated by the policy model itself and optimized jointly with tool use actions. The summarization threshold is set to $L=0.95\times 30\mathrm{K}=28.5\mathrm{K}$ tokens, where $30$ K is our maximum input length. Following SUPO, we set the maximum number of interaction steps to $H=100$ and the maximum number of summaries to $S=2$ , and apply overlong masking to rollouts that fail to produce a final answer before reaching either limit. The remaining RL hyperparameters are also the same as SUPO: batch size $B=32$ , group size $G=8$ , learning rate $1\times 10^{-6}$ with a constant learning-rate scheduler, and clipping coefficients $\epsilon_{\mathrm{high}}=0.28$ and $\epsilon_{\mathrm{low}}=0.20$ .

Appendix F Additional Token Efficiency Analysis

Figure 10 extends the token-efficiency analysis to both BrowseComp and BrowseComp-ZH. We consider trajectories with at least $15$ interaction turns and report the average number of input tokens per turn. Across both benchmarks, the WebExplorer-8B agent accumulates increasingly long contexts as the interaction proceeds. In contrast, ContextPilot keeps the working context substantially more compact, indicating that its context management tools reduce redundant history without relying on a larger context window.

(a) BrowseComp.

(b) BrowseComp-ZH.

Figure 10: Token usage per turn on BrowseComp and BrowseComp-ZH.

Table 7: Tool execution failure reason analysis.

Tool	Failure Reason
plan	The predicted summary is empty or unparseable.
searchContext	The search query is empty or malformed, or the model searches before a valid index has been built.
readChunk	The requested chunk id is out of range, deleted, or not associated with the current index.
readMultiChunks	Any requested chunk id is invalid, duplicated, deleted, or exceeds the allowed batch size.
note	The note content is empty, unparseable, or does not specify the value to be stored.
updateNote	The target note does not exist, or the update content is empty or unparseable.
readNote	The requested note does not exist.
memorize	The memory content is empty or unparseable (e.g., inconsistent with the memory storage schema).
updateMemory	The target memory item does not exist, or the update content is empty, unparseable, or inconsistent with the memory schema.
readMemory	The requested memory item does not exist.
deleteContext	The target message index does not exist, has already been deleted, or refers to a protected message (e.g., system prompt or user query).
summarizeContext	The target message does not exist, has already been offloaded, or the generated summary is unparseable.
compressContext	The target message does not exist, has already been offloaded, or specifies an invalid compression ratio.
foldHistory	There is no historical context to fold, or the generated summary and keywords are unparseable.

Table 8: Complete ablation results of the RL algorithm for Qwen3-8B and Gemma4-E4B-it. Numbers in parentheses indicate changes relative to the previous line.

Model	Training method	NovelQA	$\infty$ Bench	LongMemEval-S	BrowseComp+	Avg.
Qwen3-8B	SFT	82.56	71.03	60.67	48.84	65.78
	GRPO	83.53 (+0.97)	72.78 (+1.75)	60.07 (-0.60)	50.96 (+2.12)	66.84 (+1.06)
	+ Entropy-based partial rollout	82.52 (-1.01)	73.07 (+0.29)	62.13 (+2.06)	49.64 (-1.32)	66.84 (+0.00)
	+ Context-aware partial rollout	83.05 (+0.53)	73.94 (+0.87)	61.40 (-0.73)	51.08 (+1.44)	67.37 (+0.53)
	+ Fine-grained credit assignment	83.88 (+0.83)	75.25 (+1.31)	64.27 (+2.87)	54.18 (+3.10)	69.40 (+2.03)
Gemma4-E4B-it	SFT	66.80	55.02	55.07	42.05	54.74
	GRPO	71.12 (+4.32)	57.06 (+2.04)	57.00 (+1.93)	43.41 (+1.36)	57.15 (+2.41)
	+ Entropy-based partial rollout	70.81 (-0.31)	58.22 (+1.16)	60.67 (+3.67)	46.27 (+2.86)	58.99 (+1.85)
	+ Context-aware partial rollout	71.24 (+0.43)	59.68 (+1.46)	60.33 (-0.34)	46.14 (-0.13)	59.35 (+0.36)
	+ Fine-grained credit assignment	72.92 (+1.68)	60.99 (+1.31)	62.47 (+2.14)	47.47 (+1.33)	60.96 (+1.62)

Appendix G Tool Failure Definitions

We consider a tool invocation failed if its execution triggers an environment-side error. Concretely, this includes malformed calls, invalid arguments, and violations of tool-specific preconditions. For retrieval tools, returning no matched evidence is not counted as a failure if the query and index are valid. Table 7 summarizes the specific cases that may lead to tool execution errors in our analysis. Figure 6 reports the execution failure rate for different tool categories on $\infty$ Bench and NovelQA. Across both benchmarks, memory-related and context offloading tools have higher failure rates at the early stage of RL training, indicating that the SFT model can invoke these tools but has not yet fully learned their correct usage conditions. These failure rates decrease substantially during RL training, showing that RL improves the reliability of context management operations while reshaping the model’s tool use strategy.

Appendix H Additional RL Training Ablation

Table 8 reports the complete RL training ablation results on both Qwen3-8B and Gemma4-E4B-it. The results show a consistent trend across the two models: context-aware partial rollout improves over entropy-only branching, and fine-grained credit assignment further strengthens the final average performance.

Appendix I Use of Scientific Artifacts

We cite the original creators of scientific artifacts wherever they are introduced. For models, Qwen3 and Gemma4 are released under the Apache License 2.0. For datasets, OpenSeeker-V1 is released under the MIT License, while the other datasets used in this work are released under the Apache License 2.0. We use all artifacts only for research purposes and in a manner consistent with their intended use in benchmarking, model training, and evaluation. We do not redistribute the original datasets or model weights as part of this work. The artifacts cover two primary task domains: long-context QA and deep search. Language includes English and Chinese. We do not collect new data from human subjects. Because the experiments rely on existing public benchmarks and datasets, we do not perform additional manual anonymization or offensive-content filtering. Relevant data statistics, including the number of questions, retained trajectories, and trajectory snapshots used for training, are reported in Table 6.

Appendix J Use of AI Assistants

We used Codex to assist with code modification, and ChatGPT for writing polishing. All assisted outputs were manually reviewed, verified, and finalized by the authors.