Languages: English | 中文

Paper | Environment | Data

Fully open-sourced resources

• DeepSearch-World environment: an offline, verifiable Wikipedia search / visit tool environment is available on Hugging Face.
• 420K DeepSearch-World training data + DeepSearch-Val: the full data release is available on Hugging Face.

DeepSearch-Val is included in the test split of the DeepSearch-World data repository. The 420K data pool and the verifiable DeepSearch-World environment are both fully open sourced.

This repository contains the verifiable virtual tool environment DeepSearch-World and the self-distillation training code DeepSearch-Evolve for Deep Search Agents.

DeepSearch-World provides a deterministic and verifiable offline Wikipedia tool environment with reproducible search / page-reading tools, a multi-hop QA task pool, and trajectory generation / verification pipelines for training Deep Search Agents. DeepSearch-Evolve connects scaffold teacher rollouts, Rejection Sampling, trajectory quality filtering, data mixing, Scaffold-to-ReAct conversion, and LlamaFactory SFT into an iterative loop: the current model generates scaffold trajectories in the environment, the system keeps verified successful trajectories and converts them into ReAct-format supervision, then trains the next-round model.

• Deep Search Agent scaffold: calls a local vLLM model through an OpenAI-compatible API; supports planning, tool use, Progress State updates, grounded reflection, failure recovery, and final answers.
• DeepSearch-World Wikipedia environment: web_search_wiki uses a Pyserini/Lucene BM25 index to retrieve candidate pages; visit_wiki reads page text through a SQLite offset database plus JSONL corpus and returns reproducible observations.
• DeepSearch-Evolve self-distillation loop: scaffold trajectory generation, Rejection Sampling, trajectory quality filtering, data mixing, Scaffold-to-ReAct conversion, LlamaFactory SFT, validation, and checkpoint hot-swapping.
• Asynchronous generation and training: generation nodes and training nodes coordinate through JSON signal files on a shared filesystem, so training does not block the next generation batch.
• Multi-hop QA data construction: prepare_wiki_data_env/ builds DeepSearch-World tasks from Wikipedia pages and simple QA seeds. The paper uses a 420K multi-hop QA pool and reserves DeepSearch-Val for validation.
• Direct-inference baseline: traj/direct_infer/ provides ordinary QA inference without the agent tool loop for controlled comparisons.

Input questions
  |
  v
Scaffold teacher trajectory generation
  - Plan / Action / Reflect / State Update / End
  - web_search_wiki / visit_wiki tool calls
  - Progress State, grounded reflection, failure recovery
  |
  v
Rejection Sampling
  - LLM Judge checks whether the final answer is correct
  - trajectory quality filtering removes weak evidence, weak alignment, and inconsistent reasoning
  |
  v
Scaffold-to-ReAct conversion
  - convert correct scaffold trajectories to ReAct-format supervision / ShareGPT implementation format
  - mix correct data from multiple rounds with decay-based importance sampling
  |
  v
LlamaFactory Full SFT
  |
  v
New checkpoint
  |
  +----> next-round agent trajectory generation

virtualtools/
├── README.md
├── README_zh.md
├── .env.example                       # local path, model, and service configuration template
├── config/env.sh                      # shared environment loader for shell entry points
├── SECURITY.md                        # sensitive information and vulnerability reporting notes
├── requirements_traj.txt              # reference dependencies for trajectory generation / retrieval / evaluation
├── requirements_sft.txt               # reference dependencies for SFT training
├── traj/                              # main DeepSearch-Evolve project
│   ├── scripts/                       # orchestration, training, validation, and vLLM launch scripts
│   │   ├── orchestrate_gen_9b.sh      # TP-mode DeepSearch-Evolve loop
│   │   ├── orchestrate_gen_9b_dp.sh   # DP mode: one TP=1 vLLM instance per GPU
│   │   ├── orchestrate_gen_27b.sh     # 27B model variant
│   │   ├── train_daemon_9b.sh         # training daemon that consumes train requests
│   │   ├── train.sh                   # one-shot LlamaFactory SFT
│   │   ├── train_config.yaml          # LlamaFactory training config template
│   │   ├── run_validation.sh          # checkpoint validation entry point
│   │   └── start_summary_server.sh    # optional summary vLLM service
│   ├── traj_generation/               # core Deep Search Agent scaffold trajectory generation
│   │   ├── agent_eval.py              # main entry: read questions and generate trajectories
│   │   ├── agent_eval_val.py          # validation-set variant
│   │   ├── agent_eval_val_real.py     # real-web-tool validation variant
│   │   └── agent/llm_agent/
│   │       ├── prompt.py              # scaffold teacher prompt
│   │       └── wiki_tool_call.py      # tool-dispatch wrapper
│   ├── agent_utils/                   # wiki-agent package: LLM, tools, memory, function-call protocol
│   │   └── wiki_agent/
│   │       ├── tools/
│   │       │   ├── text_search_wiki.py
│   │       │   ├── visit_wiki.py
│   │       │   ├── web_search_real.py
│   │       │   └── tool/
│   │       └── llm/
│   ├── evaluation/                    # Rejection Sampling and evaluation scripts
│   │   ├── rejection_sampling.py
│   │   ├── evaluate_hle_official.py
│   │   └── start_judge_vllm.sh
│   ├── data_prepare/                  # trajectory cleaning, mixing, and conversion
│   │   ├── traj_to_sharegpt.py
│   │   ├── importance_sampling_mix.py
│   │   └── strip_trajectories.py
│   ├── direct_infer/                  # direct QA inference baseline
│   └── LlamaFactory/                  # SFT backend submodule
├── prepare_wiki_data_env/             # Wikipedia multi-hop QA construction
│   └── gxy/
│       ├── pipeline.py
│       ├── NlpAgenticQaListV3OP.py
│       ├── Selector.py
│       ├── Generator.py
│       └── wiki/
├── vlm/                               # optional VLM / real-search GRPO export package notes
├── scripts/open_source_check.sh        # local release-sanity checker
└── scripts/export_open_source_tree.sh  # export a clean source tree for release

traj/traj_generation/agent_eval.py is the main entry point for scaffold teacher rollouts. It reads JSONL / Parquet questions, constructs the scaffold prompt, calls an OpenAI-compatible model endpoint, executes tool calls, and writes full trajectories.

Important runtime configuration:

The paper describes reproducible search / visit tools. In code, the corresponding tool names are web_search_wiki and visit_wiki.

web_search_wiki is implemented in traj/agent_utils/wiki_agent/tools/text_search_wiki.py. It depends on the Pyserini/Lucene BM25 index at WIKI_INDEX_PATH and returns keyword, caption, and candidate URLs.

visit_wiki is implemented in traj/agent_utils/wiki_agent/tools/visit_wiki.py. It depends on:

• WIKI_DB_PATH: SQLite offset database with keyword and offset fields.
• WIKI_JSONL_PATH: Wikipedia page-text JSONL corpus.
• SUMMERY_MODEL_PATH / SUMMARY_API_BASE: optional summary model; when disabled, the tool returns truncated raw page text.

traj/evaluation/rejection_sampling.py uses an external Judge model to check whether gen matches the gold answer.

• Input: JSONL output from agent_eval.py.
• Output: correct.jsonl and incorrect.jsonl.
• Optional: --prompt_filter applies an additional tool-use quality filter on correct samples.

The Judge endpoint is configured through environment variables. It can be an internal API or an OpenAI-compatible vLLM service launched by start_judge_vllm.sh.

traj/data_prepare/traj_to_sharegpt.py implements the paper's Scaffold-to-ReAct conversion. It converts correct scaffold trajectories into ReAct-format supervision and writes the LlamaFactory ShareGPT implementation format. The conversion includes several cleanup steps:

• Inject writable Progress State fields into assistant <think> blocks.
• Remove read-only state fields to avoid training-time information leakage.
• Merge multiple system prompts.
• Remove environment-injected [REFLECTION] blocks and duplicated task prompts; rewrite them as first-person reasoning when needed.
• Fix reasoning text leaked outside <think>.
• Merge consecutive messages with the same role to satisfy LlamaFactory's user/assistant alternation constraints.

traj/scripts/train.sh runs full SFT through traj/LlamaFactory/src/train.py. The config template is traj/scripts/train_config.yaml. Main defaults include:

• stage: sft
• finetuning_type: full
• template: qwen3_5_nothink
• cutoff_len: 32768
• DeepSpeed ZeRO-3
• bf16, Flash Attention 2, gradient checkpointing, Liger Kernel

train_daemon_9b.sh polls train_request_*.json in the signal directory, launches training, and writes train_done_*.json when training finishes.

prepare_wiki_data_env/gxy/ constructs the Wikipedia multi-hop QA task pool used by DeepSearch-World:

1. Read url, query, and answer from input Parquet samples.
2. NlpAgenticQaListV3OP performs BFS deep search from the seed page and builds a Wikipedia knowledge tree.
3. Selector.py selects relevant links and extracts features.
4. Generator.py generates complex multi-hop questions from entities and relations.
5. Output completed_tree, url2feature, and multiple child_tree_* fields.

vlm/ was extracted from vlm_code_export.zip. It is an auxiliary experiment package, not the default entry point for the DeepSearch-Evolve pipeline. The open-source boundary currently keeps only the export notes, requirements, and filtering scripts; data, historical outputs, model patches, and nested vlm/virtualtools/ snapshots are ignored by .gitignore.

• vlm/README.md: English notes for the export package.
• vlm/filter.sh, vlm/filter_bc_vl_1w.sh: helper scripts for filtering VLM export data.

If you only run the DeepSearch-World / DeepSearch-Evolve flow in this README, you can ignore vlm/. If you plan to release the full vlm/ experiment package, audit its nested snapshots, experiment outputs, and data files separately.

This repository uses LlamaFactory as a Git submodule. If you are using the historical repository, initialize submodules first:

git submodule update --init --recursive

If you use a source tree exported by scripts/export_open_source_tree.sh, the export script copies the initialized LlamaFactory files into the release tree, so .git submodule metadata is not required.

Two separate environments are recommended:

# Trajectory generation, retrieval, and evaluation
conda create -n virtualtools-traj python=3.11 -y
conda activate virtualtools-traj
pip install -r requirements_traj.txt
cd traj
pip install -e agent_utils

# Training
conda create -n virtualtools-sft python=3.11 -y
conda activate virtualtools-sft
pip install -r requirements_sft.txt

The requirement files are exported from the experiment environment and pin many versions. Some packages may be platform-specific. If installation fails on a new machine, start from the core dependencies below and adapt versions to your platform:

• Trajectory environment: torch, transformers, vllm, openai, pyserini, pandas, tqdm, requests.
• Training environment: torch, deepspeed, transformers, datasets, accelerate, peft, flash-attn, liger-kernel.
• Data construction: vllm, transformers, pandas, pyarrow, mwparserfromhell, ray, requests.
• Optional GRPO / verl path: if you use traj/traj_generation/agent/llm_agent/generation.py or the verl dynamic tools under traj/agent_utils/wiki_agent/tools/tool/, install a compatible verl. The default OpenAI-compatible agent_eval.py + Rejection Sampling + LlamaFactory SFT loop does not depend on the top-level vendored verl/.

Pyserini requires Java. The scripts try to use the JVM inside the conda environment by default, but you can also set it manually:

export JAVA_HOME=/path/to/java
export JVM_PATH=$JAVA_HOME/lib/server/libjvm.so
export PATH=$JAVA_HOME/bin:$PATH

The repository root provides .env.example and config/env.sh. Common shell entry points automatically load .env from the repository root, so start with:

cp .env.example .env
# Edit PROJECT_ROOT, DATA_MODEL_DIR, TRAJ_ENV, SFT_ENV, model paths, and data paths for your machine.

Do not commit .env; it usually contains local paths, private service endpoints, or API keys.

Place data and artifacts under a consistent directory first. The layout below is only a recommendation; the actual paths are controlled by .env and command-line arguments.

$DATA_MODEL_DIR/
├── pretrain_model/
│   └── qwen35_9b/                     # base model or previous-round checkpoint
├── data/data_verl_cleaned/
│   ├── train.jsonl                    # DeepSearch-World training pool shard
│   └── test.jsonl                     # DeepSearch-Val / validation input
├── tool/
│   ├── web_search_index/              # Pyserini BM25 index for web_search_wiki
│   └── visit_sqlite/
│       ├── visit_offsets.sqlite       # keyword -> JSONL byte offset
│       └── wikiextractor_v1.jsonl     # Wikipedia page-text corpus
├── buffer_9b/                         # intermediate trajectories, RS results, ShareGPT data
└── train_results_9b/                  # SFT checkpoint outputs

The Hugging Face data repository provides the 420K DeepSearch-World data pool and DeepSearch-Val. Use the test split as DeepSearch-Val. The environment repository provides the offline Wikipedia resources required by the search / visit tools.

Minimal train.jsonl / test.jsonl format, one JSON object per line:

{"id": "demo-1", "question": "Were Scott Derrickson and Ed Wood of the same nationality?", "answer": "yes", "answers": ["yes"]}

Copy and edit .env:

cd /path/to/virtualtools
cp .env.example .env

Minimal .env example:

PROJECT_ROOT=/path/to/virtualtools/traj
DATA_MODEL_DIR=/data/virtualtools
TRAJ_ENV=/path/to/envs/virtualtools-traj
SFT_ENV=/path/to/envs/virtualtools-sft

INIT_MODEL=/data/virtualtools/pretrain_model/qwen35_9b
SUMMARY_MODEL=/data/virtualtools/pretrain_model/qwen35_9b
DATA_PATH=/data/virtualtools/data/data_verl_cleaned/train.jsonl
VAL_DATA=/data/virtualtools/data/data_verl_cleaned/test.jsonl
BUFFER_DIR=/data/virtualtools/buffer_9b
TRAIN_SAVE_ROOT=/data/virtualtools/train_results_9b
SIGNALS_DIR=/data/virtualtools/buffer_9b/signals_9b

WIKI_INDEX_PATH=/data/virtualtools/tool/web_search_index
WIKI_DB_PATH=/data/virtualtools/tool/visit_sqlite/visit_offsets.sqlite
WIKI_JSONL_PATH=/data/virtualtools/tool/visit_sqlite/wikiextractor_v1.jsonl

STEAM_HOST=http://127.0.0.1:9010
STEAM_MODEL_MARKER=/path/to/judge-model-or-served-name
STEAM_APP_ID=
STEAM_APP_KEY=

Common entry points:

If the model, wiki index/corpus, and Judge are ready, the fastest complete DeepSearch-Evolve debug run uses two terminals. The command below uses --rounds 1, --batch_size 1000, and --correct_threshold 100 for smoke testing. The paper's main experiment uses 15 evolving rounds, generates 10,000 instances per round, and triggers training after 4,000 trajectories pass Rejection Sampling and trajectory quality filtering.

# Terminal 1: training daemon
cd /path/to/virtualtools/traj
bash scripts/train_daemon_9b.sh

# Terminal 2: generation + RS + ShareGPT conversion + train request
cd /path/to/virtualtools/traj
bash scripts/orchestrate_gen_9b_dp.sh \
  --rounds 1 \
  --batch_size 1000 \
  --correct_threshold 100 \
  --init_model "$INIT_MODEL" \
  --data_path "$DATA_PATH" \
  --val_data "$VAL_DATA"

To smoke-test the training path only, prepare a small train_sharegpt.jsonl and run:

cd /path/to/virtualtools/traj
bash scripts/train.sh \
  --model_path "$INIT_MODEL" \
  --data_path /path/to/train_sharegpt.jsonl \
  --save_path "$TRAIN_SAVE_ROOT/manual_debug" \
  --project_root "$PWD" \
  --sft_env "$SFT_ENV" \
  --nproc 1 \
  --epochs 1 \
  --batch_size 1

Prepare these resources before running the full pipeline:

Agent inference input can be JSONL or Parquet. Each sample should contain at least:

{
  "id": 0,
  "prompt": "Were Scott Derrickson and Ed Wood of the same nationality?",
  "answers": ["yes"],
  "question": "Were Scott Derrickson and Ed Wood of the same nationality?",
  "answer": "yes"
}

For scaffold teacher entity tracking, optional fields can be provided:

{
  "entity_list": ["Entity A", "Entity B"],
  "fuzzed_features": {
    "Entity A exact clue": "fuzzy clue for Entity A"
  }
}

The following example generates agent trajectories on a single machine with a local vLLM service. Replace model and Wikipedia resource paths with your own paths.

cd /path/to/virtualtools/traj
conda activate virtualtools-traj

pip install -e agent_utils

export PROJECT_ROOT=$PWD
export PYTHONPATH=$PROJECT_ROOT:$PYTHONPATH
export ENABLE_WIKI_TOOLS=true
export VERL_EXTERNAL_PROMPT_FILE=$PROJECT_ROOT/traj_generation/agent/llm_agent/prompt.py
export MAX_STEPS=30
export RECENT_STEPS=2
export MAX_WORKERS=64
export LLM_MAX_TOKENS=1024

export WIKI_INDEX_PATH=/path/to/web_search_index
export WIKI_DB_PATH=/path/to/visit_offsets.sqlite
export WIKI_JSONL_PATH=/path/to/wikiextractor.jsonl

export MODEL_PATH=/path/to/qwen-or-other-instruct-model
export VLLM_MODEL=$MODEL_PATH

Start vLLM:

vllm serve "$MODEL_PATH" \
  --host 0.0.0.0 \
  --port 8001 \
  --served-model-name "$MODEL_PATH" \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85

Run inference in another terminal. Re-export or source the variables above; at minimum, PYTHONPATH, ENABLE_WIKI_TOOLS, VERL_EXTERNAL_PROMPT_FILE, WIKI_INDEX_PATH, WIKI_DB_PATH, WIKI_JSONL_PATH, OPENAI_BASE_URL, and VLLM_MODEL must be set.

cd /path/to/virtualtools/traj
conda activate virtualtools-traj

export PYTHONPATH=$PWD:$PYTHONPATH
export ENABLE_WIKI_TOOLS=true
export VERL_EXTERNAL_PROMPT_FILE=$PWD/traj_generation/agent/llm_agent/prompt.py
export WIKI_INDEX_PATH=/path/to/web_search_index
export WIKI_DB_PATH=/path/to/visit_offsets.sqlite
export WIKI_JSONL_PATH=/path/to/wikiextractor.jsonl
export OPENAI_BASE_URL=http://127.0.0.1:8001/v1
export VLLM_MODEL=/path/to/qwen-or-other-instruct-model

python traj_generation/agent_eval.py \
  --input_file /path/to/input.jsonl \
  --output_file /path/to/output_traj.jsonl

For a remote or local OpenAI-compatible Judge:

export STEAM_HOST=http://127.0.0.1:9010
export STEAM_MODEL_MARKER=/path/to/judge-model-or-served-name
export STEAM_APP_ID=""
export STEAM_APP_KEY=""

python evaluation/rejection_sampling.py \
  --input /path/to/output_traj.jsonl \
  --output_correct /path/to/correct.jsonl \
  --output_incorrect /path/to/incorrect.jsonl \
  --max_workers 8

You can also start a Judge vLLM service on another machine:

bash evaluation/start_judge_vllm.sh \
  --model /path/to/judge-model \
  --port 9010 \
  --gpus 0,1,2,3 \
  --tp 4

python data_prepare/traj_to_sharegpt.py \
  --input /path/to/correct.jsonl \
  --output /path/to/train_sharegpt.jsonl \
  --min_traj_len 3 \
  --inject_state_to_think \
  --unify_system

Mix correct data from multiple rounds:

python data_prepare/importance_sampling_mix.py \
  --inputs 0:/path/to/round_0/correct.jsonl 1:/path/to/round_1/correct.jsonl \
  --current_round 1 \
  --output /path/to/mixed_correct.jsonl \
  --decay 0.5 \
  --target_size 4000

Incorrect samples can be stripped of trajectories and retried in the next round:

python data_prepare/strip_trajectories.py \
  --input /path/to/incorrect.jsonl \
  --output /path/to/next_round_input.jsonl

cd /path/to/virtualtools/traj

bash scripts/train.sh \
  --model_path /path/to/base-or-previous-checkpoint \
  --data_path /path/to/train_sharegpt.jsonl \
  --save_path /path/to/save_dir \
  --project_root "$PWD" \
  --sft_env /path/to/conda/envs/virtualtools-sft \
  --nproc 8 \
  --epochs 1 \
  --lr 5e-6 \
  --batch_size 1

After training, the script writes the latest checkpoint path to --save_path/latest.

The full pipeline is driven jointly by the training daemon and the generation orchestrator:

# Training node
cd /path/to/virtualtools/traj
bash scripts/train_daemon_9b.sh

# Generation node
cd /path/to/virtualtools/traj
bash scripts/orchestrate_gen_9b_dp.sh \
  --rounds 5 \
  --batch_size 10000 \
  --correct_threshold 4000 \
  --decay 0.5 \
  --init_model /path/to/base-model \
  --data_path /path/to/train.jsonl \
  --val_data /path/to/test.jsonl

orchestrate_* and train_daemon_* read .env from the repository root. When moving to a new machine, update .env first instead of editing scripts directly.

Signal-file protocol:

scripts/run_validation.sh starts TP=1 x N vLLM instances, shards the validation set across N workers, merges outputs, and runs the evaluation script.

bash scripts/run_validation.sh \
  --model_path /path/to/checkpoint \
  --val_data /path/to/test.jsonl \
  --save_dir /path/to/validation_output \
  --agent_gpus "0,1,2,3,4,5,6,7" \
  --agent_port 8011 \
  --no_summary

Evaluation defaults to evaluation/evaluate_hle_official.py, which requires an OpenAI-compatible Judge configured through JUDGE_MODEL, BASE_URL, API_KEY, and related environment variables. If you do not use this evaluator, keep the inference outputs and replace it with your own evaluation script.

traj/direct_infer/chat_qwen.py directly calls vLLM for ordinary QA without the tool loop:

python direct_infer/chat_qwen.py \
  --input_file /path/to/questions.jsonl \
  --output_file /path/to/direct_answers.jsonl \
  --base_url http://127.0.0.1:8001/v1 \
  --model /path/to/model \
  --max_workers 20

Single-machine run:

cd /path/to/virtualtools/prepare_wiki_data_env/gxy
conda activate virtualtools-traj

python pipeline.py \
  --start_idx 0 \
  --input_folder /path/to/parquet_folder \
  --output_folder /path/to/output_jsonl \
  --model_path /path/to/Qwen2.5-72B-Instruct

The input Parquet should contain fields like:

{
  "p_id": 19,
  "url": "https://en.wikipedia.org/wiki/Some_Article",
  "query": "What type of production is Some Article?",
  "answer": "A New Zealand television miniseries.",
  "id": "unique_id"
}

The output JSONL contains fields like:

{
  "url2feature": "{...}",
  "completed_tree": "{...}",
  "child_tree_0": "{...}",
  "child_tree_1": "{...}"
}

prepare_wiki_data_env/gxy/wiki/ also contains Wikipedia dump preprocessing tools for merging WikiExtractor outputs, text matching, empty-text completion, format cleaning, and statistics.

This project uses or integrates vLLM, Pyserini, LlamaFactory, DeepSpeed, Transformers, OpenAI-compatible APIs, and other open-source ecosystem components. Please follow the licenses of the corresponding projects when using or releasing derived work.

If you find this repository useful for your research, please cite:

@misc{geng2026deepsearchworldselfdistillationdeepsearch,
      title={DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment}, 
      author={Xinyu Geng and Xuanhua He and Sixiang Chen and Yanjing Xiao and Fan Zhang and Shijue Huang and Haitao Mi and Zhenwen Liang and Tianqing Fang and Yi R. Fung},
      year={2026},
      eprint={2607.07820},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.07820}, 
}