Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

Xuehai Wang Haowei Qin Tongxin Liu Junkai Li Buqiang Xu Jintian Zhang Yijun Chen Zirui Xue Shumin Deng\corresponding

Abstract

Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research¹¹ 1 The code is available at https://github.com/zjunlp/AutoSciRub..

¹Zhejiang University

²University of Electronic Science and Technology of China

³Beijing University of Posts and Telecommunications

⁴Zhejiang University of Technology

22651308@zju.edu.cn, 231sm@zju.edu.cn

1 Introduction

Large language model agents are increasingly being used to automate scientific research workflows, including literature review, hypothesis formulation, experimental design, code implementation, result analysis, and scientific report generation (Wei et al. 2025). Recent systems have further advanced toward multi-agent and long-horizon research processes that produce both scientific reports and supporting experimental artifacts (Lu et al. 2024; Schmidgall et al. 2025; Mitchener et al. 2025). However, as their execution capabilities improve, a fundamental challenge becomes more pressing: how can we ensure that the resulting research actually fulfills the scientific intent of the original instruction? Scientific instructions often specify only a high-level objective while leaving intermediate goals, methodological requirements, expected evidence, and success conditions implicit. As a result, an agent may produce a plausible-looking report while omitting essential analyses, using inappropriate procedures, or making claims unsupported by experimental evidence. Figure 1 illustrates how these implicit requirements can lead to failures throughout research planning and execution.

Refer to caption — Figure 1: Why autonomous research agents struggle with open-ended research tasks.

Existing scientific-agent benchmarks evaluate research outputs using executable metrics, expert-authored rubrics, and LLM-based judgments (Yehudai et al. 2026). Expert-authored rubrics are particularly useful because they provide explicit and interpretable criteria for assessing complex research artifacts (Starace et al. 2025). However, constructing them requires substantial domain expertise and manual effort, making them difficult to scale to new research tasks. Recent work on automatic rubric generation offers a promising alternative by deriving task-specific criteria from natural-language instructions and contextual information (Chen et al. 2026a; Siro, Aliannejadi, and Aliannejadi 2026; Ding 2026). Nevertheless, scientific rubric generation is especially challenging because valid criteria depend not only on the instruction, but also on task-relevant literature, available data, domain conventions, and execution constraints. Without such grounding, automatically generated criteria may be incomplete, scientifically unjustified, or infeasible to verify.

Moreover, rubrics in existing scientific-agent benchmarks are predominantly treated as post-hoc evaluation instruments. They score completed artifacts, but do not help the agent determine what evidence should be produced or how an incomplete artifact should be improved. We argue that rubrics should instead serve as intermediate scientific specifications. They should make the implicit requirements of a research task explicit and connect task interpretation, research execution, output verification, and iterative revision. Under this view, a reliable research agent should learn to evaluate before improving.

Based on this principle, we introduce AutoSciRub, a framework that automatically induces evidence-grounded scientific rubrics and uses them to guide iterative research improvement. AutoSciRub consists of two stages. First, Automatic Rubric Induction decomposes the original instruction into atomic scientific goals and grounds them in relevant literature, web evidence, available task data, and environmental constraints. The resulting task-specific rubric makes the required analyses, evidence, and success conditions explicit. Second, after the agent produces a research report and supporting artifacts, Rubric-Guided Iterative Revision evaluates the available evidence criterion by criterion, identifies unmet scientific requirements, and provides targeted feedback for revision.

We evaluate AutoSciRub on all 40 tasks in ResearchClawBench and a fixed subset of 20 end-to-end scientific discovery tasks from AstaBench. On ResearchClawBench, AutoSciRub yields an average improvement of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using the fixed DeepSeek-V4-Flash backbone. On AstaBench, it further improves scores by an average of 16.8 points across three agent systems. These results demonstrate that automatically induced, evidence-grounded rubrics can provide scalable evaluation criteria while also serving as effective guidance for autonomous scientific research.

2 Related Work

Autonomous Scientific Research Agents and Benchmarks. Early systems organize end-to-end scientific research as single-agent pipelines or centrally orchestrated workflows, coordinating literature review, experimentation, and manuscript preparation within a unified process (Lu et al. 2024; Yamada et al. 2025; Ifargan et al. 2024). Multi-agent systems instead distribute these stages across specialized roles for collaborative ideation, critical discussion, experiment execution, and research coordination (Schmidgall et al. 2025; Ghareeb et al. 2026; Tang et al. 2025; Gottweis et al. 2026), while shared research states support long-horizon exploration, failure recovery, and iterative hypothesis refinement (Mitchener et al. 2025; Lyu et al. 2026; Gao, Fang, and Zitnik 2026; Liu et al. 2026; Lu et al. 2026). Complementary memory and context-management systems further support persistent agents through structured long-horizon memory (Xu et al. 2026a; Li et al. 2026a), cache-efficient context control (Xu et al. 2026b), and hierarchical multimodal experience storage (Chen et al. 2026c). In parallel, scientific-agent benchmarks have progressed from executable coding and model-development tasks under fixed objectives (Chen et al. 2025; Chan et al. 2025), to research reproduction that requires reconstructing methods and experiments (Starace et al. 2025; Luo et al. 2026), and further to end-to-end settings that assess experiment formulation, raw-evidence analysis, and scientific report generation from high-level instructions (Bragg et al. 2025; Garikaparthi, Patwardhan, and Cohan 2026; Wang et al. 2026; Xu et al. 2026c).

Rubric-Guided Evaluation and Iterative Refinement. Rubric-based evaluation offers a structured alternative to holistic judging through explicit, fine-grained criteria (Hashemi et al. 2024; Chen et al. 2026a). Expert-authored hierarchical rubrics decompose complex research artifacts into independently assessable components (Starace et al. 2025; Sharma et al. 2025), while later benchmarks introduce atomic, verifiable criteria and question-specific scoring or deduction points (Li et al. 2026b; Fan et al. 2024). Rubric construction has become increasingly automated (Siro, Aliannejadi, and Aliannejadi 2026; Ding 2026; Yu et al. 2026). Recent methods generate dataset- or instance-specific criteria (Wang and Blanco 2026), retrieve rubric knowledge from related queries (Dhole and Agichtein 2026; Chen et al. 2026b), or recursively expand questions into structured criteria trees (Gao et al. 2026; Zhu et al. 2026); recursive decomposition and filtering further reduce redundancy, overlap, and preference misalignment (Shen et al. 2026; Hong et al. 2026). Recent refinement frameworks combine explicit rubrics with test-time verifiers or pre-execution checks to assess outputs against task-specific criteria (Wan et al. 2026; LeVine et al. 2026; Ye et al. 2026; Zheng et al. 2026). Criterion-level diagnoses are then fed back into generation for targeted revision of responses (Madaan et al. 2023; Gou et al. 2024), agent trajectories, and tool-use programs (LeVine et al. 2026).

3 Method

Autonomous research agents often revise their outputs without an explicit specification of the scientific requirements that the task entails. Our framework addresses this limitation by first transforming an underspecified research instruction into a task-specific executable rubric and then using the rubric to guide research execution, verification, and revision. As illustrated in Figure 2, Automatic Rubric Induction first constructs an instruction-derived rubric skeleton through Rubric Skeleton Induction, and then transforms it into a task-specific executable rubric through Scientific Literature Grounding, Task-Data Exploration, and Criterion Synthesis. Rubric-Guided Iterative Revision subsequently evaluates the generated research artifact criterion by criterion and provides targeted feedback on missing experiments, comparisons, and evidence. We first formulate the research task and then describe the two stages in detail.

3.1 Task Formulation

We consider a collection of multi-domain, end-to-end autonomous scientific research tasks $\mathcal{D}$ spanning Astronomy, Chemistry, Earth Science, Energy Science, Information Science, Life Science, Materials Science, Mathematics, Neuroscience, and Physics. Each task is represented as $\tau_{i}=(x_{i},\mathcal{E}_{i})$ , where $x_{i}$ is a high-level research instruction and $\mathcal{E}_{i}$ is the task-visible environment, including available literature, data, search, code execution, and domain-specific tools.

Given $\tau_{i}$ , a research agent $\pi$ performs scientific reasoning, experimentation, and analysis to produce

\mathcal{A}_{i}^{\pi}=\pi(x_{i},\mathcal{E}_{i})=(r_{i},\mathcal{S}_{i}),

(1)

where $r_{i}$ is the scientific report and $\mathcal{S}_{i}$ contains its supporting code, results, analyses, tables, and figures.

Research instructions are typically underspecified: they state a broad objective without fully defining the scientific questions, comparisons, or evidence required. We therefore associate each task with a latent scientific specification

\mathcal{Z}_{i}^{*}=(\mathcal{G}_{i}^{*},\mathcal{C}_{i}^{*}),

(2)

where $\mathcal{G}_{i}^{*}$ denotes the required scientific goals and $\mathcal{C}_{i}^{*}$ the evidence requirements for verifying them. Artifact quality is defined by

q_{i}^{\pi}=\operatorname{Eval}\left(\mathcal{A}_{i}^{\pi};\mathcal{Z}_{i}^{*}\right),

(3)

where $\operatorname{Eval}(\cdot)$ assesses goal coverage and evidence adequacy. Our objective is to improve the artifact quality across tasks.

Since neither $\mathcal{Z}_{i}^{*}$ nor its evaluator is available during execution, AutoSciRub induces an instruction-derived rubric skeleton $\widehat{\mathcal{G}}_{i}$ and a task-specific executable rubric $\widehat{\mathcal{R}}_{i}$ from the instruction, scientific literature, and task-visible data. The rubric operationalizes the latent specification and guides execution, criterion-level verification, and iterative revision.

3.2 Automatic Rubric Induction

Rather than selecting criteria from a fixed library, we induce a task-specific executable rubric at inference time. As shown in Figure 2, this process consists of four steps: Rubric Skeleton Induction, Scientific Literature Grounding, Task-Data Exploration, and Criterion Synthesis. These steps make the instruction’s implicit requirements explicit, ground them in established scientific practice and task-visible resources, and convert them into verifiable criteria.

Rubric Skeleton Induction.

A high-level research instruction often implies multiple analyses, comparisons, and interpretations without listing them explicitly. We organize these requirements into a compact set of atomic scientific goals:

\widehat{\mathcal{G}}_{i}=\phi_{\mathrm{inst}}(x_{i})=\left\{g_{i,k}\right\}_{k=1}^{K_{i}},

(4)

where $K_{i}$ is the number of induced goals and $g_{i,k}=(n_{i,k},s_{i,k})$ contains a concise goal name $n_{i,k}$ and a concrete scientific requirement $s_{i,k}$ . Together, these goals form an instruction-derived rubric skeleton.

The goals must be traceable to the instruction, cover its main requirements, and avoid unnecessary overlap. At this stage, the agent observes only $x_{i}$ and does not introduce specific methods, metrics, baselines, or expected results. The skeleton therefore defines what the task should address, while later steps determine how each goal should be established.

Scientific Literature Grounding.

The rubric skeleton defines the task scope but not the scientific practice needed to satisfy each goal. For every $g_{i,k}$ , the agent forms broad queries over relevant concepts, methods, metrics, and standard protocols. It first consults task-provided related literature and then supplements missing coverage through the native web-search capability of the agent harness and external services, including arXiv, OpenAlex, Semantic Scholar, and Tavily. The retrieved sources may include papers, preprints, official dataset or benchmark pages, method repositories, and software documentation.

Queries avoid long instruction spans, task identifiers, and task-specific filenames. Candidate papers are ranked by relevance to the induced goals. Before full-text access, the retrieval layer removes candidates whose titles match a hidden target-paper blocklist maintained by the evaluation harness and unavailable to the induction agent. The agent then retains five to seven core papers for the task and consults additional documentation when needed.

For each goal, the selected sources are summarized as

\mathcal{K}_{i,k}=\phi_{\mathrm{lit}}\left(g_{i,k};\widetilde{\mathcal{L}}_{i}\right),

(5)

where $\widetilde{\mathcal{L}}_{i}$ is the filtered source set. $\mathcal{K}_{i,k}$ records relevant methods and protocols, typical analyses, metrics, baselines, controls, ablations, robustness checks, and useful evidence forms. The information is organized by scientific goal rather than by individual paper. Retrieved sources guide experiment and evidence design but are not treated as experimental evidence for the current task.

Task-Data Exploration.

Literature grounding describes what is scientifically appropriate, but the proposed analyses must also be supported by the available resources. The agent therefore performs a lightweight inspection of the task-visible data:

\mathcal{P}_{i}=\phi_{\mathrm{data}}\left(\mathcal{E}_{i}\right),

(6)

where $\mathcal{P}_{i}$ records the available files and datasets, their formats and dimensions, key fields, units, labels, experimental conditions, cross-source relations, and relevant constraints.

This profile identifies which data sources support each goal and which literature-suggested analyses are feasible without assuming unavailable labels, conditions, or reference values. It is used only for planning; full experiments and analyses are deferred to task execution.

Criterion Synthesis.

Finally, the agent combines the rubric skeleton, literature-grounded knowledge, and task-data profile. It selects feasible experiments and analyses for each goal and specifies the evidence needed to support the resulting claims. These requirements are synthesized into a task-specific executable rubric:

\widehat{\mathcal{R}}_{i}=\phi_{\mathrm{syn}}\left(\widehat{\mathcal{G}}_{i},\left\{\mathcal{K}_{i,k}\right\}_{k=1}^{K_{i}},\mathcal{P}_{i}\right)=\left\{\rho_{i,j}\right\}_{j=1}^{M_{i}},

(7)

where $M_{i}$ is the number of synthesized criteria. Each criterion $\rho_{i,j}$ links to one or more scientific goals and specifies the relevant data sources, required experiments or analyses, evaluation metrics and comparisons, expected evidence artifacts, and its satisfaction condition.

The resulting rubric aligns established scientific practice with the evidence that can be produced from the current task, providing an explicit specification for subsequent research execution and verification.

3.3 Rubric-Guided Iterative Revision

The induced rubric is provided to the backbone research agent as an execution-time specification. Conditioned on $\widehat{\mathcal{R}}_{i}$ , the agent performs the required experiments and analyses and produces an initial artifact $\mathcal{A}_{i}^{(0)}=(r_{i}^{(0)},\mathcal{S}_{i}^{(0)})$ , containing the report and its supporting code, results, tables, and figures.

Criterion-Level Verification.

The initial execution may still leave some rubric requirements unmet. At revision round $t$ , a verifier checks each criterion against the current artifact:

\left(z_{i,j}^{(t)},d_{i,j}^{(t)}\right)=\operatorname{Verify}\left(\rho_{i,j},\mathcal{A}_{i}^{(t)}\right),

(8)

where $z_{i,j}^{(t)}\in\{0,1\}$ indicates whether the criterion is satisfied, and $d_{i,j}^{(t)}$ describes the remaining evidence gap. The verifier considers the required experiments, reported results, supporting artifacts, and whether the conclusions are adequately supported. This criterion-level check identifies specific omissions rather than returning only a holistic score.

Targeted Revision.

Failed criteria and their diagnoses form the revision feedback $\Delta_{i}^{(t)}$ . The agent then updates the current artifact by addressing these gaps:

\mathcal{A}_{i}^{(t+1)}=\pi_{\mathrm{rev}}\left(\mathcal{A}_{i}^{(t)},\Delta_{i}^{(t)},\mathcal{E}_{i}\mid\widehat{\mathcal{R}}_{i}\right).

(9)

A revision may add missing experiments or comparisons, correct evidence artifacts, strengthen the analysis, or remove unsupported claims. The process stops when all criteria are satisfied or the revision budget is reached, yielding the final artifact $\mathcal{A}_{i}^{*}=\mathcal{A}_{i}^{(T)}$ .

Figure 3: Performance comparison on AstaBench. AutoSciRub consistently improves mean benchmark scores across all three evaluated agent configurations.

4 Experiments

A. Cross-Model Generalization Fixed agent harness:

Codex

Backbone LLM

Setting

Overall

Astro.

Chem.

Earth

Energy

Info.

Life

Mater.

Math

Neuro.

Phys.

vanilla

18.66

29.69

10.97

21.25

17.87

17.61

16.02

17.55

16.65

5.60

33.43

GPT-5.4

+AutoSciRub

21.04

\uparrow

2.38

31.43

\uparrow

1.74

11.62

\uparrow

0.65

22.22

\uparrow

0.97

22.70

\uparrow

4.83

13.92

\downarrow

3.69

19.85

\uparrow

3.83

22.70

\uparrow

5.15

20.57

\uparrow

3.92

7.72

\uparrow

2.12

37.69

\uparrow

4.26

vanilla

20.86

31.56

11.96

23.99

17.95

15.69

16.88

22.14

24.20

5.84

38.40

GLM-5.2

+AutoSciRub

22.73

\uparrow

1.87

31.99

\uparrow

0.43

17.85

\uparrow

5.89

23.25

\downarrow

0.74

20.70

\uparrow

2.75

18.00

\uparrow

2.31

17.34

\uparrow

0.46

23.48

\uparrow

1.34

23.03

\downarrow

1.17

12.58

\uparrow

6.74

39.06

\uparrow

0.66

vanilla

19.05

28.01

13.36

19.14

19.65

9.31

15.10

22.31

22.96

6.30

34.37

MiniMax-M3

+AutoSciRub

21.04

\uparrow

1.99

27.10

\downarrow

0.91

13.94

\uparrow

0.58

21.53

\uparrow

2.39

26.46

\uparrow

6.81

12.20

\uparrow

2.89

17.58

\uparrow

2.48

20.86

\downarrow

1.45

22.38

\downarrow

0.58

12.25

\uparrow

5.95

36.10

\uparrow

1.73

B. Cross-Harness Generalization Fixed backbone LLM:

DeepSeek-V4-Flash

Agent Harness

Setting

Overall

Astro.

Chem.

Earth

Energy

Info.

Life

Mater.

Math

Neuro.

Phys.

vanilla

16.60

28.35

9.39

15.32

24.60

9.27

13.57

20.09

12.64

4.04

28.70

Claude Code

+AutoSciRub

18.74

\uparrow

2.14

28.15

\downarrow

0.20

9.48

\uparrow

0.09

20.30

\uparrow

4.98

24.75

\uparrow

0.15

11.06

\uparrow

1.79

12.79

\downarrow

0.78

21.30

\uparrow

1.21

17.91

\uparrow

5.27

7.67

\uparrow

3.63

33.95

\uparrow

5.25

vanilla

17.25

24.26

8.90

22.39

18.93

9.29

15.70

19.07

15.08

5.11

33.71

OpenClaw

+AutoSciRub

20.36

\uparrow

3.11

30.31

\uparrow

6.05

12.59

\uparrow

3.68

22.85

\uparrow

0.45

25.24

\uparrow

6.32

16.20

\uparrow

6.90

17.18

\uparrow

1.47

17.95

\downarrow

1.13

23.12

\uparrow

8.03

8.56

\uparrow

3.45

29.62

\downarrow

4.09

vanilla

14.49

21.12

9.59

15.20

16.96

10.68

9.29

20.28

12.05

7.05

22.73

OpenScience

+AutoSciRub

18.09

\uparrow

3.60

26.58

\uparrow

5.46

12.04

\uparrow

2.45

19.34

\uparrow

4.14

22.94

\uparrow

5.98

13.82

\uparrow

3.14

17.53

\uparrow

8.24

16.02

\downarrow

4.26

17.45

\uparrow

5.40

7.06

\uparrow

0.01

28.11

\uparrow

5.38

Table 1: Main results on ResearchClawBench. Panel A evaluates generalization across backbone LLMs under the fixed Codex harness, while Panel B evaluates generalization across agent harnesses using the fixed DeepSeek-V4-Flash backbone. Vanilla denotes the unmodified configuration, and +AutoSciRub denotes the same configuration augmented with our plugin layer. Bold values indicate the better result within each paired comparison. Green and red arrows denote absolute improvements and degradations, respectively.

4.1 Experimental Setup

Benchmarks and Metrics.

We evaluate AutoSciRub on ResearchClawBench (Xu et al. 2026c) and the End-to-End Discovery category of AstaBench (Bragg et al. 2025). ResearchClawBench contains 40 tasks across ten scientific domains. AstaBench evaluates end-to-end AI and NLP research from experimentation to report generation, and we randomly sample 20 tasks from its Easy split. We report the average rubric-based score on both benchmarks and task completion on AstaBench. Further details are provided in Appendix A.

Implementation Details.

We compare each vanilla agent with its AutoSciRub-augmented counterpart under identical task inputs, tools, and execution environments. For ResearchClawBench, we evaluate both cross-model and cross-harness generalization. In cross-model evaluation, we fix Codex (OpenAI 2026a) as the agent harness and use GPT-5.4 (OpenAI 2026b), GLM-5.2 (Z.ai 2026), and MiniMax-M3 (MiniMax 2026) as backbone models. In cross-harness evaluation, we fix DeepSeek-V4-Flash (DeepSeek-AI 2026) as the backbone model and compare Claude Code (Anthropic 2026), OpenClaw (OpenClaw Foundation 2026), and OpenScience (Synthetic Sciences 2026). Following the benchmark-specific evaluation protocols, we use GPT-5.1 (OpenAI 2025) to evaluate report quality.

For AstaBench, we evaluate three representative agent configurations: Claude Code with DeepSeek-V4-Flash, OpenClaw with DeepSeek-V4-Flash, and Codex with GPT-5.4-mini. The generated artifacts are evaluated using MiniMax-M3 as the evaluator. Across both benchmarks, each resulting submission is independently scored three times, and the mean score is reported. Additional implementation details, including agent configurations, prompts, and execution settings, are provided in Appendix A.

(a) Rubric quality

(b) Scientific core coverage vs. report score.

Figure 4: Rubric quality and its downstream effects. (a) Mean quality profiles of rubric skeletons and executable rubrics across 40 tasks. (b) Association between scientific core coverage and downstream report score. (c) Task-level report-score changes from rubric skeletons to executable rubrics.

4.2 Overall Performance

Results on ResearchClawBench.

As shown in Table 1, AutoSciRub improves the overall ResearchClawBench score in all six model–harness configurations. Under the fixed Codex harness, AutoSciRub yields gains of 2.38 points for GPT-5.4, 1.87 points for GLM-5.2, and 1.99 points for MiniMax-M3. These gains remain consistent across model families and baseline strengths. Among these configurations, GLM-5.2 with AutoSciRub achieves the highest overall score of 22.73. Together, the consistent gains across three backbone LLMs indicate that the effectiveness of AutoSciRub is not model-specific.

The improvements are similarly preserved when varying the agent harness. With DeepSeek-V4-Flash as the shared backbone, AutoSciRub yields gains of 2.14 points for Claude Code, 3.11 points for OpenClaw, and 3.60 points for OpenScience. The larger OpenClaw and OpenScience gains further show that AutoSciRub remains effective across agent systems with distinct workflows and baseline capabilities. At the domain level, AutoSciRub improves 49 of 60 paired comparisons, including consistent gains in chemistry, energy science, and neuroscience across all six configurations. Overall, the improvements across backbone models, agent harnesses, and scientific domains demonstrate that AutoSciRub generalizes well to heterogeneous scientific research agents.

Generalization to AstaBench.

We further evaluate whether these benefits transfer to a different task collection and evaluation protocol. As shown in Figure 3, AutoSciRub substantially improves the mean scores of all three evaluated agents on the fixed 20-task AstaBench End-to-End Discovery subset. It yields gains of 19.36 points for Claude Code, 12.61 points for Codex, and 18.38 points for OpenClaw, corresponding to an average improvement of 16.78 points. Claude Code and Codex also increase their successful-task counts from 18/20 to 20/20. OpenClaw completes 19/20 tasks under both settings while gaining 18.38 points, showing that AutoSciRub improves the quality of completed research outputs in addition to task completion.

(a) Overall refinement dynamics.

(b) Distribution of task-level score changes.

Figure 6: Analysis of rubric-guided refinement. (a) Overall score and task stopping behavior across refinement steps. (b) Distribution of score changes from the initial reports to the final reports. (c) Mean score improvement across scientific domains.

4.3 Ablation Study

Setting	Skel.	Grd.	Rev.	Score $\uparrow$	$\Delta_{\mathrm{Base}}$
Base	✗	✗	✗	17.25	–
w/ Skeleton	✓	✗	✗	17.61	+0.36
w/ Grounded Rubric	✓	✓	✗	18.31	+1.06
Full	✓	✓	✓	20.36	+3.11

Table 2: Stage-wise cumulative ablation on ResearchClawBench. Skel. denotes Rubric Skeleton Induction; Grd. adds Scientific Literature Grounding, Task-Data Exploration, and Criterion Synthesis; and Rev. adds Rubric-Guided Iterative Revision. Scores are averaged over the same 40 tasks, and

\Delta_{\mathrm{Base}}

denotes the improvement over the base setting.

Table 2 presents a cumulative stage-wise ablation on all 40 ResearchClawBench tasks using OpenClaw with DeepSeek-V4-Flash. Starting from the vanilla agent, Rubric Skeleton Induction (Skel.) increases the score from 17.25 to 17.61. This modest gain suggests that making the instruction’s implicit scientific requirements explicit helps the agent identify what should be investigated, but offers limited guidance on how these requirements should be carried out and verified.

Adding the grounded-rubric stage (Grd.) further raises the score to 18.31, corresponding to a gain of 1.06 points over the base and an additional 0.70 points over skeleton induction alone. By grounding the induced goals in scientific literature, examining the task-visible data, and synthesizing feasible experiments and evidence requirements, this stage turns a high-level rubric skeleton into more concrete and verifiable criteria.

Rubric-Guided Iterative Revision (Rev.) achieves the best score of 20.36, improving by 3.11 over the base and 2.05 over the grounded-rubric setting. This largest incremental gain indicates that the rubric is most effective when it is used not only to guide execution, but also to identify unmet criteria and support targeted revision of the report and its supporting artifacts. Overall, the monotonic gains show that rubric skeleton induction, grounding, and iterative revision play complementary roles in turning underspecified research instructions into better-supported research outputs.

4.4 Analysis

Rubric Induction Produces Executable Guidance.

Open-ended research instructions rarely specify the experiments, evidence, and success conditions needed to complete a task. Figure 5 shows how AutoSciRub addresses this underspecification. While the benchmark checklist mainly describes what should appear in the final report, the executable rubric also specifies how the required evidence should be produced and verified through concrete experiments, artifacts, metrics, baselines, and data sources. It therefore serves as an execution-time specification rather than only a post-hoc evaluation instrument.

For each of the 40 ResearchClawBench tasks, we compare the instruction-derived rubric skeleton with the task-specific executable rubric produced after literature grounding, task-data exploration, and criterion synthesis. As shown in Figure 4(a), the mean score across the four evaluated dimensions increases from 2.20 to 3.84. The largest improvement occurs in specificity, which increases from 1.65 to 4.40 ( $+2.75$ ), followed by evidence verifiability from 1.78 to 4.08 ( $+2.30$ ) and actionability from 2.00 to 3.83 ( $+1.83$ ). These improvements show that rubric induction effectively operationalizes broad scientific goals into concrete, actionable, and verifiable requirements for research execution and verification.

Rubric Induction Improves Operationalization but Not Scientific Framing.

Although rubric induction substantially improves specificity, actionability, and evidence verifiability, scientific core coverage slightly decreases from 3.35 to 3.07. This contrast suggests that AutoSciRub is more effective at translating an identified scientific direction into concrete and verifiable requirements than at revising the underlying framing of the research problem.

As shown in Figure 4(b), scientific core coverage remains positively associated with report quality for both rubric skeletons and executable rubrics. Figure 4(c) further shows that executable rubrics yield a mean report-score gain of 0.70 across all 40 tasks, with improvements observed on 34 tasks. These results suggest that AutoSciRub is effective at translating an identified scientific core into concrete experiments and verifiable evidence requirements, but is less effective at identifying that core when it is missing from the initial rubric skeleton. In such cases, grounding and synthesis may elaborate secondary or off-target analyses rather than correct the research direction. Higher-level scientific judgment therefore remains largely dependent on the backbone model.

Revision Round	Rubric-Free Self-Refinement		Rubric-Guided Revision
Revision Round	Score	$\Delta$	Score	$\Delta$
1	18.80	+0.49	19.47	+1.16
2	18.52	+0.21	20.08	+1.77
3	19.08	+0.77	20.36	+2.05

Table 3: Comparison of rubric-free self-refinement and rubric-guided revision across three revision rounds on all 40 ResearchClawBench tasks.

\Delta

values denote score changes relative to the shared unrevised baseline score of 18.31.

Rubric-Guided Revision Outperforms Generic Self-Refinement.

We analyze whether the improvement from iterative revision is caused by rubric guidance or merely by giving the agent additional opportunities to rewrite its report. Starting from the same checkpoint-0 reports, we compare rubric-guided revision with a rubric-free self-refinement baseline, which asks the agent to inspect and improve its previous output without criterion-level feedback. As shown in Table 3, rubric-free refinement increases the average score from 18.31 to only 19.08 after three rounds and even causes a regression at checkpoint 2. In contrast, rubric-guided revision improves the score monotonically to 20.36, yielding a cumulative improvement of $2.05$ , compared with only $0.77$ without rubrics. Thus, rubric-guided revision achieves approximately $2.7$ times the cumulative improvement of rubric-free self-refinement. This result shows that the gain does not come from repeated rewriting alone: explicit criteria identify concrete deficiencies and direct the agent toward unsatisfied scientific requirements, whereas generic self-refinement produces less stable improvements.

Figure 6(a) further shows how the guided improvement accumulates across revision rounds. The first revision raises the average score from 18.31 to 19.47, contributing an improvement of $1.16$ . The second and third rounds add improvements of $0.61$ and $0.28$ , respectively. This decrease in aggregate gain mainly results from adaptive stopping rather than ineffective later revisions. After the first round, 23 tasks have already passed the verifier, leaving only 17 tasks for a second revision and six for a third. For these remaining tasks, the average per-round improvements are still $1.43$ and $1.89$ , respectively. Overall, 35 of the 40 tasks pass within three revisions. These results suggest that one revision offers a strong default cost/performance trade-off, while additional rounds provide targeted improvements for harder tasks.

The task- and domain-level results in Figures 6(b) and 6(c) show that the overall gain is broadly distributed rather than dominated by a few high-improvement cases. Specifically, 36 of the 40 reports improve over their initial versions, and positive average gains are observed in most scientific domains. Mathematics, earth science, and information science achieve the largest average improvements of $4.12$ , $3.73$ , and $3.32$ , respectively. These results demonstrate that the improvement of rubric-guided revision is consistent across tasks and domains, rather than being driven by a single high-scoring case.

5 Conclusion

We introduced AutoSciRub, a general framework that automatically induces task-specific scientific rubrics and uses them as execution-time guidance for autonomous research agents. AutoSciRub decomposes underspecified research instructions into scientific goals, grounds evaluation criteria in external evidence, and guides criterion-level verification and revision of generated research artifacts. Experiments on ResearchClawBench and AstaBench show that AutoSciRub consistently improves different backbone models and agent harnesses, demonstrating its effectiveness and generalizability as a plugin layer for autonomous scientific research.

References

Anthropic (2026) Anthropic. 2026. Claude Code. https://github.com/anthropics/claude-code.
Bragg et al. (2025) Bragg, J.; D’Arcy, M.; Balepur, N.; Bareket, D.; Dalvi, B.; et al. 2025. AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite. CoRR, abs/2510.21652.
Chan et al. (2025) Chan, J. S.; Chowdhury, N.; Jaffe, O.; Aung, J.; Sherburn, D.; et al. 2025. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
Chen et al. (2026a) Chen, H.; Han, Z.; Yan, Y.; Zhu, Q.; Sun, M.; and Che, W. 2026a. From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape. CoRR, abs/2606.08625.
Chen et al. (2026b) Chen, Y.; Maiga, A.; Rahmani, H. A.; and Yilmaz, E. 2026b. Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems. arXiv preprint arXiv:2601.15161.
Chen et al. (2026c) Chen, Y.; Xiao, B.; Zhao, Y.; Xia, H.; Xu, B.; Fang, J.; Li, Y.; Zheng, Y.; Wang, X.; Xue, Z.; Zhang, L.; Li, H.; and Zhang, N. 2026c. LightMem-Ego: Your AI Memory for Everyday Life. CoRR, abs/2607.11487.
Chen et al. (2025) Chen, Z.; Chen, S.; Ning, Y.; Zhang, Q.; Wang, B.; et al. 2025. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
DeepSeek-AI (2026) DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. CoRR, abs/2606.19348.
Dhole and Agichtein (2026) Dhole, K. D.; and Agichtein, E. 2026. RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation. In Moffat, A.; Scholer, F.; Bast, H.; Najork, M.; and Zhang, M., eds., Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2026, Melbourne, VIC, Australia, July 20–24, 2026, 3681–3687. ACM.
Ding (2026) Ding, L. 2026. AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation. CoRR, abs/2603.21362.
Fan et al. (2024) Fan, Z.; Wang, W.; Wu, X.; and Zhang, D. 2024. SedarEval: Automated Evaluation using Self-Adaptive Rubrics. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, volume EMNLP 2024 of Findings of ACL, 16916–16930. Association for Computational Linguistics.
Gao, Fang, and Zitnik (2026) Gao, S.; Fang, A.; and Zitnik, M. 2026. AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation. CoRR, abs/2605.28655.
Gao et al. (2026) Gao, S.; Su, Y.; Sui, P.; Ginder, C.; and Zitnik, M. 2026. Qworld: Question-Specific Evaluation Criteria for LLMs. CoRR, abs/2603.23522.
Garikaparthi, Patwardhan, and Cohan (2026) Garikaparthi, A.; Patwardhan, M.; and Cohan, A. 2026. ResearchGym: Evaluating Language Model Agents on Real-World AI Research. CoRR, abs/2602.15112.
Ghareeb et al. (2026) Ghareeb, A. E.; Chang, B.; Mitchener, L.; Yiu, A.; Szostkiewicz, C. J.; et al. 2026. A Multi-Agent System for Automating Scientific Discovery. Nature, 655: 497–505.
Gottweis et al. (2026) Gottweis, J.; Weng, W.-H.; Daryin, A.; Tu, T.; Sirkovic, P.; et al. 2026. Accelerating Scientific Discovery with Co-Scientist. Nature, 655: 487–496.
Gou et al. (2024) Gou, Z.; Shao, Z.; Gong, Y.; Shen, Y.; Yang, Y.; Duan, N.; and Chen, W. 2024. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Hashemi et al. (2024) Hashemi, H.; Eisner, J.; Rosset, C.; Durme, B. V.; and Kedzie, C. 2024. LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. In Ku, L.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, 13806–13834. Association for Computational Linguistics.
Hong et al. (2026) Hong, H.; Li, Y.; Chen, J.; Huy, L. G.; Ananiadou, S.; Kim, J.-j.; and Lin, C. 2026. Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction. arXiv preprint arXiv:2607.12835.
Ifargan et al. (2024) Ifargan, T.; Hafner, L.; Kern, M.; Alcalay, O.; and Kishony, R. 2024. Autonomous LLM-driven research from data to human-verifiable research papers. CoRR, abs/2404.17605.
LeVine et al. (2026) LeVine, W.; Evers, B.; Saltwick, S.; and Venkatesh, A. 2026. RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement. CoRR, abs/2605.09730.
Li et al. (2026a) Li, D.; Liu, Z.; Wang, J.; Huang, J.; Li, F.; Jia, B.; Hu, B.; and Zhang, M. 2026a. LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation. arXiv:2608.12990.
Li et al. (2026b) Li, R.; Du, M.; Xu, B.; Zhu, C.; Wang, X.; et al. 2026b. DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report. CoRR, abs/2601.08536.
Liu et al. (2026) Liu, J.; Qiu, S.; Li, M.; Li, B.; Ji, H.; et al. 2026. AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration. CoRR, abs/2605.20025.
Lu et al. (2024) Lu, C.; Lu, C.; Lange, R. T.; Foerster, J. N.; Clune, J.; et al. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. CoRR, abs/2408.06292.
Lu et al. (2026) Lu, Z.; Li, D.; Shi, Y.; Wang, B.; Wang, L.; and Hu, B. 2026. Structured Episodic Event Memory. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 6125–6141. Association for Computational Linguistics.
Luo et al. (2026) Luo, Y.; Yu, Z.; Wang, X.; Zhu, Y.; Zhang, N.; Wei, L.; Du, L.; Zheng, D.; and Chen, H. 2026. What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 841–861. Association for Computational Linguistics.
Lyu et al. (2026) Lyu, Y.; Zhang, X.; Yi, X.; Zhao, Y.; Guo, S.; et al. 2026. EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery. CoRR, abs/2603.08127.
Madaan et al. (2023) Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
MiniMax (2026) MiniMax. 2026. MiniMax M3: Frontier Coding, 1M Context, Native Multimodality—All in One Model. https://www.minimax.io/blog/minimax-m3. Accessed: 2026-07-24.
Mitchener et al. (2025) Mitchener, L.; Yiu, A.; Chang, B.; Bourdenx, M.; Nadolski, T.; et al. 2025. Kosmos: An AI Scientist for Autonomous Discovery. CoRR, abs/2511.02824.
OpenAI (2025) OpenAI. 2025. GPT-5.1: A Smarter, More Conversational ChatGPT. https://openai.com/index/gpt-5-1/. Accessed: 2026-07-24.
OpenAI (2026a) OpenAI. 2026a. Codex. https://github.com/openai/codex.
OpenAI (2026b) OpenAI. 2026b. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/. Accessed: 2026-07-24.
OpenClaw Foundation (2026) OpenClaw Foundation. 2026. OpenClaw. https://github.com/openclaw/openclaw.
Schmidgall et al. (2025) Schmidgall, S.; Su, Y.; Wang, Z.; Sun, X.; Wu, J.; et al. 2025. Agent Laboratory: Using LLM Agents as Research Assistants. In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, 5977–6043. Association for Computational Linguistics.
Sharma et al. (2025) Sharma, M.; Zhang, C. B. C.; Bandi, C.; Wang, C.; Aich, A.; Nghiem, H.; Rabbani, T.; Htet, Y.; Jang, B.; Basu, S.; Balwani, A.; Peskoff, D.; Ayestaran, M.; Hendryx, S. M.; Kenstler, B.; and Liu, B. 2025. ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents. CoRR, abs/2511.07685.
Shen et al. (2026) Shen, W. F.; Qiu, X.; Whitehouse, C.; Alazraki, L.; Goel, S.; et al. 2026. Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks. CoRR, abs/2602.05125.
Siro, Aliannejadi, and Aliannejadi (2026) Siro, C.; Aliannejadi, P.; and Aliannejadi, M. 2026. Learning to Judge: LLMs Designing and Applying Evaluation Rubrics. In Demberg, V.; Inui, K.; and Marquez, L., eds., Findings of the Association for Computational Linguistics: EACL 2026, Rabat, Morocco, March 24-29, 2026, Findings of ACL, 6371–6389. Association for Computational Linguistics.
Starace et al. (2025) Starace, G.; Jaffe, O.; Sherburn, D.; Aung, J.; Chan, J. S.; et al. 2025. PaperBench: Evaluating AI’s Ability to Replicate AI Research. In Singh, A.; Fazel, M.; Hsu, D.; Lacoste-Julien, S.; Berkenkamp, F.; Maharaj, T.; Wagstaff, K.; and Zhu, J., eds., Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net.
Synthetic Sciences (2026) Synthetic Sciences. 2026. OpenScience: The Open-Source AI Workbench for Scientific Research. https://github.com/synthetic-sciences/openscience.
Tang et al. (2025) Tang, J.; Xia, L.; Li, Z.; and Huang, C. 2025. AI-Researcher: Autonomous Scientific Innovation. CoRR, abs/2505.18705.
Wan et al. (2026) Wan, Y.; Fang, T.; Li, Z.; Huo, Y.; Wang, W.; Mi, H.; Yu, D.; and Lyu, M. R. 2026. Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, 24822–24835. Association for Computational Linguistics.
Wang et al. (2026) Wang, Z.; Bai, F.; Luo, Z.; Su, J.; Sun, K.; Yu, X.; Liu, J.; Zhou, K.; Cardie, C.; Dredze, M.; Xing, E. P.; and Hu, Z. 2026. FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights. CoRR, abs/2602.02905.
Wang and Blanco (2026) Wang, Z.; and Blanco, E. 2026. Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge. CoRR, abs/2605.30568.
Wei et al. (2025) Wei, J.; Yang, Y.; Zhang, X.; Chen, Y.; Zhuang, X.; Gao, Z.; Zhou, D.; Wang, G.; Gao, Z.; Cao, J.; Qiu, Z.; He, X.; Zhang, Q.; You, C.; Zheng, S.; Ding, N.; Ouyang, W.; Dong, N.; Cheng, Y.; Sun, S.; Bai, L.; and Zhou, B. 2025. From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery. CoRR, abs/2508.14111.
Xu et al. (2026a) Xu, B.; Chen, Y.; Fang, J.; Zhong, R.; Yao, Y.; Zhu, Y.; Du, L.; and Deng, S. 2026a. StructMem: Structured Memory for Long-Horizon Behavior in LLMs. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 122–146. Association for Computational Linguistics.
Xu et al. (2026b) Xu, B.; Xue, Z.; Chen, D.; Fu, C.; Wu, C.; Huang, C.; Jiang, C.; Fang, J.; Deng, X.; Chen, Y.; Yao, Y.; Wang, X.; Shang, J.; Yu, G.; and Zhang, N. 2026b. TokenPilot: Cache-Efficient Context Management for LLM Agents. CoRR, abs/2606.17016.
Xu et al. (2026c) Xu, W.; Li, S.; Ye, T.; Cao, Q.; Chen, Y.; et al. 2026c. ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research. CoRR, abs/2606.07591.
Yamada et al. (2025) Yamada, Y.; Lange, R. T.; Lu, C.; Hu, S.; Lu, C.; et al. 2025. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. CoRR, abs/2504.08066.
Ye et al. (2026) Ye, H.; Lin, H.; Tang, J.; Luo, Y.; Yang, C.; Su, C.; Thapa, R.; Yang, R.; Liu, R.; Li, Z.; Gao, C.; Ding, D.; He, G.; Zhang, M.; Sun, L.; Wang, W.; Zhong, Y.; Shen, Z.; He, D.; Ma, J.; Ermon, S.; Li, T.; Chu, X.; Wang, J. Z.; and Xu, Y. 2026. Evaluation-driven Scaling for Scientific Discovery. CoRR, abs/2604.19341.
Yehudai et al. (2026) Yehudai, A.; Eden, L.; Li, A.; Uziel, G.; Zhao, Y.; Bar-Haim, R.; Cohan, A.; and Shmueli-Scheuer, M. 2026. A Survey on Evaluation of LLM-based Agents. In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, 26690–26714. Association for Computational Linguistics.
Yu et al. (2026) Yu, J.; Xu, Z.; Wang, J.; and Yang, Y. 2026. Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance. CoRR, abs/2605.07461.
Z.ai (2026) Z.ai. 2026. GLM-5.2: Built for Long-Horizon Tasks. https://z.ai/blog/glm-5.2. Accessed: 2026-07-24.
Zheng et al. (2026) Zheng, J.; Zhang, J.; Luo, Y.; Mao, Y.; Gao, Y.; Du, L.; Chen, H.; and Zhang, N. 2026. Can We Predict Before Executing Machine Learning Agents? In Liakata, M.; Moreira, V. P.; Zhang, J.; and Jurgens, D., eds., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, 3941–3974. Association for Computational Linguistics.
Zhu et al. (2026) Zhu, M.; Wei, C.; Xu, J.; Cheng, Y.; Chen, Z.; and He, J. 2026. DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents. CoRR, abs/2606.17029.

Appendix A Additional Experimental and Methodological Details

A.1 Benchmark and Evaluation Details

ResearchClawBench.

ResearchClawBench contains 40 end-to-end scientific research tasks, with four tasks from each of ten domains: astronomy, chemistry, earth science, energy science, information science, life science, materials science, mathematics, neuroscience, and physics. Each task is constructed from a published target paper with a clearly defined scientific question and accessible raw data. The evaluated agent is provided with a task instruction, related literature, raw data, and a workspace in which it can inspect files, implement analyses, execute experiments, generate figures, and write a final scientific report. The hidden target paper and its official evaluation rubric are not accessible to the agent during rubric induction, research execution, verification, or revision.

We use the official ResearchClawBench evaluator without modifying its scoring protocol. Each benchmark rubric consists of weighted criteria associated with concrete scientific artifacts in the hidden target paper. The rubric contains both textual criteria, which assess scientific claims, methods, quantitative results, and explanations, and image criteria, which assess figure-level evidence. GPT-5.1 is used to evaluate the final report and its supporting artifacts against these criteria. The resulting task score ranges from 0 to 100, with a score around 50 intended to represent approximately target-paper-level re-discovery under the benchmark protocol.

For each configuration, the overall score is the arithmetic mean over all 40 tasks. Each domain score is the arithmetic mean over the four tasks belonging to that domain. All reported improvements are absolute differences between paired vanilla and $+{AutoSciRub}$ scores on the same task set.

AstaBench End-to-End Discovery.

We additionally evaluate AutoSciRub on a fixed subset of 20 tasks from the Easy split of the AstaBench End-to-End Discovery benchmark. The subset is sampled once from the original 40-task test set using random seed 20260707. The sampling procedure does not preserve the order of the source dataset; instead, the sampled order is retained as the execution order. The same fixed subset and execution order are used for all evaluated configurations. Table 4 lists the exact task identifiers and task names.

Order	Task ID	Task name
1	idea-226-simplified	simple-knowledge-enhanced-vqa
2	idea-206-simplified	selective-entity-quantization
3	idea-131-simplified	simple-semantic-backdoor-defense
4	idea-21-simplified	text-to-code-prompt-retrieval
5	idea-78-simplified	simple-bias-detection-worldmodel
6	idea-337-simplified	medical-verbalization-calibration
7	idea-17-simplified	topic-focused-memory-summarization
8	idea-86-simplified	simple-language-skill-transfer
9	idea-259-simplified	simple-relevance-metrics
10	idea-174-simplified	simple-position-aware-prompts
11	idea-5-simplified	simple-entity-debate
12	idea-159-simplified	confidence-based-contamination-detection
13	idea-304-simplified	dual-task-adversarial-robustness
14	idea-192-simplified	simple-fact-unlearning
15	idea-147-simplified	mini-socratic-code-generation
16	idea-301-simplified	simplified-task-learning-transfer
17	idea-134-simplified	simple-cross-lingual-backdoor
18	idea-278-simplified	visual-entity-knowledge-test
19	idea-68-simplified	evolutionary-prompt-optimization
20	idea-141-simplified	biobert-pet-ner

Table 4: The fixed 20-task subset used for AstaBench evaluation, listed in the retained execution order. The tasks were sampled from the original 40-task test set using random seed 20260707.

Each AstaBench task requires an agent to complete an end-to-end research cycle involving experimental implementation, execution, result analysis, and technical report generation. We use the official task-specific rubrics and LLM-as-judge evaluation pipeline, with MiniMax-M3 serving as the evaluator.

For every rubric criterion, the evaluator independently examines three output facets: the generated report, the generated code, and the supporting artifacts, including datasets, execution logs, figures, and model outputs. Each facet is classified as meets criterion, fails criterion, or no evidence either way. The facet-level assessments are then combined into a binary verdict for the corresponding criterion. The official task score is the arithmetic mean of the binary criterion verdicts, multiplied by 100 for presentation.

In addition to the mean evaluation score, we report the number of successfully completed tasks. A task is considered successfully completed when the agent run terminates without an execution-level failure and produces a valid submission that can be processed and scored by the official evaluator.

Models and agent harnesses.

We evaluate generalization along two complementary axes on ResearchClawBench. In cross-model evaluation, we hold the Codex agent harness fixed and vary the backbone LLM among GPT-5.4, GLM-5.2, and MiniMax-M3. In cross-harness evaluation, we hold the DeepSeek-V4-Flash backbone fixed and vary the surrounding agent implementation among Claude Code, OpenClaw, and OpenScience.

For AstaBench, we evaluate three representative configurations: Claude Code with DeepSeek-V4-Flash, OpenClaw with DeepSeek-V4-Flash, and Codex with GPT-5.4-mini. The complete configurations and corresponding benchmark evaluators are summarized in Table 5.

Benchmark	Setting	Agent harness	Backbone LLM	Evaluator
ResearchClawBench	Cross-model	Codex	GPT-5.4	GPT-5.1
ResearchClawBench	Cross-model	Codex	GLM-5.2	GPT-5.1
ResearchClawBench	Cross-model	Codex	MiniMax-M3	GPT-5.1
ResearchClawBench	Cross-harness	Claude Code	DeepSeek-V4-Flash	GPT-5.1
ResearchClawBench	Cross-harness	OpenClaw	DeepSeek-V4-Flash	GPT-5.1
ResearchClawBench	Cross-harness	OpenScience	DeepSeek-V4-Flash	GPT-5.1
AstaBench	Transfer	Claude Code	DeepSeek-V4-Flash	MiniMax-M3
AstaBench	Transfer	OpenClaw	DeepSeek-V4-Flash	MiniMax-M3
AstaBench	Transfer	Codex	GPT-5.4-mini	MiniMax-M3

Table 5: Model, agent-harness, and evaluator configurations used in the ResearchClawBench and AstaBench experiments.

For every paired comparison, the vanilla setting denotes the original agent without AutoSciRub. The $+{AutoSciRub}$ setting uses the same backbone model, agent harness, task instruction, task-visible data, workspace, and available tools, while additionally performing rubric induction, criterion-level verification, and targeted revision. Accordingly, the paired configurations are controlled for the underlying agent and execution environment, but are not compute-matched, because AutoSciRub introduces the additional model calls required by the method.

Execution and evaluation protocol.

Each task–configuration pair is executed once, producing one scientific report and one set of supporting artifacts. Each resulting submission is then independently evaluated three times using the same benchmark evaluator and evaluation protocol. The arithmetic mean of the three judge scores is used as the task-level score.

For ResearchClawBench, the overall score is computed as the arithmetic mean over all 40 task-level scores, and each domain score is computed over its four constituent tasks. For AstaBench, the mean score is computed over the fixed 20-task subset. Successful-task counts are computed independently from the rubric-based evaluation scores.

Within each paired comparison, the vanilla and $+{AutoSciRub}$ settings receive the same task inputs, task-visible resources, workspace, and tool access. Available tools include the native capabilities of the corresponding agent harness, such as file inspection, code execution, and web search, as well as the external retrieval services used by AutoSciRub. The hidden target paper and the official benchmark evaluation rubric remain inaccessible to both settings during research execution.

We do not impose a shared explicit cap on tokens, tool calls, wall-clock runtime, or monetary cost across heterogeneous agent harnesses. Each system instead operates under the execution, context, and tool constraints of its native environment. Our primary evaluation therefore focuses on output quality rather than budget-normalized efficiency.

Rubric-guided verification and revision use a maximum of three revision rounds with adaptive early stopping. To distinguish the effect of structured rubric guidance from that of additional opportunities for revision, we additionally evaluate a three-round rubric-free self-refinement baseline starting from the same initial reports and using the same underlying research agent. The complete verification, revision, early-stopping, and self-refinement protocols are described in the following subsection.

A.2 Rubric Quality Evaluation

Compared rubric variants and scoring protocol.

For each of the 40 ResearchClawBench tasks, we evaluate two rubric variants: the instruction-derived rubric skeleton produced after Rubric Skeleton Induction, and the task-specific executable rubric produced after Scientific Literature Grounding, Task-Data Exploration, and Criterion Synthesis.

Each rubric is evaluated along four dimensions: Specificity, Actionability, Evidence Verifiability, and Scientific Core Coverage. All dimensions use a five-point ordinal scale, where a higher value indicates stronger rubric quality. We use MiniMax-M3 as the rubric-quality judge for all evaluations. To reduce cross-dimensional interference, we use four standalone judge prompts, each of which evaluates exactly one dimension. The judge receives only one candidate rubric in each call and is explicitly instructed not to score the other three dimensions.

For every judge call, the evaluator is provided with the task instruction, a description of the task-visible data, the candidate rubric, and the official benchmark checklist. The checklist is used only as evaluator-side evidence for identifying the intended scientific success conditions. It is never exposed to the agent during rubric construction, research execution, verification, or revision. The judge is instructed to assess semantic and scientific equivalence rather than lexical overlap with the checklist.

The rubric skeleton and executable rubric are evaluated in separate calls. The judge is not provided with the generated report or the alternative rubric variant. The overall quality score of rubric $\mathcal{R}$ is computed as the unweighted arithmetic mean of the four dimension scores:

Q(\mathcal{R})=\frac{1}{4}\left(s_{\mathrm{spec}}+s_{\mathrm{act}}+s_{\mathrm{ver}}+s_{\mathrm{core}}\right),

where each score is an integer between 1 and 5.

Evaluation dimensions.

Specificity. Specificity measures how precisely the rubric states what is to be done and produced. A specific rubric names the relevant methods or experiments, variables, conditions, metrics, comparators, thresholds, and expected artifacts at a level that makes the intended analysis unambiguous. Specificity concerns the concreteness of the specification; it does not by itself establish that the plan is executable, scientifically valid, or directed at the correct scientific objective.

Actionability. Actionability measures whether a competent agent can execute the rubric using the available inputs without inventing consequential parts of the plan. An actionable rubric provides sufficient operational steps, dependencies, parameter or method choices, decision rules, and expected outputs to support implementation. Actionability does not reward detail that is irrelevant to execution and does not guarantee that an executable plan addresses the task’s scientific core.

Evidence Verifiability. Evidence Verifiability measures whether the rubric requires inspectable and traceable evidence that permits independent verification of both task completion and the resulting scientific claims. Verifiable evidence may include quantitative tables, figures, machine-readable outputs, controls, baseline comparisons, uncertainty estimates, provenance, and explicit links between claims and generated results. This dimension evaluates the checkability and diagnostic value of the evidence rather than the number of artifacts requested.

Scientific Core Coverage. Scientific Core Coverage measures the extent to which the rubric faithfully covers and operationalizes the task’s core scientific success conditions. It evaluates whether the rubric’s objectives, methods or scientifically equivalent alternatives, metrics, comparisons, artifacts, scope, conventions, evidence, and required conclusions collectively target the intended scientific problem. Scores decrease when important success conditions are omitted or underweighted, when the rubric drifts toward an adjacent problem, or when core goals are replaced by non-equivalent proxies. Alternative methods and artifacts receive full credit when they provide scientifically equivalent or stronger evidence for the same core claim.

The four dimensions capture complementary properties. A rubric may be highly specific and actionable but receive a low Scientific Core Coverage score if it precisely operationalizes the wrong analysis. Conversely, a concise rubric may identify the correct scientific core but receive lower Specificity or Actionability scores if important implementation choices remain unspecified. Evidence Verifiability is assessed separately: a scientifically aligned and executable plan may still be difficult to verify when it does not require inspectable outputs that directly support its conclusions.

Complete rubric-quality scoring prompts.

The following four prompts are used independently. In the JSON templates, the value of score is replaced with an integer from 1 to 5. Rubric item identifiers refer to goal IDs for rubric skeletons and criterion IDs for executable rubrics.

Complete Rubric-Quality Scoring Prompts

The following four prompts independently evaluate Specificity, Actionability, Evidence Verifiability, and Scientific Core Coverage. Each judge call receives one candidate rubric and evaluates exactly one dimension.

⬇

1 You are a senior scientific-methods reviewer. Evaluate exactly one construct: SPECIFICITY.

3 Definition:

4 Specificity is the precision with which the rubric states what is to be done and produced. It concerns whether the relevant methods or experiments, variables, conditions, metrics, comparators, thresholds, and expected artifacts are concrete enough to make the intended analysis unambiguous.

6 Evaluation rules:

7 1. Inspect the rubric’s core items rather than rewarding length or technical jargon.

8 2. Credit concrete methods, conditions, variables, metrics, comparisons, thresholds, named outputs, and artifact requirements when they remove genuine ambiguity.

9 3. Do not reward irrelevant detail or copied wording from the task or benchmark checklist.

10 4. Do not lower the score merely because the specified plan may be difficult to execute, scientifically questionable, weakly evidenced, or misdirected; those properties belong to other dimensions.

11 5. Cite rubric item IDs and identify the most consequential specified and unspecified details.

13 Score anchors:

14 1 = The rubric is almost entirely generic; core methods, conditions, metrics, and outputs are not concretely specified.

15 2 = A few concrete details are present, but most consequential aspects remain ambiguous.

16 3 = The main analysis is identifiable, but important conditions, metrics, comparisons, thresholds, or artifacts are underspecified.

17 4 = Nearly all core details are explicit; only minor ambiguities remain.

18 5 = The core analysis is precise and unambiguous across methods, conditions, variables, metrics, comparisons, thresholds, and expected artifacts.

20 Return JSON only. Set "score" to an integer from 1 to 5:

21 {

22 "dimension": "specificity",

23 "score": 1,

24 "evidence": "Concise justification citing rubric item IDs.",

25 "specified_strengths": ["..."],

26 "remaining_ambiguities": ["..."]

27 }

29 TASK INSTRUCTION:

30 [task_instruction]

32 AVAILABLE-DATA DESCRIPTION:

33 [available_data]

35 BENCHMARK CHECKLIST -- EVALUATOR-ONLY CONTEXT:

36 [benchmark_checklist]

38 RUBRIC TO SCORE:

39 [rubric]

⬇

1 You are a senior scientific-methods reviewer. Evaluate exactly one construct: ACTIONABILITY.

3 Definition:

4 Actionability is the degree to which a competent agent can execute the rubric using the available inputs without inventing consequential parts of the plan. It concerns operational steps, dependencies, method and parameter choices, decision rules, and deliverables that reduce implementation uncertainty.

6 Evaluation rules:

7 1. Determine whether the rubric supplies an executable route from the available data to the required outputs.

8 2. Identify consequential methods, parameters, dependencies, preprocessing choices, decision rules, or deliverables that the agent would otherwise have to invent.

9 3. Credit flexibility when multiple choices are genuinely interchangeable and the rubric supplies a sound decision rule or acceptable range.

10 4. Do not reward detail that does not help execution. Do not judge whether the chosen scientific target is the correct one; that belongs to Scientific Core Coverage.

11 5. Cite rubric item IDs and focus on implementation consequences.

13 Score anchors:

14 1 = No executable route is supplied; the agent must design nearly the entire plan.

15 2 = Major methods, dependencies, parameters, or decisions must be invented.

16 3 = An executable outline exists, but some consequential implementation choices remain unresolved.

17 4 = The core workflow is directly executable, with only minor or low-impact choices left open.

18 5 = An end-to-end operational plan specifies all consequential steps, dependencies, decision rules, and outputs needed for execution.

20 Return JSON only. Set "score" to an integer from 1 to 5:

21 {

22 "dimension": "actionability",

23 "score": 1,

24 "evidence": "Concise justification citing rubric item IDs.",

25 "executable_elements": ["..."],

26 "missing_decisions": ["..."]

27 }

29 TASK INSTRUCTION:

30 [task_instruction]

32 AVAILABLE-DATA DESCRIPTION:

33 [available_data]

35 BENCHMARK CHECKLIST -- EVALUATOR-ONLY CONTEXT:

36 [benchmark_checklist]

38 RUBRIC TO SCORE:

39 [rubric]

⬇

1 You are a senior scientific-methods reviewer. Evaluate exactly one construct: EVIDENCE VERIFIABILITY.

3 Definition:

4 Evidence Verifiability is the degree to which the rubric requires inspectable and traceable evidence that allows an independent reviewer to verify task completion and evaluate whether the scientific claims follow from the generated results.

6 Evaluation rules:

7 1. Inspect required quantitative tables, figures, machine-readable outputs, controls, baseline comparisons, uncertainty estimates, provenance, and claim-to-evidence links when scientifically applicable.

8 2. Judge the diagnostic and confirmatory value of evidence, not the number of files.

9 3. Credit explicit must-show criteria, metrics, uncertainty, comparison targets, and traceable paths that make an artifact independently checkable.

10 4. Penalize rubrics that allow conclusions to be asserted without supporting results, or request decorative outputs that cannot test the claims.

11 5. Do not judge whether the rubric targets the correct scientific goal; that belongs to Scientific Core Coverage.

13 Score anchors:

14 1 = Conclusions can be asserted without any inspectable supporting evidence.

15 2 = Outputs are requested only vaguely or lack checkable criteria and claim links.

16 3 = Some core results are inspectable, but important claims, comparisons, uncertainty, or diagnostics remain weakly supported.

17 4 = Core claims are supported by checkable quantitative artifacts and appropriate diagnostics; only minor traceability gaps remain.

18 5 = The evidence chain is complete and traceable from method to result to claim, including controls, uncertainty, comparisons, and provenance where relevant.

20 Return JSON only. Set "score" to an integer from 1 to 5:

21 {

22 "dimension": "evidence_verifiability",

23 "score": 1,

24 "evidence": "Concise justification citing rubric item IDs.",

25 "verifiable_artifacts": ["..."],

26 "verification_gaps": ["..."]

27 }

29 TASK INSTRUCTION:

30 [task_instruction]

32 AVAILABLE-DATA DESCRIPTION:

33 [available_data]

35 BENCHMARK CHECKLIST -- EVALUATOR-ONLY CONTEXT:

36 [benchmark_checklist]

38 RUBRIC TO SCORE:

39 [rubric]

⬇

1 You are a senior scientific-methods reviewer. Evaluate exactly one construct: SCIENTIFIC CORE COVERAGE.

3 Definition:

4 Scientific Core Coverage is the extent to which the rubric faithfully covers and operationalizes the task’s core scientific success conditions through aligned objectives, scientifically equivalent methods, metrics, comparisons, artifacts, scope, conventions, evidence, and required conclusions. It excludes material omission, goal drift, and substitution by adjacent but non-equivalent proxies.

6 Evaluation rules:

7 1. Infer the task’s core scientific success conditions from the instruction and available data. Use the benchmark checklist and its criterion weights, when available, only as evaluator-side evidence of importance.

8 2. Map each core condition to rubric items and classify it as aligned, scientifically equivalent, partial, proxy, missing, or conflicting.

9 3. Judge semantic and scientific-function equivalence, never wording overlap. Alternative methods and artifacts deserve full credit when they provide equivalent or stronger evidence for the same core claim.

10 4. Penalize omission or underweighting of high-importance goals, goal drift, non-equivalent proxy substitution, artifact or metric mismatch, scope dilution, incompatible conventions, and conclusion anchoring.

11 5. Missing low-level implementation detail alone belongs to Specificity or Actionability and must not reduce this score when the scientific target remains intact.

13 Score anchors:

14 1 = The rubric operationalizes a substantially different problem, contradicts a core success condition, or is dominated by non-equivalent proxies.

15 2 = Major omissions, conflicts, or proxy substitutions affect one or more core or high-importance success conditions.

16 3 = The main objective is recognizable, but an important success condition is missing, weakened, underweighted, or replaced.

17 4 = Core success conditions are faithfully operationalized; only minor or peripheral mismatches remain, and valid equivalent alternatives are used appropriately.

18 5 = All core success conditions are tightly covered by aligned or scientifically equivalent objectives, methods, metrics, comparisons, artifacts, scope, and evidence, with no material drift, omission, or proxy replacement.

20 Return JSON only. Set "score" to an integer from 1 to 5. The allowed values of "importance" are "core", "important", and "peripheral". The allowed values of "status" are "aligned", "equivalent", "partial", "proxy", "missing", and "conflicting":

21 {

22 "dimension": "scientific_core_coverage",

23 "score": 1,

24 "evidence": "Concise justification citing rubric item IDs and core consequences.",

25 "criterion_audit": [

26 {

27 "criterion": "...",

28 "importance": "core",

29 "status": "aligned",

30 "rubric_items": ["R1"],

31 "rationale": "..."

32 }

33 ],

34 "core_gaps_or_drift": ["..."]

35 }

37 TASK INSTRUCTION:

38 [task_instruction]

40 AVAILABLE-DATA DESCRIPTION:

41 [available_data]

43 BENCHMARK CHECKLIST -- EVALUATOR-ONLY CONTEXT:

44 [benchmark_checklist]

46 RUBRIC TO SCORE:

47 [rubric]

A.3 Rubric-Guided Revision Evaluation

Experimental configuration and shared initialization.

We compare rubric-guided revision with rubric-free holistic self-refinement on all 40 ResearchClawBench tasks. All checkpoint-0 reports, verification calls, targeted revisions, and rubric-free revisions are executed with OpenClaw using deepseek/deepseek-v4-flash. There is no separately instantiated verifier model. Instead, the same OpenClaw agent invokes the report-verifier skill within the task session.

Both revision strategies start from the same set of 40 frozen checkpoint-0 reports and figures. These checkpoint-0 outputs are generated under the task-specific executable rubric after planning, literature grounding, experiment design, experiment execution, figure generation, and initial report writing, but before any verifier call or verifier-driven revision. Their mean ResearchClawBench score is 18.309083, reported as 18.31 in the main paper.

For the rubric-free condition, the canonical checkpoint-0 report.md file and its figures are copied into a new isolated workspace. SHA-256 hashes are computed before and after copying to verify that the starting report and figures are unchanged. We do not claim that the two strategies share an identical complete workspace: checkpoint-0 code and output directories are not frozen or copied because they may contain rubric-related state and later feedback. Instead, the rubric-free workspace is supplied again with the original instruction, task data, and related work, and the agent may create new code, outputs, and analyses during refinement.

Criterion-level verification.

Before each guided revision, the current artifact is evaluated by the same OpenClaw agent through a verification-only call. The verifier examines the current report, generated experiments and results, figures and tables, quantitative values, conclusions, and mechanistic or analytical explanations. It also checks whether the artifacts required by the task-specific executable rubric are present or explicitly represented in the report, including their filenames, titles, captions, display names, and detailed descriptions.

The verifier is instructed to distinguish generated experimental evidence from statements supported only by prior literature. Conclusions are considered adequately supported only when they follow from evidence produced within the current task execution.

For each scientific goal, the verifier records whether the report contains the required experiment, results, figure or table, conclusion, and mechanism or analytical explanation. It also evaluates individual rubric items and produces stable feedback items specifying the remaining issue and the required corrective action.

The top-level overall_pass field is an LLM judgment rather than a deterministic conjunction of all criterion-level Boolean values. An artifact passes when all scientific goals have sufficient experimental support, interpretation, and conclusion coverage, and all high-priority rubric items are either satisfied or explicitly and scientifically justified by the generated evidence. Lower-priority items are not required to pass individually. The verifier does not produce a separate structured not_applicable state; an inapplicable requirement may instead be justified in free text while the corresponding structured field remains Boolean.

The expected verifier output follows the schema below.

⬇

1 {

2 "source": "rcbench_report_verifier",

3 "overall_pass": false,

4 "goal_checks": [

5 {

6 "goal_id": "G1",

7 "pass": false,

8 "coverage": {

9 "experiment": true,

10 "results": true,

11 "figure_or_table": false,

12 "conclusion": true,

13 "mechanism_or_analysis": false

14 },

15 "missing": ["..."],

16 "fixes": ["..."]

17 }

18 ],

19 "rubric_item_checks": [

20 {

21 "rubric_item_id": "R1",

22 "pass": false,

23 "missing": ["..."],

24 "fixes": ["..."]

25 }

26 ],

27 "feedback_items": [

28 {

29 "feedback_id": "F001",

30 "goal_id": "G1",

31 "rubric_item_id": "R1",

32 "issue": "...",

33 "required_action": "..."

34 }

35 ],

36 "artifact_checks": [

37 "path/name, present?, described?, issue"

38 ],

39 "required_iterations": ["..."]

40 }

The runner reads the top-level overall_pass value directly and does not independently reconstruct the pass decision from the individual checks. If the verifier output is missing or cannot be parsed, the runner treats the artifact as not passing and proceeds to revision. No separate JSON-schema validation or verifier-specific retry is performed. A task run is marked as failed only when the OpenClaw process exits with a nonzero status.

Targeted revision and early stopping.

If verification does not pass, the current report_rubric_check.json is supplied to the same OpenClaw agent for one targeted revision. The revision call has access to the current workspace, INSTRUCTIONS.md, the current report, the executable rubric, goal-level experiment plans, existing code, outputs, figures, and the current verifier feedback.

The agent may modify code, rerun or add experiments and analyses, regenerate figures or tables, correct numerical results, revise unsupported claims, and update the complete report/report.md. The revision call is instructed not to invoke the verifier itself; verification and revision are executed as separate agent calls.

We allow at most three revision rounds. The resulting execution sequence is

V^{(0)}\rightarrow R^{(1)}\rightarrow V^{(1)}\rightarrow R^{(2)}\rightarrow V^{(2)}\rightarrow R^{(3)}\rightarrow V^{(3)},

where $V^{(t)}$ denotes verification of checkpoint $t$ , and $R^{(t)}$ denotes the revision producing checkpoint $t$ . Consequently, each task receives at most three revision calls and four verification calls. The procedure terminates immediately when a verification call returns overall_pass=true. If the artifact does not pass earlier, a final verification is performed after the third revision.

Adaptive stopping and checkpoint aggregation.

Rubric-guided revision generates and evaluates only reports that are actually produced. Across the 40 tasks, the experiment contains 103 distinct report checkpoints: 40 checkpoint-0 reports, 40 checkpoint-1 reports, 17 checkpoint-2 reports, and 6 checkpoint-3 reports.

When a task passes verification, no artificial reports are generated for later revision budgets. Instead, its most recent real report and its previously obtained score are carried forward when computing later checkpoint-level means. The carried-forward report is not rerun or rescored. Thus, every reported guided checkpoint mean is computed over all 40 tasks, while only tasks that have not yet stopped produce a new report at the next revision round.

Each actually generated report checkpoint is independently scored three times by the GPT-5.1 ResearchClawBench judge, and the arithmetic mean of the three scores is used as its task-level score.

Rubric-free holistic self-refinement.

The rubric-free baseline starts from the same frozen checkpoint-0 report and figures and uses the same OpenClaw and deepseek/deepseek-v4-flash configuration. It is explicitly prohibited from accessing, constructing, inferring, or searching for the task-specific executable rubric, the hidden benchmark checklist, the target study, rubric scores, or prior verifier feedback.

At each round, the agent invokes the holistic-self-review skill and performs a holistic scientific review followed immediately by one revision within the same agent call. The agent may use tools, write code, rerun or add experiments and analyses, modify figures and tables, revise results and claims, and update the complete report.

Unlike rubric-guided revision, rubric-free self-refinement does not use verifier-based early stopping. All 40 tasks execute three review–revision cycles, producing 40 real reports at each of checkpoints 1, 2, and 3. In the first round, the runner requires between one and six prioritized issues. In the second and third rounds, an empty issue list and a no-op revision are allowed when no substantive deficiency is identified. Each resulting report is independently scored three times by the same GPT-5.1 judge.

This baseline controls for repeated opportunities to inspect and revise the report, but it is not a strictly compute-matched or workspace-identical control. Guided revision separates verification and revision into two calls and uses adaptive stopping, whereas rubric-free refinement combines review and revision into one call and always performs three rounds. Guided revision also retains the original code and output state, while the rubric-free condition starts from the frozen report and figures in a newly initialized workspace.

Appendix B Rubric Output Formats

This section presents the output formats of the instruction-derived rubric skeleton and the task-specific executable rubric. One complete entry is shown for each format; the remaining entries are omitted for brevity.

Instruction-derived rubric skeleton.

The rubric skeleton represents the task as a set of atomic scientific goals. Each goal contains a unique identifier, a concise title, and a concrete task description.

⬇

1 {

2 "goals": [

3 {

4 "goal_id": "G1",

5 "title": "Characterize spatial concentration of cloud-seeding projects",

6 "task": "Analyze the geographic distribution of cloud-seeding projects across U.S. states using the published records, and identify which states account for the majority of reported activity."

7 },

8 {

9 "...": "Additional scientific goals are omitted."

10 }

11 ]

12 }

Task-specific executable rubric.

The executable rubric augments the scientific goals with concrete experiments, required evidence artifacts, metrics, comparisons, data sources, expected conclusions, priorities, and execution constraints.

⬇

1 {

2 "source": "rcbench_experiments_design",

3 "task_summary": "Reproduce and validate the central empirical conclusions from a published NOAA weather-modification dataset of U.S. cloud seeding projects (2000-2025). Generate spatial concentration maps, annual activity dynamics, purpose composition, and agent-apparatus deployment patterns using transparent, script-based analysis.",

4 "selected_environment": ".venv-rcb-base",

5 "environment_reason": "The selected environment provides the packages required for tabular analysis, statistical computation, and geospatial visualization.",

6 "selected_strategy": "Independently compute the required statistics and figures from the structured dataset and synthesize the results into an empirical assessment.",

8 "rubric_items": [

9 {

10 "id": "R1",

11 "linked_goal_ids": [

12 "G1"

13 ],

14 "goal": "Quantify state-level spatial concentration of cloud seeding projects and produce a choropleth map showing geographic distribution.",

15 "experiment_plan_path": "outputs/goals/G1/experiment_plan.md",

16 "experiments_to_run": [

17 "Group records by state and count projects",

18 "Compute rank, percentage share, and Gini coefficient",

19 "Merge state counts with GeoJSON and generate a choropleth map",

20 "Create a state-by-purpose heatmap"

21 ],

22 "required_artifacts": [

23 {

24 "type": "figure",

25 "name": "spatial_distribution.png",

26 "expected_path": "report/images/spatial_distribution.png",

27 "must_show": "A choropleth map of U.S. states colored by the number of cloud seeding projects, showing strong western concentration and identifying California, Colorado, Texas, and Utah among the leading states.",

28 "description_requirement": "Explain the geographic pattern of cloud seeding activity and the factors associated with western concentration."

29 },

30 {

31 "type": "figure",

32 "name": "state_purpose_heatmap.png",

33 "expected_path": "report/images/state_purpose_heatmap.png",

34 "must_show": "A state-by-purpose heatmap demonstrating how project purposes vary across states.",

35 "description_requirement": "Describe how project purposes vary across states and regions."

36 },

37 {

38 "type": "table",

39 "name": "top_states_table",

40 "expected_path": "report/report.md",

41 "must_show": "A ranked list of the leading states with project counts and percentage shares.",

42 "description_requirement": "Summarize the state-level concentration statistics."

43 }

44 ],

45 "metrics_or_values": [

46 "Projects per state",

47 "Percentage share of the top three and top five states",

48 "Gini coefficient"

49 ],

50 "baselines_or_comparisons": [

51 "Expected geographical concentration in water-stressed western states"

52 ],

53 "data_sources": [

54 "data/dataset1_cloud_seeding_records/cloud_seeding_us_2000_2025.csv[state]",

55 "data/dataset1_cloud_seeding_records/us_states.geojson"

56 ],

57 "expected_conclusions": [

58 "Cloud seeding is heavily concentrated in western U.S. states",

59 "California has the highest project count",

60 "The spatial distribution reflects water-resource management priorities"

61 ],

62 "priority": "high",

63 "risk_if_missing": "Loss of the primary spatial evidence supporting geographic concentration claims."

64 },

65 {

66 "...": "Additional executable rubric items are omitted."

67 }

68 ],

70 "execution_plan": [

71 {

72 "...": "Task-level execution steps are omitted."

73 }

74 ],

76 "claims_to_avoid": [

77 "Do not claim causal relationships between cloud seeding and precipitation changes.",

78 "... additional unsupported claims are omitted ..."

79 ]

80 }