Intern-S2-Preview: Scientific Agentic Foundation Model

Scientific time series often exhibit substantial variations in sequence length, sampling frequency, and channel dependency, making efficient and expressive modelling challenging. Intern-S2-Preview-397B upgrades the time series encoder over Intern-S1-Pro with improved long-sequence processing efficiency and enhanced multi-channel representation learning.

As illustrated in Figure 2(a), the input time series is first partitioned into temporal chunks, enabling localized processing of long sequences. Each chunk is processed by a compressive patching module, which consists of normalization, CNN-based local feature extraction, and Q-Former based temporal compression. During normalization, channel-wise mean and standard deviation are retained as auxiliary statistics. The extracted local representations are then divided into temporal patches, where each patch is compressed by a Q-Former with learnable queries into a fixed number of tokens. By dynamically adjusting the temporal patching process according to input length, the encoder maintains a controllable output sequence length for heterogeneous long time series. Compared with the time series encoder in Intern-S1-Pro, which directly aggregated multi-channel representations through mean pooling, Intern-S2-Preview-397B introduces a channel-wise Transformer encoder to model inter-channel dependencies before being fed into the Transformer encoder body for global temporal context modelling. The connection between the time series encoder and the LLM remains unchanged.

Compared with the previous version used in Intern-S1-Pro, the upgraded encoder increases the maximum supported input length from approximately 240,000 to 300,000 time steps. At the maximum sequence length, it achieves approximately $5\sim 6\times$ faster inference while reducing GPU memory consumption to around 20% of the previous version. Beyond improving the processing of long sequences, the new architecture also enables effective modelling of signals with high-frequency but short sequence lengths, a setting not supported by Intern-S1-Pro. This capability further expands the disciplinary coverage of the time series module, extending its existing support for astronomy, geoscience, neuroscience, physiological signal analysis, and bioacoustics to include radar signal analysis ($\sim$MHz).

Beyond time series understanding, time series forecasting is essential for scientific applications as it enables models to predict future system states and support a broader range of scientific tasks. Intern-S2-Preview-397B integrates a time series forecasting module to enable unified time series understanding and generation within the multimodal LLM framework. By introducing a dedicated numerical forecasting branch rather than generating values as discrete text tokens, the model preserves numerical fidelity while maintaining computational efficiency.

As illustrated in Figure 2(b), the forecasting module introduces a forecasting branch conditioned on multimodal representations from the LLM and the time series encoder. Semantic context from the LLM and numerical temporal representations from the time series encoder are selectively extracted by Q-Former and integrated to condition a causal Transformer forecaster via cross-attention for future sequence generation. A horizon predictor further interprets the forecasting instruction and determines the required prediction length, enabling flexible forecasting across different horizons.

Long-reasoning reinforcement learning is particularly susceptible to the long-tail distribution of response lengths. In a synchronous rollout pipeline, a small number of exceptionally long generations may delay the completion of an entire batch, leaving most GPUs idle while waiting for stragglers [111, 62, 30]. Existing systems commonly address this problem either by co-locating training and inference with partial rollouts [67, 111, 62], or by fully disaggregating rollout generation and policy optimization onto separate GPU pools [30].

After evaluating the computational characteristics of the different RL stages and our infrastructure, we develop a co-located partial-rollout system based on the XTuner training engine and the LMDeploy inference engine. As illustrated in Figure 7, the inference engine is continuously supplied with new requests during rollout generation to maintain high GPU utilization. Once the number of completed trajectories is sufficient to form a training batch, the remaining in-flight rollouts are paused at their current generation positions rather than aborted or discarded. Their generated prefixes and rollout metadata are retained, while the same GPU pool switches from inference to policy training. After the policy update, the training states are offloaded, the updated model parameters are synchronized to the inference engine, and the paused requests resume generation from their retained prefixes. Only completed trajectories are admitted into the current training batch.

This pause-and-resume mechanism avoids waiting for long-tail generations while preserving the computation already spent on unfinished responses. It also avoids the difficult producer–consumer balancing problem commonly encountered in fully asynchronous systems with disaggregated training and inference resources. However, because a resumed trajectory may contain segments generated before and after one or more policy updates, different tokens within the same trajectory can originate from different behavior-policy versions. We therefore record the behavior-policy version and generation-time log-probability for every sampled token.

For token $y_{i,t}$ in trajectory $i$, we define the importance-sampling ratio as

where $\pi_{\mathrm{beh}(i,t)}$ denotes the behavior-policy version that generated token $y_{i,t}$, and $\pi_{\theta}$ denotes the current learner policy. We explicitly bound trajectory staleness: a trajectory is discarded if its oldest retained segment was generated more than three policy updates before the current learner.

Following the clipped importance-weight formulation of [55], we truncate the importance ratio as

The clipped ratio is subsequently used as a detached importance weight in the REINFORCE objective. Unlike PPO-style clipping, which clips the surrogate objective and may completely suppress gradients from tokens outside the trust region, clipping the importance weight bounds update variance while retaining a nonzero policy-gradient contribution from every unmasked token.

For MoE policies, training–inference inconsistency arises from both expert-routing differences and numerical discrepancies between the two execution engines. We employ Rollout Routing Replay (R3) [52] to address the former: the expert selections made by LMDeploy during rollout are recorded and replayed by XTuner when evaluating the corresponding tokens. This ensures that rollout and training follow the same expert paths. Following Intern-S1-Pro [112], we additionally use an aligned mixed-precision configuration in which expert linear layers operate in FP8, the remaining layers use BF16, and numerically sensitive operations, including apply_rope, RMSNorm, the MoE router, recurrent states in Gated DeltaNet [95], and the language-model head, are computed in FP32.

After routing replay and operator-level alignment, a small number of tokens may still exhibit large probability discrepancies because of residual numerical differences. Inspired by KPop [44], we detect these outliers using the bidirectional binary KL divergence. Let $p_{i,t}^{\mathrm{train}}$ and $p_{i,t}^{\mathrm{rollout}}$ denote the sampled-token probabilities produced by the training and rollout engines under matched model parameters and replayed routing decisions. We define

and construct the token mask

R3 removes discrete expert-routing mismatch, whereas the BKL mask filters the remaining token-level numerical outliers. The clipped importance weights in Equation (11) and token masks in Equation (13) are incorporated into the unified RL objective described in Section 4.3.5.

Long-CoT reasoning models frequently exhibit overthinking on relatively simple problems, producing unnecessarily long reasoning trajectories even when the correct solution can be reached with substantially less computation [16, 61]. Existing approaches improve reasoning efficiency through explicit length-aware rewards or constraints [73, 3], query-adaptive length penalties [87], or a separate length-control fine-tuning stage [49, 103]. Although effective, reward-based approaches introduce auxiliary optimization objectives that may conflict with task rewards, whereas additional fine-tuning stages complicate the post-training pipeline and may disturb capabilities acquired during earlier RL stages.

We introduce an adaptive length regularization method that directly reweights the advantages of positive responses without adding an independent reward signal. The method follows two principles. First, we do not impose length regularization on negative responses. Since an incorrect response may fail for many different reasons, penalizing its length can prematurely suppress potentially useful exploration and consequently degrade model performance. Second, we activate length regularization only when the model achieves a sufficiently high pass rate on the corresponding query. This design allows the model to freely explore difficult queries and encourages concise reasoning only after it has largely mastered them.

For each query $q$, let $\mathcal{G}_{q}=\{1,\ldots,G\}$ index a group of $G$ sampled responses, and let $\hat{A}_{i}$ denote the original advantage of response $i$. We define the set of positive responses as

The regularized advantage is given by

where $\tau$ controls the minimum fraction of positive responses required to activate length regularization. The length-dependent weight $w_{i}$ is defined as

with

where $L_{i}$ is the reasoning length of response $i$, $\alpha$ specifies the minimum weight assigned to long responses, $\gamma$ controls the shape of the length-dependent decay, and $\epsilon$ ensures numerical stability.

Among positive responses, shorter solutions receive larger weights, whereas longer solutions are down-weighted while retaining positive advantages. The normalization term approximately preserves the total positive advantage mass, thereby changing the relative preference among successful responses without substantially altering the overall optimization scale. If the positive-response ratio is below $\tau$, or if a response has a non-positive advantage, its advantage remains unchanged. The method therefore adaptively transitions from exploration on difficult queries to efficiency optimization on queries that the model has already mastered.

As shown in Figure 8, we compare Intern-S2-Preview-35B trained with and without adaptive length regularization. Both settings achieve comparable reward curves, while adaptive length regularization substantially reduces the average output length. These results demonstrate that the proposed method improves reasoning efficiency without sacrificing model performance.

Although the co-located partial rollout system substantially improves GPU utilization, rollout generation remains one of the most time-consuming stages of RL training because long reasoning trajectories must still be generated autoregressively. Speculative decoding provides a complementary approach for accelerating this process. A lightweight draft model first proposes multiple candidate tokens, which are subsequently verified in parallel by the current policy model through an exact rejection-sampling procedure [43, 14]. Since this verification procedure preserves the sampling distribution of the policy model, speculative decoding accelerates rollout generation without introducing additional off-policy bias. Recent studies have explored speculative decoding for RL rollouts through concurrency-aware online draft learning, continuously evolving draft models, tree-structured rollout caches, and system-level integration with RL infrastructure [101, 15, 13, 37].

A central challenge in applying speculative decoding to RL is that the policy model evolves continuously during training. A fixed draft model therefore becomes increasingly stale as the policy is updated, resulting in a growing mismatch between their output distributions and a progressive decline in the token acceptance rate. To address this issue, we train the draft model online using trajectories generated by the latest policy. At each RL iteration, the draft model is updated using the token distributions of the current policy on newly collected rollout states, while gradients are stopped through the policy model. This online adaptation allows the draft model to continuously track the evolving rollout distribution throughout RL training.

We train the draft model using the hybrid LK Loss [63], which combines the stable optimization behavior of forward KL divergence with the direct acceptance-rate optimization of total variation distance. For the $k$-th draft position at a rollout state $s_{t,k}$, we denote the target-policy and draft-model distributions as

where both distributions are computed under the rollout sampling temperature and $\operatorname{sg}[\cdot]$ denotes the stop-gradient operator. The forward KL divergence and total variation distance are respectively defined as

and

Under lossless speculative sampling, the expected token acceptance probability is equal to the overlap between the target and draft distributions:

Following the hybrid LK formulation, we combine the two divergence objectives as

The mixing coefficient is adaptively determined by the acceptance rate:

where $\bar{\alpha}_{k}$ is the acceptance rate for the $k$-th draft position aggregated over the sequence and batch dimensions. The complete online draft-training objective is

where $K$ is the number of predicted draft positions and $\mathcal{T}_{k}$ contains the valid training positions for the $k$-th draft step. In our implementation, the draft model predicts tokens at $K=4$ future positions, and we set the adaptive coefficient hyperparameter to $\eta=3$.

When the draft model is poorly aligned with the current policy, the acceptance rate $\bar{\alpha}_{k}$ is low and $\lambda_{k}$ approaches one. The objective is therefore dominated by the forward KL term, which provides smooth and well-scaled gradients for rapidly aligning the draft distribution with the evolving policy. As the draft model becomes better aligned and the acceptance rate increases, $\lambda_{k}$ decreases and the TV component receives a larger weight. Since minimizing TV distance is equivalent to maximizing the distributional overlap, the objective gradually shifts from stable distribution matching to direct acceptance-rate optimization.

With online draft adaptation and the hybrid LK objective, the acceptance rate continues to improve as RL training progresses instead of degrading as the policy evolves. In our large-scale training runs, speculative decoding ultimately delivers an approximately $2\times$ speedup in rollout generation and a $1.7\times$ end-to-end speedup for the overall RL training pipeline. These results demonstrate that online draft learning provides a lossless and effective acceleration mechanism that complements our partial rollout system.

We perform RL for Intern-S2-Preview on mixtures of heterogeneous tasks, which differ in structure, solution diversity, and uncertainty of policy exploration. This induces distinct entropy regimes under the same policy, making global or token-level entropy regulation inadequate for their heterogeneous exploration requirements [94, 66, 20, 22]. This heterogeneity further makes group-based policy optimization methods induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable.

We apply Group-level Entropy-Controlled Policy Optimization (GEPO) [21], which uses group-level entropy, estimated directly from existing grouped samples, as a diagnostic signal to identify and mitigate the optimization bias induced by entropy heterogeneity. Specifically, GEPO attenuates positive advantages in low-entropy groups to prevent over-exploitation that would further amplify the entropy gap, while attenuating negative advantages in high-entropy groups to avoid prematurely suppressing exploration. This scaling is asymmetric because low-entropy groups are more susceptible to aggressive intervention, which may trigger length collapse, and therefore require milder attenuation than high-entropy groups.

In GRPO and RLOO, given group responses $\{y_{1},\ldots,y_{K}\}$ sampled from $\pi_{\theta}(\cdot|x)$ for each prompt $x$, we define group-level entropy as

Then, we shape the original advantage $\{A_{i}\}_{i=1}^{K}$ for each response as

where $\alpha_{\text{high}}\in(0,1)>\alpha_{\text{low}}\in(0,1)$ are the scaling coefficients, and $\mathcal{H}_{\text{low}}^{(t)}$ and $\mathcal{H}_{\text{high}}^{(t)}$ denote the lower and upper entropy thresholds at training step $t$.

Instead of forcing heterogeneous tasks toward a shared entropy target, GEPO preserves task-dependent exploration regimes while rebalancing their effective contributions to policy updates. It requires neither explicit task annotations nor additional rollouts and can be directly integrated into existing group-based policy optimization pipelines, providing a scalable mechanism for stable joint optimization over heterogeneous post-training tasks.

Our reasoning RL follows the leave-one-out REINFORCE formulation used in Intern-S1-Pro [112], augmented with the stabilization and efficiency techniques introduced above. For each query, we sample a group of $G$ responses and obtain their sequence-level verifier rewards $\{R_{i}\}_{i=1}^{G}$. Following the dynamic sampling strategy of DAPO [96], query groups whose rewards are all identical are filtered out online and replaced with newly sampled groups. For each retained group, the initial leave-one-out advantage is computed as

We then apply the two advantage-shaping mechanisms described in the preceding sections. GEPO first adjusts the group-relative advantages according to group-level entropy to balance exploration across heterogeneous tasks. Adaptive length regularization is subsequently applied to the entropy-adjusted advantages, encouraging shorter successful reasoning trajectories only for query groups whose positive-response ratios exceed the activation threshold. The final advantage used for policy optimization can be summarized as

where $\mathcal{R}_{\mathrm{GEPO}}$ and $\mathcal{R}_{\mathrm{len}}$ denote the entropy-control and adaptive length-regularization transformations defined above, respectively. This ordering ensures that the length-dependent weights act on the final entropy-adjusted advantages rather than modifying the verifier rewards.

Given a partial-rollout training buffer $\mathcal{B}$, we optimize the policy using

where $\bar{\rho}_{i,t}$ is the clipped token-level importance weight defined in Equation (11), $m_{i,t}^{\mathrm{BKL}}$ is the numerical-consistency mask defined in Equation (13), and $\operatorname{sg}[\cdot]$ denotes the stop-gradient operator. The sequence-level advantage $\widetilde{A}_{i}$ is shared by all policy-generated tokens in response $y_{i}$.

This objective combines three complementary stabilization mechanisms. The clipped importance weight corrects the policy mismatch introduced by pause-and-resume partial rollouts and repeated mini-batch updates. R3 aligns the expert-routing decisions of the rollout and training engines, while the BKL mask removes the remaining numerical outliers after routing and operator alignment. Meanwhile, GEPO and adaptive length regularization reshape the sequence-level advantages to balance task-dependent exploration and reasoning efficiency. Speculative decoding preserves the sampling distribution of the policy and therefore accelerates rollout generation without modifying Equation (29).

We use the Muon optimizer [39, 48] with a learning rate of $1\times 10^{-6}$ and a weight decay of $0.01$. Each rollout batch contains $8{,}192$ completed responses and is optimized through 8 mini-batch update steps. The maximum generation length is set to $65{,}536$ tokens. Together, this training configuration and the stabilization mechanisms described above support efficient and stable reasoning RL over heterogeneous scientific and general-purpose tasks.

As illustrated in Figure 9, our infrastructure consists of a unified rollout runtime and a trace-aware experience-assembly layer. The former composes heterogeneous harnesses with executable tasks and runs them under a common session contract; the latter joins semantic trajectories and verifier feedback with the exact token-level evidence required by policy optimization.

This serving path implements a token-in–token-out (TITO) interface. For each model call, the Session Server reuses the exact tokenized prefix already recorded for the session, tokenizes only newly appended context, and sends the resulting token IDs directly to the inference engine. The returned tokens and policy statistics are captured from the same response stream delivered to the agent. For sparse MoE models, Rollout Router Replay (R3) additionally records the rollout-time expert choices for reuse during training. TITO therefore preserves the sampled token sequence, while R3 preserves the conditional computation path that produced it.

The Trace Store organizes each session as an incremental PrefixTree. Each node represents a newly appended context delta or assistant response and stores its token IDs, labels, rollout log probabilities, and router experts. Longest-prefix matching reuses the stable history of a session and appends only newly observed segments. When a trajectory is selected for training, the store materializes the corresponding root-to-leaf path. System instructions, user messages, and tool observations are masked from the loss, while eligible policy-generated segments retain their training labels.

Beyond incremental storage, the PrefixTree preserves the lineage and exact boundaries of model calls across multi-turn and branching interactions. It thus establishes a stable correspondence between a semantic agent action and its rollout-time token span. Experience assembly joins Replay Buffer records and token traces by session and segment, computes advantages, and exports model-ready experiences to the common RL trainer. This correspondence also provides the basis for the process-aware credit assignment described below.

We construct the task distribution from two complementary sources. For specialized coding and terminal capabilities, we curate executable tasks from publicly released collections containing both mined and synthetic instances. For broader capabilities, a self-evolving task-synthesis system converts community-contributed skills into general-purpose agentic tasks. Tasks from both sources are normalized to a common contract consisting of an initialized execution environment, a natural-language objective, and an automatic verifier. Consequently, heterogeneous tasks can be composed with different harnesses and trained within the same RL framework.

Across sources, we map each instance into the common task contract by materializing its base repository or container and required assets as the initial environment, translating its issue statement or instruction into the task objective, and retaining its tests or reward programs as the verifier. This normalization preserves source-specific execution and reward semantics while exposing a consistent interface to the rollout runtime. In particular, these tasks remain grounded in live environments rather than being reduced to static instruction–response pairs, so their rewards reflect program behavior, repository state, and task-specific execution outcomes.

Individual skills often describe localized workflows, whereas a general agent must coordinate multiple capabilities over longer horizons. We therefore build a skill-state graph whose nodes denote observable environment states and whose edges denote state-transforming capabilities extracted from skills. Skills are composed only when their input and output states are compatible. Sampling paths of different lengths from this graph produces capability sequences with varied horizons and complexity.

Each sampled path is converted into an executable task bundle through a progressive pipeline that constructs the environment, task, and verifier in sequence. Every stage is paired with an executable validator. Rule-based checks verify structural correctness, dependency resolution, and executability, while rubric-based checks evaluate semantic quality and cross-stage consistency. Failures trigger stage-local repair or regeneration, and only validated artifacts are passed downstream. This localizes synthesis errors and reduces inconsistencies among the environment, task specification, and verifier.

Validated tasks are deployed in online RL and are also rolled out to construct reusable offline training data. Candidate trajectories passing outcome-based filtering undergo step-level curation. Each interaction step is annotated by behavior type, covering normal progress as well as tool-use errors, repetitive failed attempts, invalid recovery, premature termination, protocol violations, unsupported assumptions, and hallucinated observations. Erroneous steps remain in the interaction context but can be marked as skip and excluded from the imitation loss, while the remaining responses serve as optimization targets. This selective masking avoids imitating flawed intermediate behavior without destroying the causal context of later actions.

Downstream execution further exposes systematic weaknesses in the synthesized task distribution. We aggregate execution failures and verifier feedback by skill domain and synthesis stage, and use these statistics to update skill sampling weights as well as synthesis skills, environment templates, and stage-specific prompts. The revised system then resamples capability paths and generates the next task distribution, progressively improving executability, coverage, and difficulty from observed agent behavior.

Optimizing on agentic experience introduces challenges beyond single-turn response optimization. Verifiers score complete, tool-interleaved sessions, while only selected policy-generated segments are trainable. Moreover, executable environments expose the reward channel to solution leakage, test manipulation, and other forms of reward hacking. We first align terminal outcomes with the trainable segments of a session, then refine this signal with process-aware advantage control, and finally harden the verifier against reward leakage. We conclude by examining the resulting optimization behavior across task families and agent harnesses.

The PrefixTree maps each annotated message to its exact trainable token span. Let $w_{i,k}\in[-1,1]$ denote the process weight for segment $k$ in session $i$. For a trainable token $t$ in that segment, we use

and set the advantage to zero for non-trainable tokens. Process weights can suppress or reverse positive credit for parse and format errors, invalid tool names or arguments, repeated or failed tool calls, and context-, turn-, or session-limit termination. They are applied only to positive advantages, so the negative learning signal of failed trajectories is preserved. In short, the outcome reward determines whether the task is solved, whereas the process weight determines whether an intermediate behavior should receive positive credit.

During evaluation, canonical test files are restored or overlaid after the agent has stopped, and gold test patches are applied on top of the agent’s source changes. Modifications to agent-visible tests therefore cannot directly alter the scoring procedure. We use conservative all-correct semantics for software-engineering tasks, requiring both target-fix and regression checks to pass; for tasks with a canonical expected test-state map, the observed outcomes must match it exactly. Missing grading artifacts, execution failures, and unparseable verifier outputs are tracked separately from genuine task failure, preventing infrastructure errors from being interpreted as successful policy behavior. Together, these measures keep the reward tied to genuine changes in the executable task state rather than access to hidden solutions or corruption of the verifier.

Biology-Instructions [34] is a multi-omics benchmark that evaluates the sequence understanding capabilities of models across diverse biological scales. It integrates biological sequence-based prediction tasks with advanced reasoning requirements, challenging models to interpret complex genomic, transcriptomic, and proteomic data.

Mol-Instructions [27] is designed to bridge the gap in specialized LLM training through three primary categories: molecule-oriented, protein-oriented, and biomolecular text-oriented tasks. It includes a vast collection of instruction-following pairs that facilitate the model’s proficiency in handling complex biomolecular structures and functional descriptions.

MolecularIQ [9] evaluates the ability of language models to reason faithfully over molecular graphs represented as SMILES. It contains 5,111 symbolically verifiable questions involving 849 structurally held-out molecules and organizes them into counting, indexing, and constrained-generation task families. Its sampling procedure balances reasoning depth, multitask load, molecular complexity, and answer distributions, enabling fine-grained localization of failures in molecular-structure reasoning.

SciReasoner [80] evaluates scientific reasoning across diverse disciplines, including physics, chemistry, and medicine, 9 domains in total and 149 concrete tasks. It consists of a unified suite of ten sub-benchmarks with varying question formats such as multiple-choice, fill-in-the-blank, and protocol-based procedural questions, designed to assess both knowledge retrieval and complex deductive reasoning.

TOMG-Bench [45] evaluates open-domain, natural-language-guided molecule generation. It comprises three major tasks—molecule editing, molecular-property optimization, and customized molecule generation—each divided into three subtasks with 5,000 test samples per subtask. An automated evaluation framework measures whether generated molecules are valid, satisfy the requested structural or property constraints, and retain appropriate similarity or novelty.

MP20 is a conditional crystal structure generation benchmark for materials with at most 20 atoms per unit cell. The task requires predicting precise atomic coordinates and lattice parameters from chemical compositions under physical constraints such as periodicity and symmetry. It contains 27,136 training and 9,046 test samples with ground-truth structural targets provided without chain-of-thought reasoning annotations.

ProteinBinder-9 is a focused benchmark for evaluating de novo protein binder design across nine biologically relevant protein targets. The benchmark is designed in the spirit of the protein–protein interaction design setting introduced by ODesign, an all-atom generative world model for biomolecular interaction design [99]. For each target, the task is to generate protein binders that satisfy a predefined binding interface and pass a multi-stage structural and physicochemical evaluation pipeline. Candidate backbones and sequences are generated and evaluated using complementary computational models, including structure-generation methods such as RFdiffusion [82] and biomolecular complex prediction with AlphaFold 3 [1]. The resulting candidates are further assessed using interface-confidence, binding-energy, and molecular-contact criteria. ProteinBinder-9 covers diverse target proteins and interface geometries, thereby testing whether a design system can generalize beyond a single protein family or structural context. We report both the number and fraction of candidates that pass the complete evaluation procedure for each target, together with aggregate results across all nine targets. ProteinBinder-9 is intended to provide a reproducible testbed for evaluating end-to-end protein binder design systems, rather than isolated sequence generation or structure prediction performance.

XLRS-Bench [77] focuses on extremely large, ultra-high-resolution remote sensing (RS) imagery; this benchmark defines 16 sub-tasks to evaluate 6 types of perceptual and 4 types of reasoning abilities. It challenges MLLMs to process complex semantic relationships and facilitate real-world decision-making in high-resolution geospatial scenarios.

MicroVQA [11] focuses on microscopy-based research and consists of 1,042 expert-curated multiple-choice questions across diverse imaging modalities. The benchmark assesses three critical reasoning capabilities within biological workflows: expert image understanding, hypothesis generation, and experimental proposal.

SFE [110] is an expert-level benchmark comprising 830 verified visual question answering (VQA) pairs across 66 multimodal tasks. Spanning five high-value scientific disciplines, the dataset utilizes authentic raw scientific data formats to probe the cognitive abilities of models in perception, understanding, and advanced reasoning.

ObsCrisis-Bench [57] evaluates multimodal reasoning about extreme-weather and geophysical crises from multispectral satellite observations and optional weather-station measurements. Its official dataset card reports 4,202 visual-question-answering samples covering 127 events, eight disaster categories, and 61 countries. The tasks span early warning, event-type and timing prediction, impact assessment, and post-event recovery analysis across multiple observation timesteps.

SciCode [75] evaluates the ability of language models to write executable code for realistic scientific research problems rather than conventional algorithmic exercises. It contains 80 main problems decomposed into 338 subproblems across 16 scientific subfields, including physics, mathematics, materials science, biology, and chemistry. Each problem combines scientific knowledge, reasoning, and code synthesis and is accompanied by optional background material, scientist-written reference solutions, and executable tests.

SGI-Bench [90] evaluates Scientific General Intelligence across the complete inquiry cycle defined by the Practical Inquiry Model: deliberation, conception, action, and perception. It contains 1,263 expert-curated samples spanning ten scientific domains and 75 research directions, organized into scientific deep research, idea generation, dry and wet experiments, and multimodal experimental reasoning. Task-specific multidimensional metrics assess evidence synthesis, novelty and feasibility, computational correctness, protocol fidelity, and interpretation of experimental results.

ResearchClawBench [91] evaluates whether autonomous agents can conduct end-to-end scientific research from raw data and related literature to a publication-style research report. It contains 40 tasks derived from real papers across ten scientific domains, while withholding the target paper during evaluation. Expert-authored, multimodal rubrics measure whether agents reproduce the original experimental protocols, evidence chains, analyses, and scientific conclusions while leaving room for findings beyond the source paper. We evaluated ResearchClawBench on ResearchHarness v0.0.49.

MMLU-Pro [81] enhances the original MMLU by increasing the number of choices and focusing on more challenging, reasoning-intensive questions. It covers a broad range of subjects, requiring models to demonstrate deeper multi-task language understanding and more robust problem-solving skills.

SimpleQA-Verified [33] evaluates short-form factuality and parametric knowledge without access to retrieval tools. It consists of 1,000 human-verified prompts derived from SimpleQA through deduplication, topic balancing, source reconciliation, ambiguity removal, and adversarial difficulty filtering. Its revised autorater distinguishes correct, incorrect, and not-attempted responses while handling numeric tolerances, hedging, and answer-format variation more reliably.

AdvancedIF [35] evaluates advanced instruction following under complex single-turn instructions, multi-turn carried context, and system-prompt steerability. It contains 1,645 human-written prompts paired with expert-authored rubrics of up to 20 independently verifiable criteria. A response succeeds only when it satisfies all applicable criteria, making the benchmark sensitive to subtle omissions and conflicts across user, conversational, and system-level instructions.

HMMT-2026 [24] evaluates advanced mathematical reasoning using 33 problems from the February 2026 Harvard–MIT Mathematics Tournament released through MathArena. The problems cover algebra, combinatorics, geometry, and number theory and were converted to LaTeX and manually verified together with their reference answers. Because the problems were evaluated soon after the competition, the benchmark also serves as a relatively fresh test of competition-level mathematical problem solving.

MMMU-Pro [97] is a robust extension of the MMMU benchmark. MMMU-Pro introduces more challenging multidisciplinary multimodal tasks. It emphasizes expert-level understanding and complex reasoning across a wide array of professional domains, utilizing high-resolution images and specialized knowledge.

ChartQAPro [53] evaluates visual perception and complex reasoning over realistic charts. The published benchmark contains 1,341 charts collected from 99 diverse sources and 1,948 questions covering mathematical and visual reasoning, conversational queries, fact checking, multiple-choice questions, and hypothetical scenarios. It includes heterogeneous chart types such as dashboards, infographics, maps, bar charts, and line charts, substantially increasing both visual and linguistic diversity over earlier chart-question-answering datasets.

SkillsBench [46] evaluates whether structured packages of procedural knowledge improve the performance of language-model agents on expertise-heavy tasks. Its current inventory contains 87 tasks across eight domains, each paired with curated Skills and deterministic verifiers. The benchmark uses matched evaluations with and without Skills to separate the contribution of procedural guidance from the underlying model and agent harness. We evaluated SkillsBench on OpenClaw 2026.5.7.

Terminal-Bench 2.1 [54, 74] evaluates agents on 89 difficult, realistic tasks executed in isolated command-line environments across software engineering, machine learning, security, data science, and related workflows. Each task provides a dedicated environment, a human-written reference solution, and automated tests for outcome verification. Version 2.1 revises 28 tasks from Terminal-Bench 2.0 and introduces continuous validation to improve task correctness and benchmark reliability. We evaluated Terminal-Bench 2.1 on Terminus 2, and some results were collected from Artificial Analysis.

SWE-Bench Pro [25] evaluates coding agents on long-horizon, enterprise-oriented software-engineering tasks that may require hours or days of professional work. It contains 1,865 human-verified problems from 41 actively maintained repositories, divided into public, held-out, and commercial sets spanning open-source and proprietary codebases. Agents must implement substantial, often multi-file changes that satisfy task-specific tests without regressing existing behavior. We evaluated SWE-Bench Pro on Mini-SWE-Agent, and we modified the official evaluation image to avoid agents getting ground truth from git logs.

SWE-bench Multilingual [41, 93] extends SWE-bench-style issue resolution beyond Python with 300 manually curated tasks from 42 repositories and nine programming languages: C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust. Each task provides a real GitHub issue and a pre-solution repository snapshot, requiring an agent to generate a patch that passes both fail-to-pass tests for the requested fix and pass-to-pass tests for existing functionality. It remains compatible with the standard SWE-bench evaluation infrastructure while exposing language-specific differences in software-engineering performance. We evaluated SWE-Bench Multilingual on Mini-SWE-Agent.

WildClawBench [26] is a native-runtime benchmark designed to evaluate autonomous AI agents on complex, long-horizon tasks within real CLI harness environments. It integrates human-authored bilingual and multimodal workflows with real tools and environments to test agents’ end-to-end tool orchestration and execution capabilities.