\checkdata

[Dataset]https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax \correspondence

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

Abstract

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests—either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements—and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents.

^†^†footnotetext: ^∗Equal contribution. ^†Corresponding author.^†^†footnotetext: Published as a conference paper at COLM 2026.

1 Introduction

AI coding agents have advanced rapidly in recent years, with benchmarks playing a central role in measuring and driving progress. The field has moved from function-level evaluation, such as HumanEval [5] and MBPP [2], to repository-level challenges like SWE-bench [21] and its successors [8, 49, 26, 23, 18]. As models improve, however, a pressing question emerges: do rising scores on these benchmarks genuinely reflect stronger software engineering ability, or are they reaching the limits of what current benchmark designs can measure?

Code refactoring, the disciplined process of restructuring existing code without changing its external behavior, is one of the most frequent activities in professional software development and a key mechanism for managing technical debt in evolving codebases [27]. Industry leaders have identified refactoring as a prototypical long-horizon task: OpenAI highlights “project-scale refactors” as a primary use case for sustained, multi-context-window agents [29, 30], and Cursor reports that real-world developer tasks increasingly span many files and tools [20]. Yet as a benchmark domain, refactoring remains underexplored: existing benchmarks primarily target isolated bug fixes and feature implementations [21, 8], while the few efforts focused on refactoring are limited in scale or restricted to a single language [47, 13]. We argue that real-world refactoring tasks, which require understanding large codebases [32, 36], coordinating changes across many files, and preserving subtle behavioral invariants, offer a substantially more demanding and realistic test of agent capability than the tasks that dominate current evaluations.

Figure 1: Distribution of modified files (left) and lines of code (right) per instance across benchmarks. SWE-Bench ProMax instances are substantially larger: 30% modify more than 10 files and 32% change over 200 lines of code, while 86% of SWE-bench Verified instances modify only a single file.

Instance: nasa/fprime#3422 Language: C++ 244 files +591 / $-$ 514 lines Modified files (excerpt): Autocoders/Python/src/.../component/cpp.tmpl Drv/BlockDriver/BlockDriverImpl.cpp Fw/FPrimeBasicTypes.hpp [NEW] Fw/FPrimeBasicTypes.h [NEW] Fw/Types/BasicTypes.h FppTest/component/active/ActiveTest.cpp Os/Baremetal/TaskRunner/TaskRunner.cpp Svc/ActiveLogger/ActiveLoggerImpl.cpp … and 238 more files

Figure 2: A representative SWE-Bench ProMax instance: refactoring NASA’s F’Prime flight software framework requires coordinated changes across 244 files spanning the entire codebase.

Building a reliable refactoring benchmark, however, exposes two weaknesses that pervade existing benchmarks more broadly. The first is difficulty. On SWE-bench Verified, frontier agents now exceed 75% resolve rate [38], and the gap among top systems continues to narrow—prompting both academic [22] and industry [20] observers to question whether public benchmarks still meaningfully differentiate frontier capabilities. A key reason is scope: most existing benchmark instances involve modifications to a small number of files with limited lines of code. Real-world refactoring operates at a fundamentally different scale, often requiring coordinated changes across dozens of files and hundreds of lines of code, demanding sustained cross-file reasoning that current agents struggle to maintain [10, 9]. The second weakness is evaluation quality. Test suites in existing benchmarks suffer from two complementary defects: overly narrow tests that enforce specific implementation details and reject functionally correct solutions, and overly broad tests that check behavior not specified in the task description [8, 26]. A recent audit of SWE-bench Verified found these defects in nearly 60% of unsolved instances—35.5% had narrow tests and 18.8% had broad tests—leading OpenAI to deprecate the benchmark entirely [31]. Compounding this, problem descriptions are often imprecise or ambiguous, and growing evidence of data contamination in benchmarks sourced from public repositories further erodes the correlation between benchmark scores and genuine capability [31, 37]. Additionally, despite growing multilingual efforts [49, 33], most benchmarks remain Python-centric, leaving open how well agents generalize across language paradigms such as Rust’s ownership model, C’s manual memory management, or Java’s type hierarchies.

We introduce SWE-Bench ProMax to address these gaps. The benchmark comprises 170 refactoring instances drawn from real commits in actively maintained GitHub repositories, spanning seven programming languages: Python, Java, TypeScript, Go, C, C++, and Rust. As Figure 1 illustrates, SWE-Bench ProMax operates at a fundamentally different scale than existing benchmarks: 30% of our instances modify more than 10 files and 32% require over 200 lines of code, compared to SWE-bench Verified where 86% of instances modify only a single file. To make this concrete, Figure 2 shows a representative task from SWE-Bench ProMax: refactoring NASA’s F’Prime flight software framework requires migrating 244 files from a monolithic header to a new unified entry point---touching autocoders, drivers, OS layers, services, and build configurations---while preserving identical runtime behavior¹¹ 1 https://github.com/nasa/fprime/pull/3422. The full problem statement and additional examples across all seven languages are presented in Appendix C.

Our contributions are as follows. First, we ensure benchmark quality through rigorous, multi-stage expert curation. Issue descriptions are rewritten from scratch to provide precise, unambiguous specifications that fully define the expected refactoring. Test suites are manually reviewed to eliminate overly narrow and overly broad tests, and each problem statement is verified to serve as both a necessary and sufficient condition for the gold patch. Second, the benchmark presents a genuine challenge to frontier models. During curation, tasks with insufficient complexity, limited cross-file scope, or too few lines of code are systematically filtered out. The remaining instances require large-scale, coordinated modifications averaging 11.4 files and 261.6 lines of code, and the best model in our evaluation achieves only 41.2% resolve rate. Notably, higher cost does not guarantee proportionally better performance: Claude Sonnet 4.6 averages $4.77 per instance yet resolves 38.8%, while GLM-5 attains a comparable 36.5% at only $0.24—suggesting that open-weight models can approach frontier performance at a fraction of the cost. Agent trajectory analysis further reveals that failed attempts consistently modify fewer files than the gold patch requires while consuming more interaction rounds, pointing to incomplete cross-file coordination as the dominant failure mode. Third, the benchmark spans seven diverse programming languages, enabling analysis of how language-specific features, including type systems, memory models, and build ecosystems, affect agent performance on code transformation tasks.

2 Related work

2.1 Coding benchmarks

As frontier code models [56, 4, 50, 41, 38] and autonomous agents [48, 44, 46, 55] have rapidly advanced, with complementary work on efficient code reasoning, repository-level context construction, collaborative inference, and long-horizon agent memory [52, 45, 17, 34, 53, 24, 12, 6], benchmarks have become the critical bottleneck for measuring real progress. Table 1 compares SWE-Bench ProMax with existing benchmarks across six dimensions.

From function-level to long-horizon evaluation.

Early benchmarks such as HumanEval [5], MBPP [2], and LiveCodeBench [19] evaluate isolated code generation. SWE-bench [21] introduced repository-level evaluation from real GitHub issues, with subsequent work extending along multiple axes: Multi-SWE-bench [49] and SWE-PolyBench [33] broaden language coverage; SWE-bench Pro [8] and SWE-EVO [23] target harder, long-horizon tasks (SWE-EVO averages 21 modified files); Terminal-Bench [26] curates 89 hard CLI challenges with human-written verification; and SWE-bench Live [54], SWE-rebench [3], SWE-Factory [15] and Dockerless [51] address contamination, infrastructure and verification. Current benchmarks are rapidly saturating—frontier agents exceed 75% on SWE-bench Verified [21]—and recent studies show that agent capabilities on short tasks are doubling every few months [22], while all state-of-the-art agents achieve less than 20% on long-horizon tasks [10]. SWE-Bench ProMax targets this difficulty frontier: our instances require coordinated changes across 11.4 files and 261.6 lines of code on average, and the best evaluated model achieves only 41.2% resolve rate.

Refactoring benchmarks.

Despite the prevalence of refactoring in professional development, only two benchmarks specifically target it. RefactorBench [13] provides 100 handcrafted multi-file tasks across 9 Python repositories but is limited to a single language, with instances averaging only 4.3 modified files. SWE-Refactor [47] offers 1,099 instances from 18 Java repositories with automated validation but covers only Java and lacks human verification of test quality. Neither combines multilingual coverage, large-scale complexity, and expert-curated test suites—the combination that SWE-Bench ProMax provides.

Expert curation.

The degree of human curation varies widely across benchmarks (Table 1). At the repository level, curation ranges from fully automated (SWE-bench [21]) to extensive: Multi-SWE-bench [49] employs 68 annotators, Terminal-Bench [26] invests approximately three reviewer-hours per task, and SWE-bench Pro [8] rewrites issue descriptions and reviews test scope. However, even well-curated benchmarks retain quality issues: an audit of SWE-bench Verified found that nearly 60% of unsolved instances had material defects in test design or problem descriptions [31]. SWE-Bench ProMax applies expert curation systematically to every instance: issue descriptions are rewritten from scratch as precise specifications informed by the gold patch and test suite, and inappropriate tests (overly narrow or overly broad) are identified and removed, ensuring mutual alignment between specification and evaluation.

2.2 Code refactoring

Traditional refactoring tools such as RefactoringMiner [43] focus on detecting refactorings from commit histories rather than generating them, and empirical studies reveal that 61 out of 100 identified refactoring types remain unsupported by existing engines [27, 25]. Recent work has begun evaluating LLMs as refactoring agents, revealing consistent limitations: agents perform only low-level edits and fail to address high-level design issues [16], struggle with complex context-dependent refactoring [7], and achieve only 7.7% alignment when autonomously discovering needed changes [42]. Crucially, each study uses different evaluation methodologies—code smell counts [7], compilability [39], alignment scores [42]—making cross-study comparison impossible and motivating a standardized, execution-based benchmark with human-verified tests. SWE-Bench ProMax fills this role.

3 SWE-Bench ProMax

Benchmark	Execution Based	Repo Level	Multi Lingual	Refac- toring	Avg. $>$ 5 Files	Expert Curated
HumanEval [5]	✓					✓
MBPP [2]	✓					✓
LiveCodeBench [19]	✓
SWE-bench [21]	✓	✓
Multi-SWE-bench [49]	✓	✓	✓			✓
SWE-PolyBench [33]	✓	✓	✓
SWE-bench Pro [8]	✓	✓				✓
SWE-EVO [23]	✓	✓			✓
Terminal-Bench [26]	✓	✓				✓
RefactorBench [13]	✓	✓		✓		✓
SWE-Refactor [47]	✓	✓		✓
SWE-Bench ProMax (Ours)	✓	✓	✓	✓	✓	✓

Table 1: Comparison of SWE-Bench ProMax with existing benchmarks. Avg.

>

5 Files: the average gold patch modifies more than 5 files. Expert Curated: problem descriptions and/or test suites undergo manual expert review or authoring beyond automated collection.

This section describes how the benchmark is constructed, verified, and composed.

3.1 Task formulation

Each instance in SWE-Bench ProMax consists of four components: (1) a pre-configured Docker environment containing the target repository at the commit immediately before the refactoring, with all dependencies installed; (2) an issue description specifying the intended refactoring in precise natural language; (3) a test suite that validates whether the refactoring has been correctly applied; and (4) a gold patch recording the original developer’s solution. Given the environment and the issue description, an agent must autonomously modify the repository so that all tests pass. An instance is considered resolved if and only if the agent’s modifications pass every test in the suite. This formulation is outcome-driven: we evaluate the final state of the repository rather than the specific commands or intermediate steps the agent takes.

3.2 Dataset construction and curation

Figure 3 illustrates our three-stage pipeline, which progresses from automated collection through environment validation to expert-driven curation. The quality of a benchmark is determined not by its size but by the reliability of its evaluation [26, 23]; accordingly, every instance in SWE-Bench ProMax undergoes multi-stage review before inclusion, and our final benchmark retains only 170 out of 29,782 initial candidates.

Refer to caption — Figure 3: Data collection and curation pipeline for SWE-Bench ProMax.

Stage 1: Data collection.

We use the GitHub API to identify candidate repositories that meet three criteria: at least 500 stars, an approved open-source license, and a primary language (comprising at least 80% of the codebase) among our seven target languages. From these repositories, we extract commits submitted after January 2025 whose messages contain the keyword “refactor” but not “bug fix,” and that modify both test and non-test files. This yields a large initial pool of refactoring-related commits across all seven languages.

Stage 2: Environment construction.

For each candidate commit, we construct an isolated Docker environment containing the repository at the pre-refactoring state with all build dependencies installed, leveraging automated environment construction tools [15, 11]. The repository is cloned at the pre-refactoring commit, the gold patch (comprising both source and test changes) is applied, and the full test suite is executed. Instances where a working environment cannot be established or where the gold patch fails to pass the test suite are discarded, yielding a set of validated instances.

Stage 3: Filtering and problem rewriting.

Because our instances are mined from commits rather than curated issues, the raw data lacks the precise problem descriptions that a benchmark requires—commit messages are written for fellow developers, not as task specifications for AI agents. The final stage therefore involves human experts working with LLM assistance to transform validated instances into high-quality benchmark tasks through four steps.

(1) Commit analysis. Experts analyze the commit diff, assisted by LLMs that summarize changes and identify affected components, to understand the scope, intent, and structural impact of each refactoring before any filtering or rewriting decisions are made.

(2) Quality filtering. Instances with insufficient complexity are removed: we discard tasks confined to a single file, those with too few lines of code modified, or those involving overly simplistic patterns. We also review each test suite to identify and remove overly narrow tests that enforce specific implementation details rather than behavioral outcomes, and overly broad tests that check behavior beyond the scope of the refactoring—directly addressing the quality defects that have undermined prior benchmarks [31].

(3) Problem statement rewriting. Original commit messages are typically terse (e.g., “refactor auth module”), ambiguous, or reference internal context unavailable to an agent. Experts therefore rewrite the issue description from scratch with LLM assistance, producing a precise, self-contained specification that states which components should change, what the expected transformation is, and what behavioral invariants must be preserved. Each description is verified to serve as both a necessary and sufficient condition for the gold patch: a correct solution should satisfy the description, and the description should not admit unintended solutions.

(4) Human verification. A final round of expert review ensures consistency across the issue description, test suite, and gold patch. The description must fully specify the refactoring, the tests must pass if and only if the refactoring is correctly applied, and no unstated requirements remain.

3.3 Dataset composition

	Mean	Max
Issue Description
Tokens	685.3	2,092
Gold Patch (source)
# Files	11.4	182
Lines of code	261.6	4,503
Tokens	8,179.5	72,623
Test Patch
# Files	4.5	66
Lines of code	185.5	1,959
Tokens	3,980.8	52,031
Total
# Files	15.9	244

Table 2: Dataset statistics of SWE-Bench ProMax.

Figure 4: Language distribution in SWE-Bench ProMax.

Language coverage.

SWE-Bench ProMax spans seven programming languages that represent diverse paradigms: dynamically typed (Python), statically typed with garbage collection (Java, Go), gradually typed (TypeScript), systems languages with manual memory management (C, C++), and a language with an ownership-based memory model (Rust). Instances are drawn from 70 distinct repositories (Appendix A), ensuring broad coverage beyond a small number of projects. This diversity enables us to examine how language-specific features affect agent performance on refactoring tasks.

Scale and complexity.

A defining characteristic of SWE-Bench ProMax is the scale of required code changes. As shown in Table 2, gold patches average 11.4 source files and 261.6 lines of code (8,179.5 tokens), with the most complex instances modifying up to 182 source files. Test patches add a further 4.5 files and 185.5 lines on average, bringing the total to 15.9 files per instance. Issue descriptions average 685.3 tokens, providing detailed specifications of the required refactoring. As shown in Figure 1, our instances require substantially more extensive modifications than those in SWE-bench Verified or SWE-bench Pro, reflecting the true complexity of production-scale refactoring.

4 Experiments

4.1 Experimental setup

Agent scaffolds.

We evaluate all models under two agent scaffolds. The first is mini-swe-agent [48], a minimal reimplementation of the widely adopted SWE-agent scaffold that has been used as the default evaluation framework [8, 49, 26]; it provides file viewing, editing, searching, and bash execution in an iterative observe-think-act loop. The second is OpenHands [44], an open platform for generalist software agents that equips models with a richer runtime, including sandboxed command execution and structured file-editing tools. Given the large scale of refactoring patches in SWE-Bench ProMax, we set a step limit of 300 and a cost limit of $10 per instance under both scaffolds. Applying the same scaffolds and limits to all models ensures fair comparison.

Execution environment.

Each instance ships with a pre-built, isolated Docker container constructed during benchmark curation using SWE-Factory [15] and manually verified to be functional. The container contains the repository at the pre-refactoring commit with all dependencies installed, so evaluation requires no additional environment setup.

Evaluation protocol.

An instance is resolved if the agent’s modifications pass every test in the suite. Our primary metric is the resolve rate (Pass@1): the percentage of resolved instances. We evaluate on all 170 instances and report both overall and per-language resolve rates. We also analyze average cost ($) per instance across models.

4.2 Models

We evaluate six frontier models spanning both proprietary and open-weight families:

Proprietary models. (1) Gemini-3-Pro [14], Google’s flagship model; (2) Claude Sonnet 4.6 [1], Anthropic’s latest Sonnet-class model; (3) GPT-5.2 [28], OpenAI’s most recent reasoning model.

Open-weight models. (4) GLM-5 [50], a mixture-of-experts model from Zhipu AI; (5) Kimi-K2.5 [40], Moonshot AI’s agentic intelligence model; (6) Qwen3.5 [4], Alibaba’s latest code-oriented model.

This selection covers diverse architectures, scales, and training paradigms, enabling analysis of how these factors affect refactoring performance.

5 Results and analysis

5.1 Main results

	Overall	Avg. Steps	Avg. Cost	Per-Language Resolve Rate
				Py	Java	TS	Go	C	C++	Rust
Mini-SWE-Agent
Proprietary
Gemini-3-Pro	26.5	58.0	$0.60	17.2	15.4	14.3	26.1	50.0	45.5	27.3
Claude Sonnet 4.6	30.6	99.5	$2.32	13.8	19.2	32.1	13.0	65.0	40.9	40.9
GPT-5.2	21.8	25.2	$0.19	17.2	15.4	21.4	13.0	45.0	31.8	13.6
Open-weight
GLM-5	22.9	108.9	$0.10	13.8	7.7	25.0	21.7	50.0	22.7	27.3
Kimi-K2.5	26.5	85.3	$0.37	17.2	23.1	21.4	17.4	60.0	31.8	22.7
Qwen3.5	20.6	155.4	$0.93	17.2	7.7	10.7	13.0	45.0	27.3	31.8
OpenHands
Proprietary
Gemini-3-Pro	19.4	51.2	$1.49	13.8	19.2	0.0	8.7	45.0	36.4	22.7
Claude Sonnet 4.6	38.8	117.9	$4.77	17.2	30.8	53.6	26.1	50.0	36.4	63.6
GPT-5.2	41.2	115.1	$3.60	48.3	19.2	35.7	26.1	75.0	36.4	54.5
Open-weight
GLM-5	36.5	114.2	$0.24	20.7	34.6	28.6	34.8	65.0	45.5	36.4
Kimi-K2.5	32.9	99.6	$0.72	24.1	30.8	10.7	43.5	70.0	45.5	18.2
Qwen3.5	36.5	141.2	$0.78	37.9	26.9	17.9	39.1	65.0	54.5	22.7

Table 3: Resolve rates (%) of all evaluated models across scaffolds on SWE-Bench ProMax.

All models find SWE-Bench ProMax challenging. The best-performing model, GPT-5.2, achieves only 41.2% resolve rate—far below the 75%+ that frontier agents achieve on SWE-bench Verified [38]. This confirms that multi-file refactoring, with its requirements for cross-file coordination and behavioral preservation, remains a substantial unsolved challenge.

Open-weight models are competitive with proprietary ones at a fraction of the cost. Under OpenHands, GLM-5 and Qwen3.5 (both 36.5%) and Kimi-K2.5 (32.9%) come within a few points of GPT-5.2 (41.2%) and Claude Sonnet 4.6 (38.8%), while spending only a fraction as much per instance ($0.24, $0.78, and $0.72 versus $3.60 and $4.77). The choice of scaffold also matters substantially: every model except Gemini-3-Pro improves markedly when moving from mini-swe-agent to OpenHands (e.g., GPT-5.2 from 21.8% to 41.2%), suggesting that richer runtime tooling is particularly beneficial for large-scale refactoring tasks.

Performance varies substantially across languages. No single model dominates all languages. Claude Sonnet 4.6 leads on TypeScript (53.6%) and Rust (63.6%), GLM-5 leads on Java (34.6%), while GPT-5.2 performs best on Python (48.3%) and C (75.0%). Kimi-K2.5 achieves its best result on Go (43.5%), while Qwen3.5 performs best on C++ (54.5%). This diversity suggests that different model architectures and training data compositions lead to complementary language-level strengths.

TypeScript and Rust show surprising variance. Despite their reputation as complex languages, TypeScript and Rust yield high resolve rates for some models (Claude Sonnet 4.6: 53.6% on TypeScript and 63.6% on Rust; GPT-5.2: 54.5% on Rust) while remaining difficult for others (Gemini-3-Pro: 0.0% on TypeScript; Kimi-K2.5: 18.2% on Rust). This variance may reflect differences in language-specific training data rather than inherent language difficulty.

5.2 Agent behavior analysis

Figure 5: Left: Cumulative distribution of files modified by agents (Claude Sonnet 4.6, Kimi-K2.5) versus the gold patch. Agents modify fewer files than required overall, with the gap widening for larger patches. Right: Cumulative distribution of interaction rounds on resolved (solid) versus unresolved (dashed) instances. Failed attempts consume substantially more rounds than successful ones.

Figure 5 reveals two complementary failure patterns across two representative models (Claude Sonnet 4.6 and Kimi-K2.5).

The left panel compares the number of files modified by each agent against the gold patch. Both Claude Sonnet 4.6 and Kimi-K2.5 closely track the gold patch distribution for small changes (up to ${\sim}$ 5 files), but diverge sharply for larger patches: whereas the gold patch CDF reaches 90% only around 20 files, both agents reach 90% by approximately 10 files. This indicates that the dominant failure mode is incomplete refactoring—agents identify and modify some affected files but fail to propagate changes to all locations requiring coordinated updates. Critically, this is not a matter of failing to find the right files: agents often correctly locate and edit the core files involved in a refactoring, but stop short of applying the same transformation to peripheral call sites, documentation, configuration files, and test fixtures that also need updating. This partial coverage means that even when the central logic is correctly refactored, the test suite fails because downstream dependencies remain inconsistent.

The right panel separates resolved (solid) from unresolved (dashed) instances. For both models, successful resolutions complete in markedly fewer interaction rounds: the pass curves rise steeply and plateau early, while the fail curves are shifted rightward and more gradual. This gap reveals that agents entering unproductive cycles—repeatedly reading files, attempting edits, encountering test failures, and reverting—consume many rounds without expanding the scope of their modifications. In contrast, successful agents exhibit focused behavior, efficiently identifying and modifying the full set of required files in fewer steps. Together with the file-count analysis, these patterns suggest that the key bottleneck is not reasoning ability per se, but rather the capacity to maintain a coherent plan across many files and persist through the cascading consequences of a large-scale structural change.

5.3 Cost and efficiency analysis

Table 3 reports the average number of agent steps and API cost per instance for each model. A striking pattern emerges: higher cost does not translate to proportionally higher resolve rate. Under OpenHands, Claude Sonnet 4.6 is the most expensive model ($4.77 per instance, 117.9 steps on average) yet trails GPT-5.2 (41.2% at $3.60), and Gemini-3-Pro spends $1.49 per instance for only 19.4%. Open-weight models are dramatically more cost-efficient: GLM-5 resolves 36.5% at just $0.24 per instance—roughly one-twentieth the cost of Claude Sonnet 4.6—and Kimi-K2.5 achieves 32.9% at $0.72.

We also observe that a larger step budget does not guarantee progress. Manual inspection of trajectories shows that models facing large multi-file patches can enter repetitive edit–revert cycles—which we term unproductive exploration—consuming many steps without expanding the scope of their modifications. Qwen3.5 represents an extreme case: it takes the most steps under both scaffolds (155.4 and 141.2 on average) yet never leads overall, and under mini-swe-agent it achieves the lowest resolve rate (20.6%), suggesting that excessive exploration without effective cross-file coordination is counterproductive.

6 Conclusion

We presented SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark designed to address three critical gaps in current AI coding evaluation: the saturation of existing benchmarks, the lack of evaluation quality assurance, and the absence of refactoring as a benchmark domain. SWE-Bench ProMax comprises 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust) and 70 repositories, selected from 29,782 initial candidates through a rigorous three-stage pipeline. Every instance undergoes expert curation: issue descriptions are rewritten from scratch to eliminate ambiguity and answer leakage, and test suites are manually reviewed to remove overly narrow tests that reject valid solutions and overly broad tests that check unstated requirements. Evaluation of six frontier models reveals that SWE-Bench ProMax remains far from saturated: the best model (GPT-5.2) achieves only 41.2% resolve rate, and no model dominates across all languages. Our analysis uncovers two key findings. First, open-weight models are highly competitive with proprietary ones on large-scale refactoring: GLM-5 and Qwen3.5 come within five points of the best proprietary model at a fraction of the cost, and manual trajectory inspection shows that higher spending often reflects unproductive exploration cycles rather than genuine progress. Second, the dominant failure mode is incomplete refactoring: agents consistently modify fewer files than the gold patch requires, indicating that sustained cross-file coordination remains a fundamental bottleneck for current AI coding agents.

Acknowledgments

This research was supported by the National Key Research and Development Program of China (Grant No. 2023YFB4503802), the Natural Science Foundation of Shanghai (Grant No. 25ZR1401175), the Hong Kong Research Grant Council General Research Fund (Grant No. 16206524), and the Hong Kong Research Grant Council Theme-based Research Scheme (Grant No. T41-517_25-N). We thank Kexin Pei for valuable feedback and advice.

Ethics statement

All repositories included in our benchmark are publicly available on GitHub and use approved open-source licenses (e.g., MIT, Apache 2.0, BSD), and we respect each project’s licensing terms in our distribution. No personal or sensitive data is collected; our dataset consists solely of code, commit metadata, and test cases derived from public repositories. The benchmark is intended exclusively for the evaluation and improvement of AI coding agents, and we encourage responsible use that aligns with the goals of advancing software engineering research.

LLM usage disclosure

In accordance with the policy on LLM usage, we disclose the following uses of large language models in this work: (1) Human annotators collaboratively used LLMs as writing assistants to rewrite problem statements and review test suites during benchmark curation (Stage 3 of our pipeline); the LLM served as an interactive tool under human direction, not as an autonomous generator. (2) Claude Sonnet 4.6 was used for multi-label classification of task categories and required reasoning skills (Appendix B); these classifications are used for analysis only and do not affect the benchmark instances or evaluation results. (3) LLMs were used for minor assistance in drafting and editing portions of this paper.

References

[1] Anthropic (2026) Claude sonnet 4.6. Note: https://www.anthropic.com/news/claude-sonnet-4-6 Cited by: §4.2.
[2] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. (2021) Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §1, §2.1, Table 1.
[3] I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel (2025) Swe-rebench: an automated pipeline for task collection and decontaminated evaluation of software engineering agents. arXiv preprint arXiv:2505.20411. Cited by: §2.1.
[4] R. Cao, M. Chen, J. Chen, Z. Cui, Y. Feng, B. Hui, Y. Jing, K. Li, M. Li, J. Lin, et al. (2026) Qwen3-coder-next technical report. arXiv preprint arXiv:2603.00729. Cited by: §2.1, §4.2.
[5] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §1, §2.1, Table 1.
[6] S. Chen, S. Lin, Y. Shi, H. Lian, X. Gu, L. Yun, D. Chen, L. Cao, J. Liu, N. Xia, et al. (2025) SWE-exp: experience-driven software issue resolution. arXiv preprint arXiv:2507.23361. Cited by: §2.1.
[7] J. Cordeiro, S. Noei, and Y. Zou (2024) An empirical study on the code refactoring capability of large language models. ACM Transactions on Software Engineering and Methodology. Cited by: §2.2.
[8] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, et al. (2025) Swe-bench pro: can ai agents solve long-horizon software engineering tasks?. arXiv preprint arXiv:2509.16941. Cited by: §1, §1, §1, §2.1, §2.1, Table 1, §4.1.
[9] R. Desai, J. Hu, J. Cabezas, N. Harsola, P. Shukla, R. B. Chaim, A. E. Assadi, O. M. Kamath, F. Faldu, P. Hebbar, et al. (2026) SWE-marathon: can agents autonomously complete ultra-long-horizon software work?. arXiv preprint arXiv:2606.07682. Cited by: §1.
[10] Y. Feng, J. Sun, Z. Yang, J. Ai, C. Li, Z. Li, F. Zhang, K. He, R. Ma, J. Lin, et al. (2026) Longcli-bench: a preliminary benchmark and study for long-horizon agentic programming in command-line interfaces. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 29952–29963. Cited by: §1, §2.1.
[11] D. Fu, S. Wu, Y. Wu, Z. Peng, Y. Huang, J. Sun, J. Zeng, M. Jiang, L. Zhang, Y. Li, et al. (2026) Davinci-env: open swe environment synthesis at scale. arXiv preprint arXiv:2603.13023. Cited by: §3.2.
[12] S. Gao, W. Zeng, Z. Yu, J. Wangni, C. Wang, K. Cai, S. He, and M. R. Lyu (2026) SWE-mem: learning adaptive memory management for long-horizon coding agents. arXiv preprint arXiv:2606.28434. Cited by: §2.1.
[13] D. Gautam, S. Garg, J. Jang, N. Sundaresan, and R. Z. Moghaddam (2025) Refactorbench: evaluating stateful reasoning in language agents through code. arXiv preprint arXiv:2503.07832. Cited by: §1, §2.1, Table 1.
[14] Google DeepMind (2025) Gemini 3 pro. Note: https://blog.google/products-and-platforms/products/gemini/gemini-3/ Cited by: §4.2.
[15] L. Guo, Y. Wang, C. Li, W. Tao, P. Yang, J. Chen, H. Song, D. Tang, and Z. Zheng (2025) Swe-factory: your automated factory for issue resolution training data and evaluation benchmarks. arXiv preprint arXiv:2506.10954. Cited by: §2.1, §3.2, §4.1.
[16] K. Horikawa, H. Li, Y. Kashiwa, B. Adams, H. Iida, and A. E. Hassan (2025) Agentic refactoring: an empirical study of ai coding agents. arXiv preprint arXiv:2511.04824. Cited by: §2.2.
[17] C. Hu, W. Zeng, Y. Shi, B. Shen, and X. Gu (2026) In line with context: repository-level code generation via context inlining. Proceedings of the ACM on Software Engineering 3 (FSE), pp. 1469–1491. Cited by: §2.1.
[18] W. Huang, C. Lee, L. Tng, and S. Ge (2026) DeepSWE: measuring frontier coding agents on original, long-horizon engineering tasks. arXiv preprint arXiv:2607.07946. Cited by: §1.
[19] N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica (2024) Livecodebench: holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. Cited by: §2.1, Table 1.
[20] N. Jain (2026) How we compare model quality at Cursor. Note: https://cursor.com/blog/cursorbenchAccessed: 2026-03-11 Cited by: §1, §1.
[21] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024) Swe-bench: can language models resolve real-world github issues?. In International Conference on Learning Representations, Vol. 2024, pp. 54107–54157. Cited by: §1, §1, §2.1, §2.1, Table 1.
[22] T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. Von Arx, et al. (2025) Measuring ai ability to complete long tasks. arXiv preprint arXiv:2503.14499 352. Cited by: §1, §2.1.
[23] T. Le, M. V. Thai, D. N. Manh, H. P. Nhat, and N. D. Bui (2025) SWE-evo: benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470. Cited by: §1, §2.1, §3.2, Table 1.
[24] H. Li, Y. Shi, S. Lin, X. Gu, H. Lian, X. Wang, Y. Jia, T. Huang, and Q. Wang (2025) Swe-debate: competitive multi-agent debate for software issue resolution. arXiv preprint arXiv:2507.23348. Cited by: §2.1.
[25] B. Liu, H. Liu, N. Niu, Y. Zhang, G. Li, H. Jiang, and Y. Jiang (2025) An automated approach to discovering software refactorings by comparing successive versions. IEEE Transactions on Software Engineering 51 (5), pp. 1358–1380. Cited by: §2.2.
[26] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. (2026) Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868. Cited by: §1, §1, §2.1, §2.1, §3.2, Table 1, §4.1.
[27] B. Nyirongo, Y. Jiang, N. Niu, and H. Liu (2025) An empirical study of software refactorings in real-world open-source java projects. IEEE Transactions on Software Engineering. Cited by: Appendix B, Appendix B, §1, §2.2.
[28] OpenAI (2025) GPT-5 system card update: GPT-5.2. Note: https://openai.com/index/gpt-5-system-card-update-gpt-5-2/ Cited by: §4.2.
[29] OpenAI (2025) GPT-5.1-Codex-Max. Note: https://openai.com/index/gpt-5-1-codex-max/Accessed: 2025-11-19 Cited by: §1.
[30] OpenAI (2025) Introducing upgrades to Codex. Note: https://openai.com/index/introducing-upgrades-to-codex/Accessed: 2025-09-15 Cited by: §1.
[31] OpenAI (2026) Why SWE-bench Verified no longer measures frontier coding capabilities. Note: https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/Accessed: 2026-02-23 Cited by: §1, §2.1, §3.2.
[32] W. Peng, Y. Shi, Y. Wang, X. Zhang, B. Shen, and X. Gu (2025) SWE-qa: can language models answer repository-level code questions?. arXiv preprint arXiv:2509.14635. Cited by: §1.
[33] M. S. Rashid, C. Bock, Y. Zhuang, A. Buchholz, T. Esler, S. Valentin, L. Franceschi, M. Wistuba, P. T. Sivaprasad, W. J. Kim, et al. (2025) Swe-polybench: a multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703. Cited by: §1, §2.1, Table 1.
[34] Y. Shi, Y. Qian, H. Zhang, B. Shen, and X. Gu (2025) LongCodeZip: compress long context for code language models. In 2025 IEEE/ACM 40th International Conference on Automated Software Engineering (ASE), Cited by: §2.1.
[35] Y. Shi, S. Wang, C. Wan, M. Wang, and X. Gu (2024) From code to correctness: closing the last mile of code generation with hierarchical debugging. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE 2026), Cited by: Appendix B.
[36] Y. Shi, C. Xie, Z. Sun, Y. Chen, C. Zhang, L. Yun, C. Wan, H. Zhang, D. Lo, and X. Gu (2026) CodeOCR: on the effectiveness of vision language models in code understanding. arXiv preprint arXiv:2602.01785. Cited by: §1.
[37] Y. Shi, H. Zhang, C. Wan, and X. Gu (2024) Between lines of code: unraveling the distinct patterns of machine and human programmers. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE 2025), Cited by: §1.
[38] A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. (2025) Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Cited by: §1, §2.1, §5.1.
[39] M. R. Tapader, M. M. Rahman, A. I. Shiplu, M. F. I. Amin, and Y. Watanobe (2025) Code refactoring with llm: a comprehensive evaluation with few-shot settings. arXiv preprint arXiv:2511.21788. Cited by: §2.2.
[40] K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. (2026) Kimi k2. 5: visual agentic intelligence. arXiv preprint arXiv:2602.02276. Cited by: §4.2.
[41] K. Team, Y. Bai, Y. Bao, Y. Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chen, et al. (2025) Kimi k2: open agentic intelligence. arXiv preprint arXiv:2507.20534. Cited by: §2.1.
[42] A. Thillen, N. Mündler, V. Raychev, and M. Vechev (2026) CodeTaste: can llms generate human-level code refactorings?. In Forty-third International Conference on Machine Learning, Cited by: §2.2.
[43] N. Tsantalis, A. Ketkar, and D. Dig (2020) RefactoringMiner 2.0. IEEE Transactions on Software Engineering 48 (3), pp. 930–950. Cited by: §2.2.
[44] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. (2025) Openhands: an open platform for ai software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025, pp. 65882–65919. Cited by: §2.1, §4.1.
[45] Y. Wang, Y. Shi, M. Yang, R. Zhang, S. He, H. Lian, Y. Chen, S. Ye, K. Cai, and X. Gu (2026) Swe-pruner: self-adaptive context pruning for coding agents. arXiv preprint arXiv:2601.16746. Cited by: §2.1.
[46] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang (2024) Agentless: demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489. Cited by: §2.1.
[47] Y. Xu J. Yang et al. (2026) SWE-refactor: a repository-level benchmark for real-world llm-based code refactoring. arXiv preprint arXiv:2602.03712. Cited by: §1, §2.1, Table 1.
[48] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press (2024) Swe-agent: agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §2.1, §4.1.
[49] D. Zan, Z. Huang, W. Liu, H. Chen, S. Xin, L. Zhang, Q. Liu, L. Aoyan, L. Chen, X. Zhong, et al. (2026) Multi-swe-bench: a multilingual benchmark for issue resolving. Advances in Neural Information Processing Systems 38. Cited by: §1, §1, §2.1, §2.1, Table 1, §4.1.
[50] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. (2026) Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Cited by: §2.1, §4.2.
[51] W. Zeng, Y. Shi, X. Gu, C. Hu, C. Wang, Y. Cui, H. Zhou, M. Qi, J. Wangni, Z. Yu, et al. (2026) Dockerless: environment-free program verifier for coding agents. arXiv preprint arXiv:2606.28436. Cited by: §2.1.
[52] W. Zeng, Y. Wang, C. Hu, Y. Shi, C. Wan, H. Zhang, and X. Gu (2025) Pruning the unsurprising: efficient code reasoning via first-token surprisal. arXiv preprint arXiv:2508.05988. Cited by: §2.1.
[53] W. Zeng, X. Zhang, Y. Shi, C. Hu, Y. Chen, B. Shen, and X. Gu (2026) Glimprouter: efficient collaborative inference by glimpsing one token of thoughts. arXiv preprint arXiv:2601.05110. Cited by: §2.1.
[54] L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, et al. (2026) Swe-bench goes live!. Advances in Neural Information Processing Systems 38. Cited by: §2.1.
[55] J. Zhao, G. Chen, F. Meng, M. Li, J. Chen, H. Xu, Y. Sun, W. X. Zhao, R. Song, Y. Zhang, et al. (2026) Immersion in the github universe: scaling coding agents to mastery. arXiv preprint arXiv:2602.09892. Cited by: §2.1.
[56] Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, et al. (2024) Deepseek-coder-v2: breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931. Cited by: §2.1.

Appendix A Additional dataset details

Table 4: Repositories and licenses used in SWE-Bench ProMax, grouped by language. 70 repositories across 7 languages.

Language	Repository	License
C	betaflight/betaflight	GPL-3.0
	aviggiano/redis-roaring	MIT
	davidesantangelo/krep	BSD-2-Clause
	radareorg/radare2	LGPL-3.0
	CESNET/libyang	BSD-3-Clause
	arkq/bluez-alsa	MIT
	aws/s2n-tls	Apache-2.0
	bitcoin-core/secp256k1	MIT
	openssl/openssl	Apache-2.0
C++	deskflow/deskflow	GPL-2.0
	ETLCPP/etl	MIT
	nasa/fprime	Apache-2.0
	Icinga/icinga2	GPL-3.0
	LMMS/lmms	GPL-2.0
	OpenOrienteering/mapper	GPL-3.0
	WasmEdge/WasmEdge	Apache-2.0
	bloomberg/blazingmq	Apache-2.0
	biojppm/rapidyaml	MIT
Go	cli/cli	MIT
	go-gitea/gitea	MIT
	TecharoHQ/anubis	MIT
	restic/restic	BSD-2-Clause
	OpenListTeam/OpenList	AGPL-3.0
	caddyserver/caddy	Apache-2.0
	derailed/k9s	Apache-2.0
	gitleaks/gitleaks	MIT
	gohugoio/hugo	Apache-2.0
	grpc/grpc-go	Apache-2.0
	istio/istio	Apache-2.0
	jesseduffield/lazygit	MIT
	kubernetes/kubernetes	Apache-2.0
	rqlite/rqlite	MIT
	samber/lo	MIT
	trufflesecurity/trufflehog	AGPL-3.0
Java	bazelbuild/bazel	Apache-2.0
	plantuml/plantuml	GPL-3.0
	hibernate/hibernate-orm	Apache-2.0
	apache/hbase	Apache-2.0
	apache/fesod	Apache-2.0
	apache/iceberg	Apache-2.0
	apache/maven	Apache-2.0
	apache/pinot	Apache-2.0
	google/gson	Apache-2.0
	alibaba/nacos	Apache-2.0
	swagger-api/swagger-core	Apache-2.0
Python	confident-ai/deepeval	Apache-2.0
	google/adk-python	Apache-2.0
	optuna/optuna	MIT
	stanfordnlp/dspy	MIT
	vibrantlabsai/ragas	Apache-2.0
	albumentations-team/albumentations	MIT
	huggingface/transformers	Apache-2.0
	langchain-ai/langchain	MIT
	verl-project/verl	Apache-2.0
	django/django	BSD-3-Clause
	google/langextract	Apache-2.0
	huggingface/lerobot	Apache-2.0
	hummingbot/hummingbot	Apache-2.0
	icloud-photos-downloader/icloud_photos_downloader	MIT
	mikf/gallery-dl	GPL-2.0
	pandas-dev/pandas	BSD-3-Clause
	pypa/pipenv	MIT
	roboflow/supervision	MIT
Rust	astral-sh/ruff	MIT
	rust-lang/cargo	MIT/Apache-2.0
	openai/codex	Apache-2.0
	qdrant/qdrant	Apache-2.0
	tracel-ai/burn	MIT/Apache-2.0
TypeScript	angular/angular	MIT
	ant-design/ant-design	MIT

The 70 repositories span a broad range of open-source licenses—predominantly Apache-2.0 and MIT, with representation from GPL, BSD, and AGPL—reflecting the diversity of the open-source ecosystem. Repository concentration varies by language: Go draws from 16 distinct repositories yielding 23 instances, while TypeScript’s 28 instances come from only 2 repositories, with Angular alone contributing 25. This skew is a natural consequence of selecting large, actively maintained projects with substantial refactoring activity.

A.1 Per-language summary

Language	#Repos	#Inst.	Avg. #Files	Avg. LOC	Avg. #Non-test
C	9	20	17.9	424.1	15.2
C++	9	22	21.4	196.3	16.0
Go	16	23	16.0	227.4	9.4
Java	11	26	20.8	309.8	16.8
Python	18	29	10.6	299.8	7.0
Rust	5	22	14.5	284.8	11.0
TypeScript	2	28	11.9	122.6	7.5
Overall	70	170	15.9	261.6	11.4

Table 5: Per-language statistics for SWE-Bench ProMax, including number of repositories, instances, and average patch complexity.

Java and C++ exhibit markedly larger patches, averaging 20.8 and 21.4 modified files per instance respectively, with C having the highest average LOC (424.1)—reflecting the cross-cutting nature of refactoring in codebases with deep type hierarchies and extensive header dependencies. In contrast, higher-level languages such as Python, Rust, and TypeScript average 10.6–14.5 files per instance, yet still require non-trivial multi-file coordination that distinguishes SWE-Bench ProMax from single-file benchmarks.

Appendix B Task category analysis

Real-world refactoring commits rarely involve a single type of change. To characterize the diversity of skills required by SWE-Bench ProMax, we use Claude Sonnet 4.6 to perform multi-label classification of each instance into ten predefined categories: API Interface Change, Refactoring Cleanup, Bug Fix, New Feature, Documentation, Error Handling, Performance Optimization, Dependency Integration, Test Improvement, and Security Patch.

Figure 6: Distribution of task categories across SWE-Bench ProMax instances (multi-label; percentages sum to more than 100%). The dominant categories—Refactoring Cleanup (66.5%) and API Interface Change (65.3%)—confirm the refactoring focus of the benchmark, while the substantial presence of New Feature (43.5%) and Bug Fix (41.2%) reflects the multi-faceted nature of real-world code restructuring.

Figure 7: Number of categories per instance. Every instance involves at least two categories, and nearly half (46.5%) involve three or more simultaneously, indicating that SWE-Bench ProMax tasks require holistic software engineering skills rather than isolated refactoring ability.

Figure 6 shows that while refactoring-related categories dominate—Refactoring Cleanup appears in 66.5% of instances and API Interface Change in 65.3%—a substantial fraction of instances simultaneously involve new features (43.5%), bug fixes (41.2%), or documentation updates (21.2%). As Figure 7 illustrates, 46.5% of instances span three or more categories, with no instance involving fewer than two. The co-occurrence matrix (Figure 8) reveals strong coupling between API changes and refactoring cleanup (79 co-occurrences), as well as between refactoring and bug fixing (48 co-occurrences)—patterns consistent with the empirical finding that restructuring code frequently exposes latent defects [27]. This multi-faceted nature distinguishes SWE-Bench ProMax from benchmarks that test isolated skills and better reflects the compound challenges that developers face in practice.

The high prevalence of Bug Fix (41.2%) is particularly noteworthy. Empirical studies of software maintenance have long observed that refactoring and bug fixing are deeply intertwined: restructuring code frequently exposes latent defects that were masked by the original design, and developers routinely address these defects within the same commit rather than deferring them to a separate change [27]. This coupling means that an agent attempting to resolve a refactoring task in SWE-Bench ProMax must not only apply the intended structural transformation but also recognize and correctly fix any bugs that surface during the process [35]—a compound challenge that synthetic benchmarks, which typically isolate refactoring from bug fixing, cannot reproduce. Similarly, the presence of New Feature (43.5%) reflects cases where refactoring serves as a prerequisite for introducing new capabilities: the structural improvement enables or unblocks a feature addition that is delivered in the same commit. These overlapping concerns make SWE-Bench ProMax tasks substantially harder than they would be if each category were tested in isolation.

Required skills.

Complementing the task-type analysis above, we also classify each instance by the reasoning skills required for successful resolution. Figure 9 shows the distribution.

Figure 9: Distribution of required reasoning skills across SWE-Bench ProMax instances (multi-label). Nearly all instances require cross-file reasoning (99.4%) and API semantics understanding (98.8%), confirming that the benchmark systematically tests sustained multi-file comprehension. Pattern matching (91.8%) and interface contract reasoning (97.1%) are also near-universal, while data flow (88.8%), domain knowledge (79.4%) and type system reasoning (50.6%) appear in the majority of instances.

The near-universal prevalence of cross-file reasoning (99.4%) and API semantics (98.8%) validates the benchmark’s design goal of testing agents’ ability to coordinate changes across file boundaries. The high frequency of pattern matching (91.8%) reflects that refactoring tasks often require identifying and systematically transforming recurring code patterns throughout a codebase. Domain knowledge (79.4%) and type system reasoning (50.6%) are also common, indicating that many instances demand understanding of project-specific conventions or language-specific type constraints beyond generic code manipulation.

Appendix C Representative instances

We present one representative instance per language to illustrate the scale and diversity of SWE-Bench ProMax. For each instance we show the repository, number of modified files and lines of code, the commit URL for full inspection, an abbreviated file tree, and a summary of the rewritten problem statement. Table 6 provides an overview.

Lang.	Repository	Files	LOC	Refactoring summary
C++	nasa/fprime	244	559	Unify header includes across framework
Java	plantuml/plantuml	94	1,629	Add hour-level time resolution to Gantt engine
C	betaflight/betaflight	62	846	Rename motor protocol configuration fields
Rust	tracel-ai/burn	49	1,084	Unify scalar arguments across tensor operations
Go	OpenListTeam/OpenList	47	608	Refactor upload stream buffering across drivers
Python	google/langextract	30	1,960	Centralize provider output-format handling
TS	ant-design/ant-design	27	97	Unify destroyOnHidden across components

Table 6: Summary of representative instances in SWE-Bench ProMax, one per language, sorted by number of modified files.