CUDA-Harness: Harnessing Agentic CUDA Kernel Generation and Optimization from Natural Language

The kernel implementation manifest is a structured per-kernel specification, where each entry stands alone. Each entry records the essential information to implement a kernel, including the kernel name, the input and output descriptions of the kernel, a functional description that reveals algorithmic detail, and a language-agnostic reference code snippet. Additionally, each entry carries detailed test information, encompassing the shapes and data types of the test inputs and outputs.

When the agent produces the manifest from the natural language input, it does not focus on writing code for concrete kernel implementation. Therefore, the agent can concentrate on understanding the high-level semantics and on completing details that the original input leaves underspecified.

The self-contained CUDA scaffold is a code template that tells the agent how to implement the self-contained CUDA program based on the manifest entry. Fig. 4 illustrates a simplified example of the self-contained CUDA scaffold. The scaffold includes placeholders for the components of the self-contained CUDA program illustrated in Sec. 3.1.

Within the scaffold, we aggregate the operations related to kernel launch, including memory management, data transfer, and launch configuration definition, into a dedicated launcher function rather than scattering it across the host-side code. With this arrangement, the agent can concentrate on the kernel and its launcher and ignore the remaining host-side details. However, because the launch configuration determines the memory-access pattern of the kernel, the order in which kernel and launcher are generated matters. During initial kernel generation, the agent assumes a reasonable memory access pattern. However, when generating the launcher, the launch configuration is specified. When the implicitly assumed memory access pattern in the kernel conflicts with the explicitly defined launch configuration in the launcher, the agent will reflect (Shinn et al., 2023) and retrospectively edit the already generated kernel, producing invalid CUDA programs. Therefore,

Moreover, as an initial step towards mitigating reward hacking, the scaffold prohibits the use of dummy inputs and provides helper functions for reading the well-prepared test inputs. The test input paths are passed as command-line arguments, and the calling convention is emitted at compile time. Thus, the agent remains unaware of the details of the test inputs when implementing the kernel and cannot tailor an overfitted implementation to them.

Although the self-contained CUDA scaffold eliminates a pathway for reward hacking by using well-prepared test inputs, there remain risks of reward hacking during test data synthesis. If test data synthesis shares context with kernel generation, the agent may adapt test inputs and reference outputs to the kernel implementation rather than to the intended algorithm. Therefore, we decouple test data synthesis and kernel generation. Specifically, the test inputs and reference outputs are produced through standard numerical libraries (e.g., NumPy and PyTorch) in an isolated environment, where context isolation is enforced. Namely, kernel generation and test data synthesis use separate contexts to avoid cross-contamination.

However, decoupling test data synthesis is insufficient, as reward hacking can still arise through the validation rule due to how the test data is synthesized. The functional validation for a kernel $\mathcal{K}$ under test, with test inputs $x$ and reference outputs $y$, is defined as

where $\mathcal{K}(x)$ denotes the kernel outputs, $\mathbb{I}$ is the indicator function, $\epsilon_{\text{atol}}$ and $\epsilon_{\text{rtol}}$ are the absolute and relative tolerances, respectively. According to Eq. equation 1, when the test inputs concentrate on small-magnitude values, the output magnitude $\lvert y\rvert$ also tends to be small, causing $\epsilon_{\text{rtol}}\lvert y\rvert$ to diminish and $\epsilon_{\text{atol}}$ to dominate the criterion. In such cases, a numerically incorrect kernel may still pass the validation, leading to reward hacking when this feedback is used for optimization.

Hence, we require the agent to draw test inputs from a broad numeric range. For instance, test inputs following a Gaussian distribution with a mean of 5.0 and a standard deviation of 10.0 are required to induce distribution shift and cover broader numeric ranges than dummy or small-magnitude inputs. However, larger-magnitude inputs may amplify floating-point accumulation error, which makes validation stricter and can increase false negatives (i.e., rejecting correct kernels). Nevertheless, we adopt this trade-off deliberately to reduce false positives (i.e., accepting incorrect kernels), since accepting an incorrect kernel would mislead subsequent optimization.

Given synthesized test data, CUDA-Harness employs a progressive multi-stage protocol for kernel validation. As validation stages are naturally ordered, with performance mattering only after a kernel compiles and produces correct outputs, CUDA-Harness rejects invalid kernels early to avoid unnecessary runs on target hardware. Specifically, CUDA-Harness follows a fail-fast order that checks compilation first, then functional correctness, and finally runtime performance.

To support the validation protocol, CUDA-Harness exposes a validation toolkit based on the Model Context Protocol (MCP) (Anthropic, 2024). These tools provide a uniform interface to the agent, while their concrete execution remains bound to the environment where the toolkit is deployed, enabling hardware-scalable validation. The toolkit comprises tools that handle compilation, functional validation, and performance profiling.

The compilation tool checks whether a kernel $\mathcal{K}$ can be built successfully under a given compilation setting, which can be represented as

where $\phi$ is the compilation parameters, such as "-arch=sm_80". Given the test inputs $x$ and reference outputs $y$, the functional validation tool verifies whether the kernel outputs match the reference, producing the result $\mathcal{V}(\mathcal{K},x,y)$. Utilizing profilers such as Nsight Systems and Nsight Compute, the performance profiling tool can measure the runtime performance $\mathcal{P}(\mathcal{K},x)$, where a larger $\mathcal{P}(\cdot)$ indicates a better performance. Taking latency as an example, the performance can be defined as

Integrating these tools, the agent first invokes the compilation tool and is informed of the calling convention emitted by the scaffold via a compile-time #pragma message as described in Sec. 4.1.2. When the kernel passes compilation, the agent calls the functional validation tool based on the calling convention, followed by the performance profiling tool upon successful validation.

As performance only becomes meaningful after correctness is in place, the agent must care about whether the generated kernel compiles and produces the correct outputs. However, the agent tends to introduce advanced and aggressive optimization strategies before the implementation is fully stable and the correctness is guaranteed. When optimization is built on a kernel that looks high-performance but is functionally flawed, subsequent refinement typically remains trapped in the same faulty pattern. Therefore, CUDA-Harness introduces correctness-first refinement into test-time kernel evolution.

The refinement branches based on which validation stage reports the failure. If the generated kernel fails to compile, the agent is steered towards a specific repair based on the error details provided by the compilation tool. When the kernel fails functional validation, the agent is instructed to retreat to a more conservative implementation, prioritizing correctness above all else.

Through correctness-first refinement, once the kernel is reset to a simpler implementation that compiles and passes functional validation, correctness is initially guaranteed. Therefore, subsequent performance optimization can proceed from a trustworthy and well-behaved base.

Building on correctness-first refinement, CUDA-Harness performs reward-driven optimization for the test-time kernel evolution. Benefiting from the progressive multi-stage validation in Sec. 4.2.2, there is a natural and verifiable reward signal characterizing kernel quality. Since the signal comes from compilation, functional validation, and profiling on the target hardware, it is measurable and trustworthy rather than estimated. Therefore, we employ Reinforcement Learning with Verifiable Reward (RLVR) (Lambert et al., 2024) to drive optimization. With compilation parameters $\phi$, test inputs $x$, and reference outputs $y$ fixed, the results of compilation, functional validation, and performance profiling for a generated kernel $\mathcal{K}$ can be simplified as $\mathcal{C}(\mathcal{K})$, $\mathcal{V}(\mathcal{K})$ and $\mathcal{P}(\mathcal{K})$, respectively. Hence, the reward signal can be defined as

Concretely, a penalty is applied when the generated kernel fails to compile. If the kernel compiles and passes functional validation, the performance determines the reward, with higher performance yielding a higher reward.

However, there are no updatable parameters to drive the optimization. Hence, we introduce optimization insight summarization, in which the agent extracts and updates the experience from what changed and how the reward moved to steer the optimization. Given the pre-optimization kernel $\mathcal{K}_{\text{pre}}$ and the optimized kernel $\mathcal{K}_{\text{post}}$, the optimization insight summarization process is defined as

where the agent $\mathcal{M}_{e}$ analyzes the difference in implementation and reward, and distills the optimization experience $e$, such as which optimization strategies are effective or ineffective.

Thus, given $N$ rounds of iterative optimization and $G$ generated kernels per round, and assuming the accumulated optimization insights at the $i$-th round are $E_{1:i}$, reward-driven optimization is written as the following objective:

where $\mathcal{M}$ is the optimization agent, $\mathcal{K}_{i}$ is the best kernel at the $i$-th round, $\mathcal{K}_{i+1,j}$ is the $j$-th generated kernel for round $i+1$, $\Delta r_{i,j}=r(\mathcal{K}_{i+1,j})-r(\mathcal{K}_{i})$ is the reward difference, and $E_{1:i}=\{E_{1},\ldots,E_{i}\}$ accumulates the optimization insights from round 1 to $i$. The set $E_{i}$ collects the $G$ optimization insights from round $i$, namely $E_{i}=\{\mathcal{M}_{e}(\text{Diff}(\mathcal{K}_{i-1,j},\mathcal{K}_{i,j}),r(\mathcal{K}_{i,j})-r(\mathcal{K}_{i-1,j}))\}_{j=1}^{G}$.

The agent operates under the reasoning-action (ReAct) paradigm (Yao et al., 2022). The number of iterative optimization rounds is set to $N=3$, with $G=3$ kernels generated per round. Leveraging Nsight Systems, the performance profiling tool measures the execution time of the kernel over multiple runs and uses the average latency to compute the performance metric $\mathcal{P}(\cdot)=\mathcal{P}_{\text{latency}}(\cdot)$, as defined in Eq. equation 3. Following the evolution process detailed in Sec. 4.3, optimization thus proceeds towards a correct and fast kernel.

Using the compilation parameters and test data provided by the benchmark, we report two correctness metrics, the compilation success rate (Comp.) and functional correctness rate (Func.). Since CUDA-Harness uses execution time to guide test-time evolution, for a fair comparison, we define the runtime performance score (RScore) as $1/\texttt{execution time (ms)}$ computed over multiple measurements, which is not a percentage. The average RScore is reported, with kernels failing the correctness validation assigned a RScore of 0.

We compare CUDA-Harness with the official CUDABench (Zhu et al., 2026a) evaluation procedure, which serves as the baseline harness. Additionally, we further compare the general harnesses, OpenCode (anomalyco, 2026) and Codex (openai, 2026), and agentic Torch2CUDA systems, KernelSkill (Sun et al., 2026) and CudaForge (Zhang et al., 2025). The results are presented in Tab. 1. Across all difficulty levels, CUDA-Harness outperforms the baseline in terms of compilation success rate, functional correctness rate, and the performance score, demonstrating the effectiveness of CUDA-Harness.

As the difficulty level increases, the performance gap between the baseline and CUDA-Harness grows, where higher levels indicate less information in the prompt. Since the baseline generates kernels directly from the raw prompt, it is forced to infer the high-level intents while simultaneously handling low-level implementation within a single generation pass. In contrast, CUDA-Harness introduces Intermediate-Structured Generation to complete underspecified details, allowing it to remain competitive even at Level 3.

To isolate the contribution of each component, we incrementally enable them in Tab. 1.

CUDA-ISG retains only Intermediate-Structured Generation (Sec. 4.1). CUDA-ISG outperforms the baseline, demonstrating the effectiveness and the necessity of the proposed intermediate blueprints.

CUDA-Harness* adds Synthesis-Based Verification (Sec. 4.2) and Reward-Driven Optimization (Sec. 4.3.2) on top of CUDA-ISG while withholding Correctness-First Refinement (Sec. 4.3.1). Compared with CUDA-ISG, its higher compilation success rate and performance score shows that Reward-Driven Optimization is effective at repairing compilation failures and optimizing the generated kernels. However, the remaining gap to the full CUDA-Harness reveals why Correctness-First Refinement matters. Through Correctness-First Refinement, CUDA-Harness achieves a higher functional correctness rate than CUDA-Harness*, further confirming the importance of correctness-first refinement in preserving correctness throughout the optimization process.

We evaluate whether the proposed Synthesis-Based Verification can provide a trustworthy correctness signal for CUDA-Harness. Since the benchmark test data is unavailable during test-time evolution, the synthesized data should lead to validation results that are consistent with the benchmark validation results. To examine the consistency, Tab. 2 compares the validation results obtained using synthesized test data with those obtained using benchmark test data.

As shown in Tab. 2, Synthesis-Based Verification is largely consistent with benchmark validation. Treating benchmark validation as the reference and Synthesis-Based Verification as the prediction, Table 2 shows an accuracy of 84.4%, with precision 89.4% and recall 91.4%. These results indicate that Synthesis-Based Verification provides a strong correctness signal for test-time evolution.

Furthermore, we manually analyze the false positives (8.7% of all cases) where Synthesis-Based Verification passes while benchmark validation fails. Among these false positive cases, 38.1% stem from prompt misinterpretation (e.g., interpreting $A^{3}$ as element-wise cubing), which misleads the subsequent test data synthesis and kernel generation. Another 30.5% arise from omitting steps during benchmark-specific post-processing (e.g., averaging the kernel outputs after execution). The remaining 31.4% are caused by a defect in the CUDABench template that loads test data as float, causing incorrect loading for underlying data types such as uint8 and int32.

As CUDA-Harness is decoupled from the underlying LLM, we further evaluate whether it generalizes across different LLMs. Besides Seed2.0 Lite (ByteDance Seed Team, 2026), we invoke DeepSeek-V3.2 (Liu et al., 2025) and GLM-5.1 (Zeng et al., 2026) as the underlying LLMs, and compare CUDA-Harness with the baseline harness on CUDABench using an NVIDIA A40 GPU. As shown in Tab. 3, CUDA-Harness consistently outperforms the corresponding baseline across all three LLMs in terms of compilation success rate, functional correctness rate, and runtime performance score. These results substantiate the LLM-agnostic design of CUDA-Harness, as its improvements do not rely on a specific underlying LLM.

We evaluate CUDA-Harness using Seed2.0 Lite on three GPUs with distinct architectures and compute capabilities, including the NVIDIA A40, the NVIDIA GeForce GTX 1660 SUPER, and the NVIDIA Jetson AGX Orin. The benchmark and the utilized LLM remain identical to Sec. 5.2. As shown in Tab. 3, the experimental results vary across hardware platforms due to differences in software (e.g., compilers) versions and hardware compute capabilities. However, CUDA-Harness maintains close compilation success rates and functional correctness rates across the three platforms. Although the performance score changes with the hardware compute capability, CUDA-Harness consistently outperforms the baseline on each platform. These results demonstrate that CUDA-Harness is not tailored to a single hardware environment and can generalize across different hardware platforms.

We further evaluate under the C-to-CUDA transpilation scenario, where the input is sequential C code rather than natural language, and we evaluate on BabelTower (Wen et al., 2022; Ke et al., 2025). C code provides comprehensive semantics, including test inputs and outputs descriptions and algorithmic implementation details. For CUDA-Harness, the number of iterative optimization rounds is set to $N=1$, with $G=1$ kernels generated per round.

As shown in Tab. 4, CUDA-Harness outperforms the baseline, even though the evolution budget is relatively small, illustrating that CUDA-Harness can easily generalize to C-to-CUDA transpilation well. Since Intermediate-Structured Generation is not tied to natural language input, decoupling understanding and implementation helps the agent capture the C code and generate the CUDA kernel.