From Search to Signal: Online Post-Training in Automatic Heuristic Design

Yilun Yuan Tianyu Zhou Zhenzhou Tang\corresponding

Abstract

Large language model (LLM)-based automatic heuristic design (AHD) iteratively proposes and refines heuristics, often pairing design rationales with executable code. Task-specific evaluators assess programs; execution outcomes and performance scores guide subsequent search. Many AHD systems keep the generator frozen; EvoTune and Co-Evolution of Algorithms and Language Model (CALM) instead update it from evaluated candidates. When such outcomes drive reinforcement learning with verifiable rewards (RLVR), they create a search-coupled loop: the same evaluated candidate stream supplies both search-state updates and training signals for the model that generates future candidates. Yet validity and performance do not uniquely determine useful model updates; converting them into learning signals must account for the prompt and evolving search state that produced each candidate. We formulate online post-training of small open-weight LLMs in AHD as context-dependent signal construction and develop alternative mappings from program validity, task performance, and generation context to update signals. Using shared evaluated rollouts and matched update budgets, controlled experiments across AHD tasks and model families compare these mappings with online post-training baselines, testing their effects on validity, performance among valid proposals, and the yield of valid proposals that improve under pre-specified contextual comparisons. Complementary checkpoint, frozen-search, and live-system evaluations assess whether proposal-level gains appear in updated checkpoint behavior and subsequent search, rather than arising solely from accumulated search state. A resource-matched comparison under pre-specified cost accounting tests whether online updating adds value beyond additional search with a frozen generator. Together, this design avoids treating end-to-end search gains alone as evidence of stronger heuristic-design capabilities.

Wenzhou University

25451354046@stu.wzu.edu.cn, 25451354054@stu.wzu.edu.cn, tzz@wzu.edu.cn

1 Introduction

Automatic heuristic design (AHD) searches for executable heuristics for a target optimization problem, extending a long line of work on generating and selecting heuristics from performance feedback (Burke et al. 2013). Large language model (LLM)-based systems generate programs, evaluate them on task instances, and use the resulting feedback to form later proposals. Population evolution, program databases, reflection, and tree search organize this process in different ways (Romera-Paredes et al. 2024; Liu et al. 2024a; Ye et al. 2024; Zheng et al. 2025; Novikov et al. 2025). Many such systems keep the generator parameters frozen during search. This choice accommodates black-box models and avoids training overhead, but confines adaptation to external state—such as retained programs, reflections, populations, and the prompts constructed from them. Recurring execution failures, ineffective modifications, or task-specific design patterns cannot accumulate in the generator parameters and must instead be filtered or represented by the search procedure.

Search experience can instead update the model, either after data collection (Liu et al. 2026a; Lee et al. 2026) or during the search itself (Šurina et al. 2025; Huang et al. 2026). Within-search updating allows recurring execution and performance feedback to alter future proposal distributions through the model as well as through external search memory. It also gives each evaluated candidate two consumers. Search may admit it to the population and use it in later prompts; learning assigns credit to the tokens that produced it. These consumers share evidence but need not assign it the same value.

Executable evaluation makes AHD amenable to reinforcement learning with verifiable rewards (RLVR), but verification reports what happened rather than how the policy should be updated. Program validity, failure type, and task score must still be mapped to reward, and the meaning of a score can depend on its parent and search state. Group Relative Policy Optimization (GRPO) then converts rewards into advantages relative to completions from the same prompt (Shao et al. 2024). Numerically different rewards can therefore become learner-equivalent after normalization, while ordering, gating, and non-affine spacing can change the update.

We isolate this search-to-signal transformation in Co-Evolution of Algorithms and Language Model (CALM; Figure 1). Evolutionary operators, context construction, parent selection, population transitions, evaluation, grouping, normalization, and optimization are shared; only the mapping from evaluated records to learner rewards varies. We compare Native CALM, a validity–quality factorization, a pre-generation tail-weighted construction, and a search-exposure residual. These constructions instantiate controlled credit-assignment hypotheses inside one search-and-learning system; they are not separate search algorithms.

Because live search couples a changing checkpoint with an accumulating population, endpoint performance cannot localize an effect. We trace reward differences through shared-record advantages, matched updates, proposal behavior, restarted frozen-checkpoint search, live co-evolution, and a timing-derived frozen-search anchor. Our contributions are:

•

Two-consumer formulation separates search-state updates from learner credit for the same evaluated stream.
•

Normalizer-aware intervention tests which record distinctions survive the actual learner pipeline.
•

Layered attribution protocol separates checkpoint, accumulated search-state, and resource-allocation effects.

The reward constructions instantiate this audit rather than constituting a claim of universal superiority. At a common 500-group horizon, every trained condition improves live search over Frozen, but the mappings trade validity, valid-only performance, contextual improvement yield, and coverage. Base-rate calibration shows that most positive advantage mass follows the dominant class of valid non-improvers, while immediate improvements are strongly enriched relative to their frequency. After population reset, the Tail-weighted checkpoint remains better than the initial Frozen model in all three seeds, but its relation to its own live endpoint is mixed. Under timing-derived budgets for additional Frozen search, Native and Factorized each win only one of three seeds. In this CALM cohort, archive utility, peer-relative learner credit, and strict discovery are therefore not interchangeable.

Figure 1: Search-coupled online post-training and our intervention boundary. Each evaluated group updates the search state and supplies learner rewards. We keep search, group normalization, and optimization fixed and vary only

\phi_{a}

. Live arms generate different future evidence once their models diverge.

2 Related Work

Search Procedures for LLM-Based AHD

LLM-based AHD instantiates a broader generate–evaluate paradigm that also underlies scored-solution prompting and executable reward evolution (Yang et al. 2024; Ma et al. 2024). Search procedures differ primarily in the state retained between generations and the way that state conditions new programs. FunSearch maintains high-scoring programs in a database (Romera-Paredes et al. 2024); EoH evolves natural-language heuristic ideas together with their implementations, and ReEvo augments evolution with reflective feedback (Liu et al. 2024a; Ye et al. 2024). LLaMEA places LLM-generated algorithms in an evolutionary loop (van Stein and Bäck 2025), whereas MCTS-AHD organizes heuristic lineages as a search tree (Zheng et al. 2025). AlphaEvolve scales evaluator-guided program evolution with model ensembles and a large program database (Novikov et al. 2025). LLM4AD provides common interfaces for search methods, model backends, algorithm-design tasks, and evaluation (Liu et al. 2024b); a complementary benchmark studies the contribution of evolutionary search across AHD methods, problems, and model backends (Zhang et al. 2024). Related search spaces include complete agent code and graph-structured multi-agent workflows (Hu et al. 2025; Zhuge et al. 2024). Although their search memories and prompting strategies differ, these systems improve candidates without updating the generator during a run.

Model Adaptation from Search Experience

Search experience can also serve as training data, as in Expert Iteration, AlphaZero, and Reinforced Self-Training (Anthony et al. 2017; Silver et al. 2018; Gulcehre et al. 2023). In AHD, Liu et al. form rank-based preference pairs from collected algorithms and apply direct preference optimization (DPO) before deployment (Rafailov et al. 2023; Liu et al. 2026a), while Evolution Fine-Tuning aggregates trajectories across tasks for mid-training (Lee et al. 2026). The trained component need not be the program generator: HeurAgenix trains a heuristic selector, whereas AHD Agent trains a multi-turn, tool-using policy (Yang et al. 2025; Lv et al. 2026). A distinct setting adapts a model during the search that supplies its training data. EvoTune performs off-policy DPO over an expanding program database, and CALM applies GRPO to groups sampled from a shared evolutionary context (Šurina et al. 2025; Huang et al. 2026). ThetaEvolve likewise trains a mutation generator during program evolution and evaluates the resulting checkpoint under inference-only search (Wang et al. 2026). PACEvolve++ instead trains a strategic advisor, delegates code implementation to a separate model, and changes the source of credit from group-relative feedback to frontier contribution across search phases (Yan et al. 2026). These systems establish online adaptation, checkpoint evaluation, and search-aware credit as close precedents. Our intervention keeps CALM’s search transition, end-to-end generator role, and GRPO normalizer fixed so that record-to-reward mappings and their downstream attribution can be compared directly.

Verifiable Feedback and Reward Construction

Executable evaluation supplies automatically checked outcomes for reinforcement learning with verifiable rewards: parsing and execution expose interface violations and runtime failures, while task evaluators score valid programs. Related systems train code generation from unit-test feedback (Le et al. 2022; Liu et al. 2023) and rank reasoning with learned or rule-based verifiers (Cobbe et al. 2021; Guo et al. 2025). These observations still require a reward function before they can update a policy. GRPO estimates a completion’s advantage relative to other samples from the same prompt (Shao et al. 2024), so reward construction and group normalization jointly determine the credit seen by the learner. GDPO further shows that changing the order of reward aggregation and normalization changes multi-reward optimization (Liu et al. 2026b). For discovery objectives, TTT-Discover combines state reuse with adaptive exponential weighting toward high-reward attempts (Yuksekgonul et al. 2026). Our tail-weighted construction explicitly reuses its concentration rule; our primary question is different: with the search procedure and GRPO estimator fixed, which distinctions in AHD evaluation records remain in the learner’s effective advantage?

3 Search-Coupled Online AHD

At round $t$ , let $S_{t}$ denote CALM’s pre-generation population state and let $\kappa_{t}$ contain the sampled operator and base heuristics. Their prompt $x_{t}=\mathcal{C}(S_{t},\kappa_{t})$ produces a group $y_{t,i}\sim\pi_{\theta_{t}}(\cdot\mid x_{t})$ , $i=1,\ldots,G$ . Execution yields an outcome $o_{t,i}$ (valid or a typed failure) and, for valid programs, a task score $f_{t,i}$ . We collect the pre-generation context and evaluation in $d_{t,i}=(S_{t},\kappa_{t},y_{t,i},o_{t,i},f_{t,i})$ and write $D_{t}=(d_{t,1},\ldots,d_{t,G})$ .

The same record group has two consumers. CALM’s fixed transition $T$ updates the population, whereas a reward construction $\phi_{a}$ maps the records to learner rewards. The first line below describes the search consumer; the remaining lines describe the learner path through reward components $\mathbf{c}_{t}^{a}$ , scalar rewards $\mathbf{r}_{t}^{a}$ , normalized advantages $\mathbf{A}_{t}^{a}$ , and the optimizer $\mathcal{U}$ :

$\displaystyle S_{t+1}$	$\displaystyle\sim T(S_{t},D_{t}),$	(1)
$\displaystyle\mathbf{c}_{t}^{a}$	$\displaystyle=\psi_{a}(D_{t}),\quad\mathbf{r}_{t}^{a}=\sigma_{a}(\mathbf{c}_{t}^{a})=\phi_{a}(D_{t}),$
$\displaystyle\mathbf{A}_{t}^{a}$	$\displaystyle=\mathcal{N}(\mathbf{r}_{t}^{a}),\quad\theta_{t+1}^{a}=\mathcal{U}(\theta_{t}^{a},D_{t},\mathbf{A}_{t}^{a}).$

Thus, evaluation outcomes and search events are evidence; they become learner credit only through $\phi_{a}$ . Across comparisons we hold $T$ , grouping, token masks, $\mathcal{N}$ , and $\mathcal{U}$ fixed and vary only $\phi_{a}$ . In shared-record experiments, $D_{t}$ is also fixed. In live search, trajectories diverge after the first distinct update even though the procedures remain identical.

CALM’s implementation standardizes rewards within each group,

A_{t,i}^{a}=\frac{r_{t,i}^{a}-\bar{r}_{t}^{a}}{s(\mathbf{r}_{t}^{a})+10^{-4}},

(2)

where $s$ is the sample standard deviation. We call two mappings learner-equivalent on $D_{t}$ when $\mathcal{N}(\phi_{a}(D_{t}))=\mathcal{N}(\phi_{b}(D_{t}))$ . A group-shared additive offset is removed exactly; positive rescaling is nearly removed when reward variance dominates the numerical stabilizer, but can leave a small scale-dependent difference in low-variance groups. Beyond this numerical edge case, context changes the effective signal through ordering, gating, ties, or non-affine spacing. We therefore audit both normalized advantages and matched parameter updates rather than treating a different reward formula as sufficient evidence of a different learning signal.

4 Reward Constructions

Table 1 summarizes four constructions embedded in the same CALM loop. Here $b_{t}$ is the best prompt-base score, $F_{t}$ the pre-generation frontier, $\Delta_{t,i}=f_{t,i}-b_{t}$ , and $\operatorname{ctr}(\cdot)$ denotes within-set centering. Frozen CALM is the no-update control, not a fifth reward construction.

Table 1: Reward constructions compared with search, evaluation, grouping, and GRPO held fixed. The tail-weighted construction uses the group-concentration rule of TTT-Discover (Yuksekgonul et al. 2026); Search-Exposure Residual is evaluated only as a mechanism probe.

Construction	Evidence and reference	Reward supplied to GRPO	Role
Native CALM	Typed failures; nonlinear comparison with $b_{t}$	Released piecewise reward $r^{\mathrm{N}}$	Online baseline
Factorized Validity–Quality	Binary validity; signed, RMS-scaled $\Delta_{t,i}$ for valid candidates	$r^{\mathrm{F}}_{t,i}=\tfrac{1}{2}v_{t,i}+\tfrac{1}{2}q_{t,i}$	Separate feasibility from quality
Pre-Generation Tail-Weighted	Typed outcomes; pre-generation score novelty, repetition, and gap above $F_{t}$	$r^{\mathrm{T}}_{t,i}=G\omega_{t,i}$	Emphasize the group upper tail
Search-Exposure Residual	Typed validity; $\Delta_{t,i}$ ; counterfactual one-step parent exposure $\rho_{t,i}$	$r^{\mathrm{S}}_{t,i}=a^{V}_{t,i}+a^{Q}_{t,i}+a^{S}_{t,i}$	Search-derived mechanism probe
Frozen CALM	Same search and evaluator; updates disabled	—	No-update control

Native CALM.

The released mapping assigns missing rationale, missing code, interface failure, runtime failure, and detected randomness rewards of $-1$ , $-0.95$ , $-0.90$ , $-0.85$ , and $-0.75$ . Valid initialization candidates receive zero. Otherwise, with $\delta=\operatorname{clip}(|f-b|/\min\{|f|,|b|\},10^{-10},1)$ , an improvement receives $1+\delta$ , equality receives zero, and a degradation receives $-3\delta/8$ ; a candidate identified as one of the prompt bases receives $-3/5$ . Native therefore combines failure severity, validity, and context-relative quality in one piecewise scale.

Factorized Validity–Quality.

Let $v_{t,i}\in\{-1,1\}$ encode invalid versus valid output. Among valid non-initialization candidates, define $q_{t,i}=\Delta_{t,i}/\operatorname{RMS}(\boldsymbol{\Delta}_{t})$ , and set $q_{t,i}=0$ otherwise or when the eligible-set RMS is numerically zero. The reward

r^{\mathrm{F}}_{t,i}=\tfrac{1}{2}v_{t,i}+\tfrac{1}{2}q_{t,i}

(3)

is followed by one joint GRPO normalization. It is a scalar reward construction, not a two-loss or separately normalized multi-reward objective.

Pre-Generation Tail Weighting.

For each group, a tie-aware midrank $u_{t,i}\in[0,1]$ places typed failures below valid candidates. Among valid candidates, it favors scores absent from the pre-generation archive and responses not repeated within the group. We retain magnitude only for positive frontier gaps, $g_{t,i}=[f_{t,i}-F_{t}]_{+}/\operatorname{RMS}([\mathbf{f}_{t}-F_{t}]_{+})$ , and set $w_{t,i}=u_{t,i}+\tfrac{1}{2}g_{t,i}$ . Following TTT-Discover’s concentration rule, but not its PUCT or leave-one-out objective, we choose $\beta_{t}$ so that

	$\displaystyle\omega_{t,i}$	$\displaystyle=e^{\beta_{t}w_{t,i}}\Big/\sum_{j}e^{\beta_{t}w_{t,j}},$		(4)
	$\displaystyle D_{\mathrm{KL}}(\boldsymbol{\omega}_{t}\\|U_{G})$	$\displaystyle=\gamma_{t},\qquad\gamma_{t}=\min\{\log 2,\log(G/k_{t})\},$		(4)

where $k_{t}$ counts tied maxima, and return $r^{\mathrm{T}}_{t,i}=G\omega_{t,i}$ . If the target equals the largest KL attainable under tied maxima, the implementation returns the limiting distribution that is uniform over those maxima; a constant group returns the uniform distribution. This KL controls concentration over the completion group; it is not policy–reference KL.

Search-Exposure Residual.

This mechanism probe asks whether CALM’s fixed parent-selection rule supplies information beyond immediate quality. After counterfactually inserting candidate $i$ into the pre-generation archive, $\rho_{t,i}$ approximates its one-step exposure through CALM’s primary and secondary parent slots. On eligible valid candidates, we residualize $\operatorname{ctr}(\boldsymbol{\rho}_{t}\odot\boldsymbol{\Delta}_{t})$ against the centered quality direction:

\widetilde{\mathbf{s}}_{t}=\operatorname{ctr}(\boldsymbol{\rho}_{t}\odot\boldsymbol{\Delta}_{t})-\operatorname{proj}_{\operatorname{ctr}(\boldsymbol{\Delta}_{t})}\operatorname{ctr}(\boldsymbol{\rho}_{t}\odot\boldsymbol{\Delta}_{t}),

(5)

then combine centered, RMS-scaled validity $\mathbf{a}^{V}_{t}$ , quality $\mathbf{a}^{Q}_{t}$ , and a bounded rescaling $\mathbf{a}^{S}_{t}$ of $\widetilde{\mathbf{s}}_{t}$ . A channel is set to zero when its centered scale is numerically zero. This changes learner credit only; parent selection and population transitions remain unchanged. Edge-case handling and scaling are fixed before the matched-update probe.

5 Controlled Evaluation

Scope.

We separate mechanism, live-system, and attribution evidence. Shared-record audits span four tasks, and matched updates span two tasks and two 7B model families with one seed per cell. The complete live cohort uses TSP and Qwen2.5-7B-Instruct with three seeds (42, 3407, 1926000), $G=4$ , a population of ten, and 500 groups (2,000 completions) per run. Native, Factorized, Tail-weighted, and Frozen enter this cohort; Search-Exposure Residual remains a mechanism-only comparison. Trained conditions use rank-32 LoRA (Hu et al. 2022) and equal update opportunities. The run seed is the statistical unit, and we report seed-level values without asymptotic significance tests.

Implementation.

We pin the released CALM implementation and preserve its TSP operators, parent sampling, population transition, collapse behavior, and evaluator across conditions. Prompts and completions are capped at 2,048 and 1,024 tokens, respectively, with a 4,096-token model context. The live horizon is 500 evaluated groups and the stagnation threshold is 25. Each online condition receives one optimizer opportunity per group; generated-token counts can still differ because completion lengths differ, so we do not claim equal training-token budgets for live runs. Resolved configurations, source bundles, checkpoint hashes, and completion-level records are retained for every reported run; software and hardware details are provided in the supplementary material.

Figure 2: Learner-path audit. (a) Group standardization removes additive shifts and nearly removes positive rescaling, while gating and nonlinear spacing can survive. (b) Shared-record advantage separation, matched-update cosine, and fixed-context log-probability changes trace signal propagation; open markers denote a degenerate cell and green denotes Search-Exposure Residual. (c) Candidate prevalence calibrates positive-advantage mass by class base rate. Panel (b) establishes mechanism separation, not efficacy.

Attribution protocol.

We first replay mappings on shared evaluated records and compare normalized advantages, including learner-equivalent and zero-advantage groups. Matched updates then hold the initial adapter, response tokens, masks, reference log probabilities, optimizer steps, and training tokens fixed; adapter deltas and log probabilities on withheld contexts test whether a signal difference reaches the checkpoint. Live runs match initial conditions, seeds, and generation budgets but necessarily diverge after updating. Restarted runs load final adapters, reset the population, and disable further training. Frozen runs retain the same search with the initial model. For the resource comparison, setup-excluded loop times determine preregistered Frozen-search group budgets without using efficacy outcomes; endpoint metrics are recomputed from exact stored prefixes. Because prefix records lack timestamps, these are timing-derived budget anchors rather than exact realized-time matches. The contract was frozen for Native and Factorized before efficacy results, so Tail-weighted was not added post hoc. Native and Frozen are the principal baselines because the causal question requires the surrounding CALM loop to remain identical; comparisons with other end-to-end AHD systems would change search and learning simultaneously.

Metrics.

For $N$ completions, valid set $\mathcal{V}$ , and comparison-eligible set $\mathcal{I}$ , we report

$\displaystyle\mathrm{ValidRate}$	$\displaystyle=\|\mathcal{V}\|/N,$	(6)
$\displaystyle\mathrm{ValidPerf}$	$\displaystyle=\frac{\sum_{i\in\mathcal{V}}f_{i}}{\|\mathcal{V}\|},$
$\displaystyle\mathrm{ImproveYield}$	$\displaystyle=\frac{\sum_{i\in\mathcal{I}}\mathbb{I}[f_{i}>b_{i}]}{N}.$

Validity requires a finite task score and is distinct from evaluator dispatch. In live runs, $b_{i}$ is induced by each condition’s evolving search state, so ImproveYield describes the contextual proposal stream rather than context-free checkpoint capability. To compare credit with immediate search utility, we classify candidates as invalid, valid non-improving, parent-improving, or strict-frontier-improving and report class prevalence, positive advantage mass, its enrichment over prevalence, and the within-class positive-credit rate:

$\displaystyle P_{D}(c)$	$\displaystyle=\frac{\|\{i:z_{i}=c\}\|}{N},$	(7)
$\displaystyle P_{A}(c)$	$\displaystyle=\frac{\sum_{i}[A_{i}]_{+}\mathbb{I}[z_{i}=c]}{\sum_{i}[A_{i}]_{+}},$
$\displaystyle E_{A}(c)$	$\displaystyle=P_{A}(c)/P_{D}(c).$

Here $z_{i}$ is the candidate class; ratios are reported only when their denominators are positive. We also report the within-class fraction receiving positive advantage. Archive admissions that beat the pre-generation frontier and exact-response, prompt, and parent-context uniqueness provide further diagnostics; the latter are coverage proxies, not semantic diversity. CALM represents TSP fitness as negative tour length, so higher is better. Search quality is final best $B_{T}$ and trajectory $\mathrm{AUC}=\frac{\sum_{t}B_{t}}{T}$ for equal-horizon runs. For unequal resource-anchor horizons, final best at the preregistered group budget is primary; AUC is descriptive only.

6 Results

Figure 3: Proposal and search outcomes. (a) Execution outcomes across three TSP–Qwen seeds. (b) Mean best-so-far gain; bands show the observed seed range. (c) Same-seed gains over Frozen under live updating (filled) and restarted, update-disabled search (open); lines connect seeds. Native/Factorized and Tail-weighted use independently matched restart cohorts.

Table 2: Core TSP–Qwen2.5-7B live-search results over three matched seeds. Entries are mean

\pm

sample standard deviation, and higher is better throughout. Bold marks the highest descriptive mean in each column, not statistical significance. Validity requires an executable completion with a finite task score and is distinct from evaluator dispatch.

	Search outcomes		Proposal stream
Condition	Final best	Trajectory AUC	Valid (%)	Valid-only perf.	Improve yield (%)
Frozen	-6.2329 $\pm$ 0.0105	-6.2417 $\pm$ 0.0046	60.85 $\pm$ 0.55	-9.8800 $\pm$ 0.8639	2.83 $\pm$ 0.50
Native	-6.2090 $\pm$ 0.0289	-6.2203 $\pm$ 0.0196	93.92 $\pm$ 1.68	-6.4966 $\pm$ 0.0986	2.98 $\pm$ 1.16
Factorized	-6.2041 $\pm$ 0.0116	-6.2179 $\pm$ 0.0087	95.60 $\pm$ 0.64	-6.7488 $\pm$ 0.1168	3.60 $\pm$ 1.25
Tail-weighted	-6.1963 $\pm$ 0.0248	-6.2123 $\pm$ 0.0148	85.12 $\pm$ 2.50	-7.0468 $\pm$ 0.2214	4.63 $\pm$ 0.33

Signal-Propagation Analysis

The normalizer audit identifies apparent context dependence that cannot affect training. In 387 groups, both prompt comparator and frontier were constant within the group, so adding either as a reward offset is eliminated by Equation 2. Among 422 midrank groups, comparator and frontier indicators changed no ordering when score remained the secondary key; pre-generation score novelty and response repetition reordered 159 ( $37.7\%$ ) and 20 ( $4.7\%$ ), respectively.

Native and Factorized produce distinct shared-record advantages on TSP and CVRP-ACO but are nearly equivalent on OBP and OP (Figure 2(b)). Of the 12 audited groups, eight have maximum advantage difference at most $0.001$ and three yield zero advantage under both mappings; the median group maximum is $0.0000672$ , despite a mean of $0.1216$ driven by the separated TSP and CVRP groups. On TSP–Qwen, five matched updates use 20 common completions and 5,957 tokens per condition. Their adapter deltas have cosine $0.9769$ , and the maximum withheld-context log-probability difference is $0.00526$ . Three of four task–model cells yield genuine update differences; CVRP–Qwen is degenerate. Search-Exposure Residual changes advantages in 52/100 shared groups, has adapter-delta cosine $0.8851$ with Native, and reaches $0.0118$ maximum fixed-context difference. These results establish mechanism separation, not search efficacy.

Proposal-Stream Analysis

Frozen yields $60.85\%$ valid completions; Native, Factorized, and Tail-weighted reach $93.92\%$ , $95.60\%$ , and $85.12\%$ (Table 2). Every trained condition improves valid-only performance over Frozen in all seeds, but Native has the best trained-condition mean. Relative to Native, Factorized and Tail-weighted increase contextual improvement yield but reduce valid-only performance; Tail-weighted also reduces validity. Feasibility, conditional quality, and contextual improvement are therefore distinct outcomes.

The feasibility gain is operator dependent. Injection shows the largest paired changes: Native, Factorized, and Tail-weighted raise validity over Frozen by $86.73$ , $91.75$ , and $58.13$ points, respectively. This shows adaptation to a difficult output contract, but does not alone establish stronger algorithmic reasoning.

Credit–Utility Alignment Analysis

The valid non-improver class accounts for $90.93\%$ of Native candidates, $92.00\%$ of Factorized candidates, and $80.48\%$ of Tail-weighted candidates. Its corresponding shares of positive advantage mass are $83.30\%$ , $83.50\%$ , and $85.51\%$ (Figure 2(c)). The resulting mass-to-prevalence ratios are $0.916$ , $0.908$ , and $1.062$ , while parent- and frontier-improving candidates are enriched by $2.83$ – $6.72\times$ . Within the non-improver class, $29.34\%$ , $22.59\%$ , and $26.32\%$ receive positive advantage. A non-improver also receives positive credit when its entire group fails to improve in $42.53\%$ , $31.33\%$ , and $78.07\%$ of rounds. Thus, most absolute credit follows the dominant candidate class, but mappings redistribute credit relative to that base rate. Group centering necessarily assigns positive advantage within any nonconstant group, so these values do not show that such credit is erroneous or harmful; they show that peer-relative credit is not immediate comparator improvement. Signal density also differs: Factorized yields zero advantage in $60.73\%$ of groups, versus $7.00\%$ for Tail-weighted.

Archive admission is different again. Only 32 of 1,282 Native admissions, 29 of 918 Factorized admissions, 54 of 3,819 Tail-weighted admissions, and 17 of 2,540 Frozen admissions beat the pre-generation frontier. Admissions may preserve future stepping stones; the point is that archive utility, learner credit, and strict discovery are not interchangeable labels.

Live-Search Results

At the common 500-group horizon, every trained condition improves final best and AUC over Frozen in every matched seed (Figure 3(b)). Mean final/AUC gains are $+0.0240/+0.0214$ for Native, $+0.0289/+0.0238$ for Factorized, and $+0.0366/+0.0294$ for Tail-weighted. When each online trajectory is truncated at the same-seed number of evaluator calls made by Frozen, Native and Factorized remain ahead in every seed. Additional evaluator calls are therefore not the sole explanation, although earlier validity and archive changes remain possible mediators.

No trained mapping uniformly dominates Native: both Factorized and Tail-weighted have mixed paired directions across seeds. Relative to Frozen, exact-response uniqueness changes by $-10.98$ , $-7.32$ , and $+0.57$ points for Native, Factorized, and Tail-weighted; prompt uniqueness changes by $-9.20$ , $-15.40$ , and $+11.00$ . These reproducible coverage proxies are not semantic diversity measures.

Single-seed live comparisons on the remaining task–model cells are directionally mixed, so we do not pool them with the three-seed cohort. In an independent Native/Factorized cohort, final heuristics were also evaluated on held-out TSP instances at three problem sizes; Factorized has lower mean optimality gap at every size, but the observed seed ranges overlap. These checks support scope and failure-mode analysis rather than a general superiority claim.

Checkpoint and Search-State Attribution Analysis

After resetting the population and disabling further updates, Factorized changes final score/AUC relative to Native by $-0.0098/-0.0139$ on average, with mixed seed directions (Figure 3(c)); its valid-only performance is lower in all three seeds. We therefore find no consistent evidence that the Factorized–Native live ordering persists after reset. In the independently matched Tail-weighted cohort, restarted search improves final score and AUC over the initial Frozen model in all three seeds, by $+0.0191$ and $+0.0205$ on average; validity rises by $18.60$ points and valid-only performance by $3.009$ . Relative to its own live parent, however, the restart has mixed final-score directions and mean final/AUC changes of $-0.0175/-0.0089$ . The updated checkpoint thus retains measurable value over the initial generator, while the live endpoint still reflects accumulated search state and its interaction with online updates.

Under preregistered Frozen-search budgets derived from measured loop times, endpoint differences are mixed. Native and Factorized each outperform the corresponding Frozen prefix in one of three seeds; their mean online-minus-Frozen final-best differences are $-0.0113$ and $+0.0035$ , respectively. Both online conditions nevertheless yield higher validity and valid-only proposal performance in every paired seed. Mean validity gains are $33.78$ and $33.45$ percentage points, and mean valid-only gains are $1.382$ and $1.388$ , for Native and Factorized, respectively. These are budget anchors, not exact time matches: full-trajectory Frozen time differs from the Native online target by $+1.61\%$ , $-13.56\%$ , and $-12.48\%$ across seeds, while timestamps are unavailable for the shorter Factorized prefixes. Online updating therefore changes the observed proposal stream consistently in this cohort, but does not reliably dominate additional Frozen search at the endpoint.

7 Discussion, Limitations, and Conclusion

At the common horizon, all trained conditions improve live search over Frozen, but reward mappings trade validity, valid-only quality, signal density, and coverage. Tail-weighted retains an advantage over the initial model after reset without consistently reproducing its live endpoint. No mapping uniformly dominates, and timing-derived anchors show no consistent endpoint advantage over additional Frozen search.

The two-consumer view explains these differences. Search may retain a stepping stone without immediate frontier gain, while GRPO may reinforce the best member of an unproductive group. Archive admission, positive advantage, and strict discovery therefore answer different questions. Our residual construction shows that search-derived information can survive the normalizer, but establishes mechanism rather than efficacy.

Our conclusions are limited to one co-evolutionary host. Multi-seed efficacy covers TSP and one 7B model; other task–model cells provide mechanism breadth only. Four-sample groups amplify ties, completion rewards cannot separate rationale from code, and coverage hashes are not semantic diversity. Matched updates test token probabilities on common contexts, while a supplementary two-seed CVRP probe evaluates final checkpoints with executable completions on a shared prompt bank but yields no stable ordering. Executable restarts subsequently diverge in context and therefore remain attribution tests rather than fixed-context capability estimates. Realized timing drift and missing prefix timestamps also preclude exact multiseed time-matched inference.

Online AHD should therefore be evaluated as a search-to-signal system: evaluated records separately reshape search and become learner credit through reward construction and normalization. Tracing both paths distinguishes parameter changes, reset-search behavior, and live gains. Reward construction is thus a testable component of search-coupled AHD rather than an implementation detail.

References

Anthony et al. (2017) T. Anthony, Z. Tian, and D. Barber Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §2.
Burke et al. (2013) E. K. Burke, M. Gendreau, M. Hyde, G. Kendall, G. Ochoa, E. Özcan, and R. Qu Hyper-heuristics: a survey of the state of the art. Journal of the Operational Research Society 64 (12), pp. 1695–1724. External Links: Document Cited by: §1.
Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. External Links: 2110.14168 Cited by: §2.
Gulcehre et al. (2023) C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, W. Macherey, A. Doucet, O. Firat, and N. de Freitas Reinforced self-training (ReST) for language modeling. External Links: 2308.08998 Cited by: §2.
Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, pp. 633–638. External Links: Document Cited by: §2.
Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §5.
Hu et al. (2025) S. Hu, C. Lu, and J. Clune Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2.
Huang et al. (2026) Z. Huang, W. Wu, K. Wu, J. Wang, and W. Lee CALM: co-evolution of algorithms and language model for automatic heuristic design. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.
Le et al. (2022) H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi CodeRL: mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 35. External Links: Document Cited by: §2.
Lee et al. (2026) Y. Lee, S. Kim, M. Kang, A. C. L. Chuen, Z. Chen, S. Han, T. Jung, and D. Kang Evolution fine-tuning: learning to discover across 371 optimization tasks. External Links: 2606.29082 Cited by: §1, §2.
Liu et al. (2024a) F. Liu, X. Tong, M. Yuan, X. Lin, F. Luo, Z. Wang, Z. Lu, and Q. Zhang Evolution of heuristics: towards efficient automatic algorithm design using large language model. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, pp. 32201–32223. External Links: Link Cited by: §1, §2.
Liu et al. (2026a) F. Liu, R. Zhang, X. Lin, Z. Lu, and Q. Zhang Fine-tuning large language model for automated algorithm design. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.
Liu et al. (2024b) F. Liu, R. Zhang, Z. Xie, R. Sun, K. Li, X. Lin, Z. Wang, Z. Lu, and Q. Zhang LLM4AD: a platform for large language model-based automatic algorithm design. External Links: 2412.17287 Cited by: §2.
Liu et al. (2023) J. Liu, Y. Zhu, K. Xiao, Q. Fu, X. Han, W. Yang, and D. Ye RLTF: reinforcement learning from unit test feedback. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §2.
Liu et al. (2026b) S. Liu, X. Dong, X. Lu, S. Diao, P. Belcak, M. Liu, M. Chen, H. Yin, Y. F. Wang, K. Cheng, Y. Choi, J. Kautz, and P. Molchanov GDPO: group reward-decoupled normalization policy optimization for multi-reward RL optimization. In Proceedings of the 43rd International Conference on Machine Learning, External Links: Link Cited by: §2.
Lv et al. (2026) H. Lv, N. Lu, Z. Zhou, and S. Liu AHD Agent: agentic reinforcement learning for automatic heuristic design. External Links: 2605.08756 Cited by: §2.
Ma et al. (2024) Y. J. Ma, W. Liang, G. Wang, D. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar Eureka: human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.
Novikov et al. (2025) A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog AlphaEvolve: a coding agent for scientific and algorithmic discovery. External Links: 2506.13131 Cited by: §1, §2.
Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36, pp. 53728–53741. Cited by: §2.
Romera-Paredes et al. (2024) B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi Mathematical discoveries from program search with large language models. Nature 625 (7995), pp. 468–475. External Links: Document Cited by: §1, §2.
Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300 Cited by: §1, §2.
Silver et al. (2018) D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362 (6419), pp. 1140–1144. External Links: Document Cited by: §2.
Šurina et al. (2025) A. Šurina, A. Mansouri, L. C. P. M. Quaedvlieg, A. Seddas, M. Viazovska, E. Abbe, and C. Gulcehre Algorithm discovery with LLMs: evolutionary search meets reinforcement learning. In The 2nd Conference on Language Modeling, External Links: Link Cited by: §1, §2.
van Stein and Bäck (2025) N. van Stein and T. Bäck LLaMEA: a large language model evolutionary algorithm for automatically generating metaheuristics. IEEE Transactions on Evolutionary Computation 29 (2), pp. 331–345. External Links: Document Cited by: §2.
Wang et al. (2026) Y. Wang, S. Su, Z. Zeng, E. Xu, L. Ren, X. Yang, Z. Huang, X. He, L. Ma, B. Peng, H. Cheng, P. He, W. Chen, S. Wang, S. S. Du, and Y. Shen ThetaEvolve: test-time learning on open problems. In Proceedings of the 43rd International Conference on Machine Learning, External Links: Link Cited by: §2.
Yan et al. (2026) M. Yan, B. Peng, B. Coleman, Z. Chen, Z. Xie, S. Chen, Z. He, N. Sachdeva, W. Wang, E. H. Chi, S. Venkataraman, W. Kang, D. Z. Cheng, and B. Wang PACEvolve++: improving test-time learning for evolutionary search agents. External Links: 2605.07039 Cited by: §2.
Yang et al. (2024) C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen Large language models as optimizers. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.
Yang et al. (2025) X. Yang, L. Zhang, H. Qian, L. Song, and J. Bian HeurAgenix: leveraging LLMs for solving complex combinatorial optimization challenges. External Links: 2506.15196 Cited by: §2.
Ye et al. (2024) H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song ReEvo: large language models as hyper-heuristics with reflective evolution. In Advances in Neural Information Processing Systems, Vol. 37, pp. 43571–43608. External Links: Document Cited by: §1, §2.
Yuksekgonul et al. (2026) M. Yuksekgonul, D. Koceja, X. Li, F. Bianchi, J. McCaleb, X. Wang, J. Kautz, Y. Choi, J. Zou, C. Guestrin, and Y. Sun Learning to discover at test time. In Proceedings of the 43rd International Conference on Machine Learning, External Links: Link Cited by: §2, Table 1.
Zhang et al. (2024) R. Zhang, F. Liu, X. Lin, Z. Wang, Z. Lu, and Q. Zhang Understanding the importance of evolutionary search in automated heuristic design with large language models. In Parallel Problem Solving from Nature – PPSN XVIII, Lecture Notes in Computer Science, Vol. 15149, pp. 185–202. External Links: Document Cited by: §2.
Zheng et al. (2025) Z. Zheng, Z. Xie, Z. Wang, and B. Hooi Monte carlo tree search for comprehensive exploration in LLM-based automatic heuristic design. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, pp. 78338–78373. External Links: Link Cited by: §1, §2.
Zhuge et al. (2024) M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber GPTSwarm: language agents as optimizable graphs. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, pp. 62743–62767. External Links: Link Cited by: §2.

Appendix A Evidence Scope and Claim Hierarchy

The experiments are deliberately layered because live search couples a changing generator to an accumulating population. Table 3 states the statistical unit and the strongest claim supported by each layer. The three-seed TSP–Qwen cohort is the only full multi-seed live efficacy study; other task/model cells establish mechanism or directional breadth.

Table 3: Evidence hierarchy and claim map. “Shared” identifies quantities held identical across conditions. A dash under seeds means that the unit is a shared record group rather than an independently generated search trajectory.

Layer	Tasks	Models	Seeds	Shared evidence	Claim supported
Normalizer replay	TSP, CVRP, OBP, OP	recorded generators	–	evaluated records	whether reward differences survive group normalization
Matched update	TSP, CVRP	Qwen2.5-7B, DeepSeek-Coder-7B	1/cell	tokens, masks, reference log probabilities, update budget	whether an advantage difference reaches parameter updates
Formal live cohort	TSP	Qwen2.5-7B	3	starts, seeds, 500 groups	proposal-stream and equal-horizon live-search effects
Frozen restart	TSP	Qwen2.5-7B	3	final adapter, reset population, zero updates	checkpoint effect after removing accumulated live population
Directional breadth	CVRP; TSP (DeepSeek only)	Qwen2.5-7B (CVRP); DeepSeek-Coder-7B	1/cell	task/model protocol within cell	directional breadth across three task–model cells, not multi-seed efficacy
Resource anchor	TSP	Qwen2.5-7B	3	preregistered timing-derived group budgets	endpoint sensitivity to allocating compute to extra Frozen search
Common-context probe	CVRP	Qwen2.5-7B	2	64 prompts, four sampling positions	executable checkpoint behavior on identical contexts
API system reference	TSP	provider-reported GPT-4o-mini	3	2,000-completion budget only	completion-matched frozen-system reference, not a causal model comparison

No result is pooled across these evidence roles as if the layers were repeated estimates of a single estimand.

Appendix B Reward Construction and Normalizer Details

Native CALM branch semantics

For completion $i$ in group $t$ , Native CALM first assigns typed invalid rewards

r^{\mathrm{N}}_{t,i}\in\{-1,-0.95,-0.90,-0.85,-0.75\}

(8)

for missing rationale, missing code, missing required function, runtime failure, and detected randomness, respectively. A valid initialization completion receives zero. Otherwise let $f$ be the task score and $b$ the best prompt-base score. With

\delta(f,b)=\operatorname{clip}\!\left(\frac{|f-b|}{\min\{|f|,|b|\}},10^{-10},1\right),

(9)

the released implementation assigns $1+\delta$ to an improvement, zero to numerical equality, $-3\delta/8$ to a degradation, and $-3/5$ to the branch for a candidate identified as one of the prompt bases. Thus Native reward is already context-relative and nonlinear; it is not raw task performance.

Factorized Validity–Quality

Let $v_{i}=1$ for a valid executable completion and $v_{i}=-1$ otherwise. For the eligible valid subset $V_{t}$ , let $\Delta_{i}=f_{i}-b_{t}$ and

q_{i}=\begin{cases}\Delta_{i}/\sqrt{|V_{t}|^{-1}\sum_{j\in V_{t}}\Delta_{j}^{2}},&i\in V_{t},\\ 0,&i\notin V_{t}.\end{cases}

(10)

The quality channel is set to zero when its RMS is numerically zero, for initialization without a comparator, or when no valid candidate is eligible. The scalar reward is $r^{\mathrm{F}}_{i}=(v_{i}+q_{i})/2$ , followed by one joint GRPO normalization. The construction is factorized at the evidence level, not as two losses or two independently normalized objectives.

Pre-Generation Tail Weighting

The Tail-weighted construction uses only the population and frontier captured before generating the group. A tie-aware midrank utility orders typed failures below valid candidates; within valid candidates it encodes score novelty with respect to the pre-generation archive and exact repetition within the response group. It retains positive frontier-gap magnitude through

g_{i}=\frac{[f_{i}-F_{t}]_{+}}{\sqrt{|V_{t}|^{-1}\sum_{j\in V_{t}}[f_{j}-F_{t}]_{+}^{2}}},\qquad w_{i}=u_{i}+\tfrac{1}{2}g_{i}.

(11)

The gap channel is zero when its denominator is zero. We then use the adaptive concentration rule attributed and discussed in the main paper:

	$\displaystyle\omega_{i}(\beta)$	$\displaystyle=\frac{\exp(\beta w_{i})}{\sum_{j}\exp(\beta w_{j})},$		(12)
	$\displaystyle D_{\mathrm{KL}}(\boldsymbol{\omega}(\beta)\\|U_{G})$	$\displaystyle=\min\{\log 2,\log(G/k)\}.$		(12)

where $k$ is the number of tied maxima. The returned reward is $G\omega_{i}$ . When a tied maximum makes the target attainable only as $\beta\rightarrow\infty$ , the implementation returns the limiting distribution uniform over the maxima. A constant group returns the uniform distribution. This group-weight KL is unrelated to the policy–reference KL coefficient in GRPO.

Search-Exposure Residual

This mechanism probe counterfactually inserts one candidate into the frozen pre-generation population and evaluates its one-step exposure through CALM’s primary-parent and crossover-secondary routes. Let $\rho_{i}$ denote total exposure and $\Delta_{i}=f_{i}-b_{t}$ . After centering on eligible valid rows, the search component removes the immediate-quality direction:

$\displaystyle\mathbf{q}$	$\displaystyle=\frac{\operatorname{ctr}(\boldsymbol{\Delta})}{\operatorname{RMS}(\operatorname{ctr}(\boldsymbol{\Delta}))+\epsilon},$	(13)
$\displaystyle\mathbf{m}$	$\displaystyle=\operatorname{ctr}(\boldsymbol{\rho}\odot\boldsymbol{\Delta}),$
$\displaystyle\mathbf{s}^{\perp}$	$\displaystyle=\mathbf{m}-\frac{\langle\mathbf{m},\mathbf{q}\rangle}{\langle\mathbf{q},\mathbf{q}\rangle+\epsilon}\mathbf{q},$
$\displaystyle\mathbf{a}^{S}$	$\displaystyle=\frac{\mathbf{s}^{\perp}}{\operatorname{RMS}(\operatorname{ctr}(\boldsymbol{\Delta}))+2\operatorname{RMS}(\mathbf{s}^{\perp})+\epsilon}.$

Here $\epsilon=10^{-8}$ . The quality and centered typed-validity channels use their respective RMS scales and are set to zero when that scale is at most $\epsilon$ . The residual is deliberately not normalized to unit RMS: the denominator above bounds its RMS below $1/2$ and prevents a numerically tiny residual from becoming a full-strength channel. The raw learner reward is the sum of the validity, quality, and $\mathbf{a}^{S}$ channels before the fixed group normalizer. This probe changes neither the actual parent sampler nor population transition, and is not assigned live-efficacy status in the paper.

Implemented learner equivalence

For group rewards $\mathbf{r}$ , the implementation uses

\mathcal{N}(\mathbf{r})=\frac{\mathbf{r}-\bar{r}\mathbf{1}}{s(\mathbf{r})+10^{-4}}.

(14)

For any shared offset $b$ ,

\mathcal{N}(\mathbf{r}+b\mathbf{1})=\mathcal{N}(\mathbf{r})

(15)

exactly. For $a>0$ ,

\mathcal{N}(a\mathbf{r})=\frac{a(\mathbf{r}-\bar{r}\mathbf{1})}{as(\mathbf{r})+10^{-4}},

(16)

so positive rescaling is approximately, rather than exactly, invariant when $as(\mathbf{r})\gg 10^{-4}$ . Low-variance groups can retain a small difference. Finally, a threshold tier that is itself monotone in performance and uses the same performance order inside each tier induces the same total order as performance alone. A subsequent midrank therefore leaves the learner signal unchanged. This no-op was verified on both synthetic and recorded groups before the final Tail-weighted construction represented frontier gap as a numeric component rather than a redundant tier.

Appendix C Implementation and Reproducibility

Source boundary and software

The search host is pinned to the recorded CALM source revision ecb3cadcf4b0. The intervention wrapper executes the native evaluator and population transition before replacing only the learner-facing scalar reward. Formal runs record resolved configuration, source-bundle identifier, source cleanliness, adapter checksums, and SHA-256 checksums of completion and step records.

Table 4: Principal software environment recorded by formal manifests.

Python	3.10.20	PyTorch	2.5.1
Transformers	4.49.0	TRL	0.15.1
PEFT	0.14.0	Unsloth	2025.3.18
vLLM	0.7.3	Ray	2.40.0
NumPy	1.26.4	bitsandbytes	0.45.2

Hardware

Table 5: Hardware used by reported experiments. CPU and operating-system fields were not consistently captured on the historical remote 3090 and A100 hosts; we report this absence instead of inferring their models. Wall-clock claims use only the preregistered timing-derived resource protocol and do not pool throughput across these heterogeneous hosts.

Host class	Accelerator	CPU / memory / OS	Evidence role
Local workstation	3 $\times$ NVIDIA RTX 4090, 24,564 MiB each; driver 550.144.03	Intel Xeon w5-2455X, 12 cores/24 threads; 125 GiB RAM; Linux 5.15	most formal live runs, including at least one run from every condition; Qwen breadth; Gate-4 prefixes; API-served frozen runs; common-context probe
Shared GPU host	4 $\times$ NVIDIA RTX 3090, 24,576 MiB each	CPU model, RAM, and OS not archived in formal manifests	selected Tail-weighted live/restart runs; one Frozen formal seed; DeepSeek-Coder breadth; sequential Frozen references
A100 host	1 $\times$ NVIDIA A100, 40,960 MiB	CPU model, RAM, and OS not archived in formal manifests	selected mechanism and breadth cells; one Tail-weighted formal seed

The CALM evaluator is CPU-intensive. Historical shared-host records show periods of CPU contention on the shared GPU host; those observations are not used as model-speed evidence. Hardware identifiers and runtime snapshots are not reported; accelerator class and memory are sufficient to describe the resource context without exposing machine-specific details.

Final experimental parameters

Table 6: Final parameters for the formal local-model experiments. Values not explicitly overridden in the runner are the recorded TRL 0.15.1 defaults shown here.

Search / generation	Value	Optimization	Value
Task / live model	TSP / Qwen2.5-7B-Instruct	Adapter	LoRA rank 32, alpha 64
Seeds	42, 3407, 1926000	Target modules	q, k, v, o, gate, up, down projections
Groups / completions	500 / 2,000	Quantization	4-bit base model
Prompts per step / group size	1 / 4	Learning rate	$5\times 10^{-5}$ , constant
Population size	10	Optimizer	8-bit AdamW
Operator weights	simplification 1; injection 1; replacement 2; crossover 4	Adam betas / weight decay	0.9, 0.99 / 0.1
Stagnation threshold	25 groups	Warmup / max grad norm	0 / 0.1
Prompt / completion limit	2,048 / 1,024 tokens	Batch / grad accumulation	1 / 1
Model context	4,096 tokens	Epochs / optimizer opportunities	1 / 500
Sampling temperature	0.9	Policy–reference KL coefficient	0.04
vLLM memory utilization	0.8	Precision	BF16 when supported, else FP16

Python random, NumPy, Hugging Face/TRL, and LoRA initialization receive the run seed. The ACO evaluators use fixed Torch generators. CUDA, vLLM, and Ray execution are not claimed to be bitwise deterministic; the independent run seed is therefore the statistical unit. Formal completion records retain outcome, score, prompt/operator context, raw reward components, scalar reward, normalized advantage, archive event, response hash, and pre-generation frontier. Prompt and parent hashes provide coverage diagnostics without claiming semantic diversity.

Appendix D Complete Formal and Breadth Results

Seed-level formal cohort

Table 7: Complete three-seed TSP–Qwen live cohort. Panel (a) reports mean

\pm

sample standard deviation; bold marks the highest descriptive mean in each column and does not imply statistical significance. Panel (b) lists all seed-level records. Higher is better for all performance columns; Valid and Improve are percentages, and every seed contains 2,000 completions.

(a) Aggregate comparison

Condition	Final best	Trajectory AUC	Valid (%)	Valid-only perf.	Improve (%)
Frozen	-6.2329 $\pm$ 0.0105	-6.2417 $\pm$ 0.0046	60.85 $\pm$ 0.55	-9.8800 $\pm$ 0.8639	2.83 $\pm$ 0.50
Native	-6.2090 $\pm$ 0.0289	-6.2203 $\pm$ 0.0196	93.92 $\pm$ 1.68	-6.4966 $\pm$ 0.0986	2.98 $\pm$ 1.16
Factorized	-6.2041 $\pm$ 0.0116	-6.2179 $\pm$ 0.0087	95.60 $\pm$ 0.64	-6.7488 $\pm$ 0.1168	3.60 $\pm$ 1.25
Tail-weighted	-6.1963 $\pm$ 0.0248	-6.2123 $\pm$ 0.0148	85.12 $\pm$ 2.50	-7.0468 $\pm$ 0.2214	4.63 $\pm$ 0.33

(b) Seed-level records

Condition	Seed	Final best	AUC	Valid	Valid-only	Improve	Frontier	Training tokens
Frozen	42	-6.2240	-6.2399	60.85	-10.7900	3.35	0.40	0
Frozen	3407	-6.2303	-6.2384	60.30	-9.0710	2.35	0.30	0
Frozen	1926000	-6.2445	-6.2470	61.40	-9.7790	2.80	0.15	0
Native	42	-6.1771	-6.2015	94.60	-6.5016	2.55	0.80	1,197,560
Native	3407	-6.2163	-6.2189	95.15	-6.3956	4.30	0.30	638,135
Native	1926000	-6.2336	-6.2406	92.00	-6.5925	2.10	0.50	1,050,993
Factorized	42	-6.1973	-6.2172	95.45	-6.8650	3.70	0.45	954,138
Factorized	3407	-6.2174	-6.2270	95.05	-6.7501	4.80	0.50	976,358
Factorized	1926000	-6.1975	-6.2095	96.30	-6.6314	2.30	0.55	820,341
Tail-weighted	42	-6.2117	-6.2237	82.60	-7.2974	4.25	0.60	1,163,619
Tail-weighted	3407	-6.1677	-6.1956	87.60	-6.8776	4.85	1.55	1,246,899
Tail-weighted	1926000	-6.2095	-6.2176	85.15	-6.9655	4.80	0.70	1,149,907

Appendix E Search–Learning Interaction Diagnostics

Figure 4: Within-seed paired differences for the formal live cohort. Diamonds denote means and bars the observed three-seed range. Positive values favor the first condition. Differences against Frozen combine online updating with the named reward construction; differences against Native isolate reward construction within the same online-training host.

Figure 5: Stage-wise proposal behavior over early (groups 1–100), middle (101–250), and late (251–500) search. These values describe each condition’s endogenous live contexts; they are not fixed-prompt checkpoint evaluations.

Task and model breadth

Table 8: Single-seed directional live-search breadth. These cells assess directional consistency across tasks and model families and do not support uncertainty or aggregate superiority claims.

Task	Model	Variant	Final best	Trajectory AUC	Valid (%)	Valid-only perf.	Improve (%)
CVRP	DeepSeek Coder	Frozen	-9.0676	-9.3136	49.20	-12.2861	3.35
CVRP	DeepSeek Coder	Native	-8.7337	-8.8805	81.00	-10.6994	3.45
CVRP	DeepSeek Coder	Factorized	-8.9977	-9.0951	81.50	-11.2008	2.85
CVRP	Qwen2.5	Frozen	-9.1271	-9.4523	63.90	-11.0669	5.60
CVRP	Qwen2.5	Native	-9.1517	-9.4046	88.85	-10.3968	5.00
CVRP	Qwen2.5	Factorized	-9.1097	-9.4950	94.80	-10.5522	5.60
TSP	DeepSeek Coder	Frozen	-6.2424	-6.2494	55.60	-8.4719	2.65
TSP	DeepSeek Coder	Native	-6.2445	-6.2566	87.55	-8.8024	2.45
TSP	DeepSeek Coder	Factorized	-6.2403	-6.2497	89.70	-7.1444	1.75

The breadth cells are single-seed directional evidence. Online updating raises validity in all six trained conditions across the three task–model cells relative to their Frozen control, but final-best and AUC directions vary. They therefore support the separation between feasibility learning and downstream search efficacy, not a cross-task superiority claim.

Refer to caption — Figure 6: Directional live breadth in three single-seed cells spanning TSP/CVRP-ACO and two 7B model families. Markers show observed directions rather than uncertainty estimates.

Figure 7: Diagnostic atlas from the 12-run formal cohort. Panels decompose operator-conditioned proposal funnels, archive admission versus immediate frontier contribution, collapse-centered observational changes, and exact response recurrence. Coverage and collapse panels are descriptive and do not identify causal effects of individual search operators.

Operators and proposal outcomes

Against Frozen, Native and Factorized improve validity for every operator in all three matched seeds. Mean validity gains range from 19.1–86.7 percentage points for Native and 19.5–91.7 points for Factorized, with injection showing the largest increase because Frozen injection frequently fails before producing an executable program. Tail-weighted also raises operator-level validity, but by a smaller 14.2–58.1 points. These changes do not translate monotonically to parent improvement or strict-frontier improvement. For example, Native crossover increases parent-improvement rate by 1.62 points on average and is positive in 3/3 seeds, whereas its replacement and simplification differences are negative on average. Operator-conditioned validity is therefore one location of training gain, not a sufficient explanation of discovery.

Population admission is not learner credit

Across formal runs, archive admission is much more frequent than strict frontier improvement. Frontier precision among admissions is only $0.37$ – $4.95\%$ for Native/Factorized/Frozen and $0.95$ – $2.14\%$ for Tail-weighted. This is expected because CALM’s population supports search coverage and later parent construction, not only immediate frontier advances. It also demonstrates why added_to_archive cannot be treated as a synonym for positive learner credit or strict discovery.

Credit allocation and base rates

Valid non-improvers constitute $90.9\%$ , $92.0\%$ , and $80.5\%$ of candidates under Native, Factorized, and Tail-weighted, and receive $83.3\%$ , $83.5\%$ , and $85.5\%$ of positive advantage mass. The corresponding mass-to-prevalence ratios are $0.916$ , $0.908$ , and $1.062$ . Immediate parent improvements are rare but strongly enriched: their mass-to-prevalence ratios are $5.19$ , $4.34$ , and $2.83$ ; the corresponding strict-frontier ratios are $6.72$ , $6.10$ , and $3.98$ . Strict-frontier candidates receive positive advantage in $100\%$ , $100\%$ , and $96.15\%$ of their occurrences, respectively. Thus the dominant class absorbs most mass by volume, while immediate improvements are disproportionately reinforced. The result is not evidence that the $83\%$ mass is erroneous; it quantifies the difference between peer-relative credit and immediate search progress.

Collapse and exact recurrence

Collapse events reset the population but not the learned adapter. In collapse-centered windows, validity and valid-only performance changes are mixed and high variance; the analysis is observational because collapse timing depends on the preceding trajectory. Exact-response recurrence is higher for positive than nonpositive completions in most trained seeds. Within 25 later groups, the positive-minus-nonpositive recurrence difference ranges from $1.66$ to $3.88$ percentage points for Native, from $-0.42$ to $1.04$ points for Factorized, and from $0.27$ to $1.91$ points for Tail-weighted. These hashes detect exact replay only; they do not measure semantic or algorithmic diversity.

Appendix F Checkpoint, Search-State, Generalization, and Resources

Live versus population-reset search

Figure 8: Same-seed live and frozen-restart results. Restart conditions load a final adapter, reset the population, and execute zero optimizer steps. A live endpoint therefore includes both checkpoint and accumulated-population effects, whereas restart tests the checkpoint in a new search trajectory.

For Tail-weighted, the restarted checkpoint beats the initial Frozen model in final best in all three seeds, with deltas $+0.0054$ , $+0.0075$ , and $+0.0444$ (mean $+0.0191$ ). Relative to its own live endpoint, however, restart deltas are $-0.0069$ , $-0.0551$ , and $+0.0095$ (mean $-0.0175$ ). The parent checkpoint checksum equals the restart initial checksum in every seed; restart runs have zero training tokens and zero optimizer steps. The checkpoint therefore retains useful behavior beyond the initial model, while the live endpoint still contains substantial search-state contribution.

Held-out TSP sizes

Figure 9: Held-out TSP evaluation of final heuristics at the available problem sizes. This evaluates discovered heuristic artifacts, not fixed-context checkpoint generation. Seed-level points remain visible.

Held-out artifact evaluation uses CALM’s native TSP evaluator at all available problem sizes. Because search selects a final heuristic using the training evaluator, held-out performance is reported separately from live proposal validity and fixed-context checkpoint behavior.

Timing-derived Frozen-search anchors

Table 9: Three-seed Gate-4 endpoint comparisons under preregistered, timing-derived Frozen-search group budgets.

\Delta

is online minus Frozen, so higher values favor online updating. Frozen metrics are recomputed from exact stored prefixes. The budgets are not exact realized-time matches; full-run timing drift and unavailable timestamps for shorter prefixes are reported in the text.

Seed	Online condition	Groups (online/Frozen)	Online final	Frozen final	$\Delta$ final	$\Delta$ valid (pp)	$\Delta$ valid-only
42	Native	500/883	-6.2176	-6.2167	-0.0010	+27.01	+0.742
42	Factorized	500/762	-6.2221	-6.2167	-0.0055	+25.42	+0.545
3407	Native	500/572	-6.2217	-6.2303	+0.0086	+34.45	+2.360
3407	Factorized	500/536	-6.1939	-6.2303	+0.0365	+34.43	+2.004
1926000	Native	500/581	-6.2419	-6.2003	-0.0415	+39.88	+1.045
1926000	Factorized	500/500	-6.2208	-6.2003	-0.0205	+40.50	+1.617

Native and Factorized each beat the longer Frozen prefix in one of three seeds. Their mean online-minus-Frozen final-best deltas are $-0.0113$ and $+0.0035$ , respectively. Online updating nevertheless increases validity and valid-only performance in every row. The measured full-run time gap is $+1.61\%$ for seed 42 and $-13.56\%$ / $-12.48\%$ for seeds 3407/1926000; shorter Factorized prefixes lack timestamps. We consequently call these preregistered timing-derived group-budget anchors, not exact realized-time matches. Unequal-horizon AUC is not used for a causal comparison.

Appendix G Common-Context Executable Checkpoint Probe

The main study’s matched-update probe shows that a reward difference reaches adapter parameters and fixed-context token probabilities. To test executable behavior more directly, we additionally evaluate six final CVRP–Qwen checkpoints on a common bank of 64 prompts. Each checkpoint generates four completions per prompt from the same prompt positions, for 256 completions and 1,536 total records. This probe uses two checkpoint seeds and was completed after the main experiment freeze; it is supplementary diagnostic evidence.

Table 10: Common-context executable CVRP–Qwen checkpoint probe. The fixed prompt bank makes validity and contextual improvement directly comparable within a seed. “Improve

\mid

valid” conditions on valid completions; “Best-of-4 improve” is the fraction of the 64 prompts whose sampled group contains an improvement. Higher is better.

Seed	Checkpoint	Valid / 256	Valid (%)	Valid-only perf.	Improve $\mid$ valid (%)	Best-of-4 improve (%)
3407	Frozen	7	2.73	-8.9086	0.00	0.00
3407	Native	12	4.69	-8.8106	8.33	1.56
3407	Factorized	33	12.89	-9.5113	12.12	3.12
1926000	Frozen	7	2.73	-8.9086	0.00	0.00
1926000	Native	8	3.12	-8.8336	0.00	0.00
1926000	Factorized	3	1.17	-8.6397	0.00	0.00

Factorized has the highest validity and improvement rate in seed 3407 but the lowest validity in seed 1926000. Native is above Frozen in validity in both seeds, but only seed 3407 produces an improvement. The probe therefore confirms that executable checkpoint effects can be measured under identical contexts, while providing no stable two-seed ordering among reward constructions. Its low absolute validity also shows that a fixed CVRP prompt bank can be more difficult than the endogenous live contexts generated by each search arm.

Appendix H API-Served Frozen-System Reference

The external evidence directory contains several API diagnostics. Only one cohort satisfies the precondition of three completed 2,000-completion runs: a provider-reported gpt-4o-mini model used as a frozen generator in the CALM search loop. The API served one completion per request and therefore used 2,000 sequential groups, whereas local conditions used 500 four-completion groups. The completion budget matches, but prompt grouping, latency, provider, and model are different. The model name is provider-reported and was not independently verified. We report this as a system reference, not as a causal model-only baseline or a claim against an official proprietary service.

Table 11: Completed provider-reported GPT-4o-mini frozen-system reference. Each row uses 2,000 sequential completions and zero updates. Higher is better for performance columns. Rows are repeated seeds rather than competing methods, so no best-seed value is highlighted.

Seed	Valid (%)	Valid-only	Improve (%)	Frontier (%)	Final best	AUC	Prompt tok.	Completion tok.
42	82.05	-9.0526	3.25	0.25	-6.2202	-6.2435	1,745,317	672,208
3407	82.15	-9.8845	4.30	0.30	-6.2254	-6.2395	1,890,022	713,864
1926000	83.15	-9.6897	4.65	0.75	-6.2109	-6.2518	1,842,358	711,707
Mean	82.45	-9.5423	4.07	0.43	-6.2188	-6.2449	1,825,899	699,260

At the same total completion count, the API-served reference has mean validity $82.45\%$ and final best $-6.2188$ . The grouped local three-seed means are $60.85\%/-6.2329$ for Frozen, $93.92\%/-6.2090$ for Native, $95.60\%/-6.2041$ for Factorized, and $85.12\%/-6.1963$ for Tail-weighted. These descriptive values do not isolate model strength: local conditions have four-way group generation and, for trained arms, online parameter updates. The comparison only shows where one completed API-served frozen system falls under its own CALM protocol.

DeepSeek-v4-flash runs stopped at 845–1,281 of 2,000 completions because of repeated empty-content responses; DeepSeek-v4-pro and GPT-5.5 runs were also incomplete. They are excluded from every efficacy table. No partial endpoint or best-so-far value from those runs is used to rank systems.

Appendix I Provenance, Exclusions, and Reproduction Map

The arXiv source package accompanying this document contains the manuscript sources, bibliography, style files, figures, and tables needed to reproduce the submitted PDF. The underlying experiment code and raw records are not included in this LaTeX package. The reported results are based on the following materials retained during the study:

•

the pinned CALM source revision, formal source-bundle patches, and intervention runner;
•

formal and breadth configuration files with local model paths omitted from the public package;
•

unit tests for reward mappings, permutation/tie invariance, record schemas, and evidence builders;
•

completion- and step-level records for the 12 formal live runs, excluding adapter checkpoints;
•

deterministic analysis scripts and aggregate CSV files used by the main paper and this supplement;
•

sanitized common-context and API summaries with source-file hashes;
•

the environment specification and figure/table build commands.

Credentials, API endpoints, machine usernames, personal absolute paths, GPU UUIDs, raw provider responses, model weights, and multi-gigabyte adapter checkpoints are excluded from the submitted package. Local model weights must be obtained from their original distributors. All incomplete, killed, and smoke-only runs remain outside formal result builders.