Harness-Agnostic Detection and Immunization of Reward Hacking in Self-Evolving Language Models

The design tension we have to resolve is that a fixed, host-visible probe set would itself be optimized against and would simply become a second $M$. We resolve it by splitting probes into two layers with orthogonal roles.

The comparison core $P^{\mathrm{core}}$ is a distribution-fixed, secret probe set covering the target capability across $S$ subdomains. Because its distribution never changes, the proxy $\widehat{C}(\theta)=\Phi\big(\psi(\theta,P^{\mathrm{core}}_{(1)}),\dots,\psi(\theta,P^{\mathrm{core}}_{(S)})\big)$, with $\psi$ a per-subdomain score and $\Phi$ an aggregate robust enough that no single subdomain can move it alone, yields differences $\Delta\widehat{C}_{t}$ that track capability change and not test drift. Freezing the distribution has a second payoff, less obvious than the first. The same items are scored at $t-1$ and at $t$, so the noise in $\Delta\widehat{C}_{t}$ is paired: per-item difficulty cancels, and the variance of the difference is far smaller than twice that of a single generation’s estimate. The core is never revealed to the host and never enters its selection or reward path.

The fresh layer $P^{\mathrm{fresh}}_{t}$ is rotated, generated online, and drifted every generation. It is used only for hardening checks and qualitative per-subdomain profiling and never contributes to $\Delta\widehat{C}_{t}$. Four principles govern it, and they pull in different directions on purpose. An air gap (H1) keeps probes out of the host’s training, selection and reward loop entirely, so neither probe text nor per-subdomain scores are ever fed back; on top of that, rotation (H2) makes sure any single instance is used at most a few times. Out-of-distribution drift (H3) then steers the layer toward whichever subdomains the host is currently weakest on, so the bank is always probing capability that has not yet been fitted, while adversarial refresh (H4) has a red-team model periodically manufacture probes that are harder and more surface-diverse still. H4’s two selection criteria — learnability (probes near the host’s competence edge) and surface diversity (embedding distance from probes already used) — are deliberately independent of the detection signal, so refresh cannot self-amplify into false positives. Non-circularity of the final measurement survives all of this because $A_{\mathrm{cal}}$ and $A_{\mathrm{eval}}$ stay disjoint and $A_{\mathrm{eval}}$ is spent exactly once, at the end.

HackProbe fuses four one-sided tests, each calibrated on honest generations so that its null $p$-value is approximately uniform on $[0,1]$. Their blind spots are deliberately different, and fusing them extends coverage across the level, divergence, stagnation and conditional-confidence regimes; format-level exploits, which act on output structure rather than on the score–capability relation, fall outside all four (Section 4).

Statistic J0 targets hacking that is present from the very first generation, where $M$ is already inflated and then stays flat so that every first-difference statistic sees nothing. We calibrate the honest affine relation $M\approx a+\lambda_{\mathrm{lvl}}\widehat{C}$ and test the residual level gap $g^{(0)}_{t}=M_{t}-(a+\lambda_{\mathrm{lvl}}\widehat{C}_{t})$, which hovers near zero under honest evolution and is persistently positive under level-type hacking. Local linear coupling makes $\lambda_{\mathrm{lvl}}$ and the difference scale $\lambda$ coincide in population; we estimate them separately so each absorbs its own drift. Statistics J1 and J2 handle the dynamic regime. On $D_{t}$ from Eq. (1) we run a one-sided test against the honest null, together with an online Page–Hinkley change-point detector (Page, 1954) on the cumulative divergence that localizes the generation at which $M$ and $\widehat{C}$ decouple. J2 tests capability stagnation directly, taking the honest null $H_{0}:\mathbb{E}[\Delta\widehat{C}_{t}]\geq 0$ conditional on $\Delta M_{t}>0$ and rejecting when the measured $\Delta\widehat{C}_{t}$ is too small, which gives a $\lambda$-free complement.

Statistic J3 exists because self-consistency bias inflates $M$ by rewarding agreement instead of correctness, which need not move aggregate accuracy at all. Once such an attack saturates, J1 and J2 see nothing in expectation and J0 retains only the residue the attack leaves in the aggregate level, diluted over all core items including the many answered correctly. On the secret core we measure, over probes answered incorrectly, the mean self-consistency $\mathrm{cwr}_{t}$ (the confidently-wrong rate), calibrate its honest affine dependence on core accuracy $\mathrm{acc}_{t}$, and test the residual $g^{(3)}_{t}=\mathrm{cwr}_{t}-(a_{3}+b_{3}\,\mathrm{acc}_{t})$, with a Page–Hinkley detector for gradual drift. Conditioning on wrong answers avoids the ceiling effect of easy probes, and using only the secret core keeps the signal non-circular. Lemma 1 in Appendix E makes precise why J3 retains an expected separation equal to the full attack size in the stable regime where J1 and J2 vanish in expectation.

Fusion takes $m=6$ one-sided $p$-values: J0, the divergence test J1 and its Page–Hinkley alarm, the stagnation test J2, and J3 with its own alarm. The two sequential alarms are converted to $p$-values through their calibrated alarm-time distribution on honest runs, putting all six on a common scale. Given $p_{1},\dots,p_{m}$ approximately independent and uniform under the honest null, we form the risk score $r_{t}=(1-\min_{j}p_{j})^{m}$ and flag hacking when $r_{t}\geq\tau$. This is exactly a Šidák correction (Šidák, 1967), and it has a calibration property that a raw $1-\min_{j}p_{j}$ rule lacks. Under the independent-uniform null $\Pr[\min_{j}p_{j}\leq u]=1-(1-u)^{m}$, so that

so $1-r_{t}$ is the Šidák-adjusted $p$-value and $\tau$ is the operating point: flagging at $\tau$ costs family-wise error exactly $1-\tau$, however many statistics we fuse, whereas the uncorrected rule degrades with $m$. In practice the guarantee is softer than that. J0, J1 and J2 are built from the same pair $(M,\widehat{C})$ and are not independent, so Eq. (2) is a design principle and the reported operating point rests instead on re-calibrating $\tau$ until the empirical false-positive rate on honest runs meets its target; Appendix H gives the Bonferroni fallback we use when even that is in doubt. The control is also per generation, so over $T$ generations the expected number of false alarms is $T(1-\tau)$ and run-level error would need a sequential correction we do not attempt. What a deployment sees, and what Table 1 reports, is the per-generation alarm rate.

Diagnosis alone recovers nothing. The tempting picture, in which rejecting or rolling back the top candidate suffices, fails on two counts: by the time the host is told, it has already passed over the honest candidate in favor of the top-$M$ one, and rejection has nothing better to fall back to. HackProbe therefore reselects from the proposal pool, and the loss it recovers is a real one for two reasons. Under strict final-answer grading, the true-capability side ($A_{\mathrm{eval}}$ and the core $\widehat{C}$) credits only a normalized final answer matching the gold one, so behaviors that inflate a lenient visible grader, such as enumerating many candidate numbers or locking onto an unverified answer, genuinely forfeit true credit. And every pool contains, alongside the exploit variants, one enforced honest candidate produced under a clean context, whose visible $M$ is typically lower but whose true $A_{\mathrm{eval}}$ is higher.

For each candidate $c$ we compute a purely structural gaming footprint $gf(c)$ from cached outputs at no extra inference cost, combining the score-versus-commitment gap with the fraction of non-final candidate artifacts, and a robust score $\mathrm{robust}(c)=\widehat{C}(c)-\gamma\,gf_{\mathrm{norm}}(c)$ with a fixed penalty weight $\gamma>0$. Four immunization levels build on it, each trusting the host’s own score a little less than the last. F0 is diagnosis without protection. It selects $\arg\max_{c}M(c)$, so it goes straight to the inflated candidate. F1 stays greedy by default and intervenes only when a lightweight risk proxy on the greedy choice, either its raw gap $M-\widehat{C}$ or its footprint $gf$, comes back high; then it reselects $\arg\max_{c}\mathrm{robust}(c)$, and after $K$ consecutive high-risk generations it rolls back to the checkpoint with the best historical $\widehat{C}$. F2 goes further, selecting $\arg\max_{c}(M(c)-u(c))$ with $u(c)$ a quantization of the raw gap $M(c)-\widehat{C}(c)$ into $L$ levels (the uncalibrated difference, not the J0 residual $g^{(0)}$). It also watches for self-destruction, whose signature is feedback $u$ falling while $\widehat{C}$ stagnates, and downgrades itself to F1 when it sees it. F3 adds a coverage bonus for collapsed subdomains and injects a diversity constraint into the proposer.

The quantization in F2 is where the bandwidth constraint stops being cosmetic. Were $u$ the exact gap, $M(c)-u(c)$ would collapse to $\widehat{C}(c)$ and F2 would be pure core-greedy selection, the failure we are trying to avoid: it would expose the secret core as a fully-resolved second objective. With $L$ levels, F2 interpolates between the host’s greedy rule ($L=1$, nothing transferred) and core-greedy selection ($L\to\infty$). The leak needs careful accounting. Per-candidate values $u(c)$ stay inside the monitor and are never returned; the host sees only which of $\Pi$ candidates won, so the return channel carries at most $\log_{2}\Pi$ bits per generation, three at our $\Pi=8$, and $T\log_{2}\Pi$ over a run of length $T$. Quoting $\log_{2}L$ here would be tighter but wrong: the winning index can say more about the vector of $u$ values than any single $u(c)$ does, and it is easy to construct pools where it does. What $L$ buys is resolution rather than volume, and resolution is what limits how much of the core’s ordering the host can reconstruct. Either way the cumulative figure must stay small against the entropy needed to identify core items, so long runs need $L$ small and the core retired once its budget is spent. The gold core enters only through $\widehat{C}(c)$, never through $M$ or an oracle label, and only the selected candidate advances the detector state.

What follows rests on standard concentration assumptions, given in full as Assumptions 1–4 in Appendix C: the core estimate is $\widehat{C}(\theta)=C(\theta)+\beta(\theta)+\xi$ with bounded bias and sub-Gaussian noise, the paired difference $\Delta\widehat{C}_{t}$ has variance proxy $v_{C}/n$ and the visible score $v_{M}/n_{M}$, and honest updates satisfy $\mu_{t}\leq 0$ with the differential core bias bounded by $b_{\Delta}$ in every generation. Then $D_{t}$ is sub-Gaussian with $\sigma_{D}^{2}=\lambda^{2}v_{C}/n+v_{M}/n_{M}$, and its honest mean is at most $\mu_{H}:=-\lambda\,\mathbb{E}[\Delta\beta(\theta_{t})]$, where $|\mu_{H}|\leq\lambda b_{\Delta}$. Proofs are in Appendix D.

The value of Proposition 1 is not the standard exponential separation but the budget in Eq. (4): detecting a divergence of size $\delta$ at error $\alpha$ needs a core growing like $1/\delta^{2}$, which turns an abstract design choice into a number one can procure and falsify against measured detection-rate-versus-$n$ curves. The differential bias $\mu_{H}$ does not enter the budget, since it shifts the honest and hacking means equally and is subtracted during calibration.

Rotation has a companion bound, Proposition 2 in Appendix D: an attacker who memorizes past instances needs $k\geq N(1-\alpha_{\mathrm{atk}})^{1/n_{f}}$ of them to match all $n_{f}$ fresh probes of a generation, which tends to the whole pool as $n_{f}$ grows. It lives in the appendix because it is narrow in a way worth admitting. It bounds memorization, and nothing else. An attacker who instead learns the generator’s regularities walks around it: if the exploitable behavior admits an $\varepsilon$-cover of size $\mathcal{N}(\varepsilon)$, defeating rotation costs $\Theta(\mathcal{N}(\varepsilon))$ rather than $\Theta(N)$, so security degrades from pool size to covering number, and a low-complexity generator buys no asymptotic protection at all. Enlarging $\mathcal{N}(\varepsilon)$ is what H3 and H4 are for. “Surface-diverse rotation forces generalization” is an empirical hypothesis here, not a theorem.

Table 1 reports detection across all channels pooled. Three of the five baselines are degenerate: no-probe, the contamination test and the single trap set flag every generation, so recall and false-positive rate are both exactly $1.000$ and precision is exactly the base rate $0.241$. Their F1 scores are evidence of nothing. What separates them is AUROC, running from $0.462$, below chance because the contamination test ranks hacking generations lower, up to $0.568$.

The real competitor is the absolute-$\widehat{C}$ trajectory, which uses the same secret core but watches its level rather than its divergence from $M$, reaching $0.663$ AUROC. HackProbe adds $0.100$ AUROC and, for a monitor meant to run continuously, does something more useful: it cuts the false-positive rate from $0.706$ to $0.434$ and lifts precision from $0.311$ to $0.382$. It pays with recall of $0.845$ rather than $1.000$ and a delay of $0.28$ generations, both consequences of being the only method whose false-positive rate falls below one half. Even so, $0.434$ interrupts more than four honest generations in ten, and we take that gap as the bar for this task. Most of it is a budget problem rather than a design one. Sweeping the core size at a fixed false-positive rate (Appendix F) puts our $n=240$ on the steepest part of the detection curve, where Eq. (4) would want $600$ probes for $0.90$ detection; the monitor is under-provisioned by roughly $360$.

Detection is a means; the objective is the true capability that survives evolution. Table 2 reports, per channel, the end-of-evolution $\Delta C$ of the greedy host F0 and the additional gain of each immunization level, all measured on the held-out $A_{\mathrm{eval}}$.

Every level improves on greedy selection on average, though not uniformly across channels. Risk-gated (F1) and bandwidth-limited (F2) reselection are positive on all four and reach nearly identical averages, $+0.053$ and $+0.052$; what separates them is price, not gain (Section 4.3). F1 wins biggest on the gameable grader ($+0.104$), where the raw gap $M-\widehat{C}$ gating its reselection is largest, even though that is the channel the fused detector ranks worst. F3 holds the single largest gain over F0, $+0.109$ on judge collusion, where a lenient judge collapses the proposer onto a narrow region and the coverage bonus pulls it back out. F3 is also the only level that goes negative anywhere, mildly, on self-consistency bias.

With gold audit data this scarce, only the two starred intervals lie strictly above zero, and two more, both under F2, have their lower endpoint exactly at zero. So the table supports an aggregate, directional claim; per-channel effects are mostly too small to certify, and sharpening them needs a larger gold audit set.

A monitor that helps under attack but degrades ordinary runs is not deployable, so we measure the clean-setting cost directly on the none channel, where greedy selection is in fact the right thing to do. Table 3 reports it alongside the diversity-bonus sweep that fixes F3’s setting.

Every level pays something on clean runs: each declines the top-$M$ candidate some of the time, and when $M$ is honest that candidate was correct. The useful comparison is what a level costs when nothing is wrong against what it returns when something is. Only F2 comes out favorable, returning $0.052$ for a cost of $0.047$, a ratio of $1.11$; F1 buys the largest gains but pays most for them ($0.053$ against $0.073$, ratio $0.73$), and F3 sits between ($0.035$ against $0.042$, ratio $0.84$). The right panel shows why F3 uses a small bonus: $0.05$ gives both the best average under hacking and the smallest clean-run cost, and $0.25$ erases the gain.

The ratio’s standing is narrow. It orders the three levels, being a comparison of two averages over the same runs that leans on no single interval in Table 2, but it does not license reading per-channel gains as effects, and at $1.11$ a larger audit set could move it. It would be falsified by a host on which reselection costs more than it returns at every bandwidth, a regime Appendix F shows is not far away.

A detector tuned on the same attacks it is evaluated on proves little, so we calibrate on three channels and evaluate on the held-out fourth (Table 4). Transfer is good to judge collusion ($0.814$ AUROC), whose accumulating leniency is the change-point J1 is built for, and to self-consistency bias ($0.787$), which leaves the signature J3 targets; contamination remains usable at $0.724$. It degrades on the gameable grader ($0.600$ AUROC, $0.661$ false-positive rate) for an instructive reason: grader exploits act on output format, not on the score–capability relation, so a threshold calibrated on the other three over-flags, and covering them needs a statistic keyed to output structure that the fusion lacks. The two held-out AUROCs above the pooled $0.763$ are no transfer bonus, since ranking within one channel is easier than across four whose divergences differ in scale.