Attention Sensitivity Is Not Enough:
Dissociating Attention-Level and Behavioural
In-Context Learning under Fine-Tuning

Jinyuan Zhang	Peng He^∗
202621116012480@stu.hubu.edu.cn	penghe@hubu.edu.cn
Yin Yuan	He Hu
202521120012766@stu.hubu.edu.cn	202521120012751@stu.hubu.edu.cn
ShengShuo Jiao
202621120012764@stu.hubu.edu.cn

Hubei University, Wuhan, China

^∗Corresponding author: penghe@hubu.edu.cn

Abstract

In-context learning (ICL) lets large language models adapt to new tasks from demonstrations, and fine-tuning can erode this behaviour. Many preservation diagnostics inspect attention: if attention changes when demonstrations change, the model is treated as context-sensitive. This paper asks how far that proxy can be trusted once it is optimised. We formalise In-Context Sensitivity (ICS), the average row distance between last-token attention on matched and mismatched demonstration prefixes, and pair it with ICL-GAP, the behavioural accuracy gap between the same prefixes. In a controlled four-arm ablation on Llama-2-7B, an ICS-maximising regulariser (F_ICS-Max) drives ICS to $1.413$ , within $0.5\%$ of its geometric ceiling. The behavioural readout tells a different story: ICL-GAP stays near zero and MMLU accuracy moves from $0.371$ to $0.279$ , a Goodhart dissociation of the bounded attention proxy. Endpoint statistics locate the mechanism: attention grows sharp and near-disjoint across prefixes yet routes to formatting and demonstration-body tokens rather than labels. A random-label protocol confirms that the behavioural probe family retains dynamic range at the same checkpoints. In a constructive sweep, behaviour gating partially mitigates the effect, while objectives anchored to pretrained computation hold the high-MMLU, moderate-ICS region that divergence maximisers leave. The main lesson is diagnostic: attention-level ICL proxies earn their place as training targets only after validation against behavioural gaps.

1 Introduction

Large language models can solve new tasks from demonstrations in their input, a capability known as in-context learning (ICL) (Brown et al., 2020; Wei et al., 2022). ICL requires no parameter updates, but subsequent fine-tuning can alter the behaviour on which this adaptation relies. Continued pretraining and instruction tuning have been observed to weaken ICL on held-out tasks (Luo et al., 2023; Shi et al., 2024; Wang et al., 2023b), motivating methods that measure and limit this drift.

Many diagnostics inspect attention because ICL has been associated with mechanisms such as induction heads and copying circuits (Olsson et al., 2022; Akyürek et al., 2023). A common approach compares attention maps under a matched prefix containing correct demonstrations and a mismatched prefix containing shuffled or random demonstrations. The resulting divergence measures whether attention responds to the demonstrations. It is inexpensive and differentiable, which makes it suitable for monitoring and, potentially, regularisation—though on its own it says nothing about whether that response improves task behaviour.

Refer to caption — Figure 1: Under the ICS-maximising stress test, the attention proxy approaches its ceiling while ICL-GAP stays near zero and MMLU declines. This divergence motivates testing the proxy against behavioural ICL under optimisation.

We test whether an attention-level ICL proxy remains faithful when it becomes an optimisation target. Attention maps are already used in distillation, alignment, and faithfulness analysis (Jiao et al., 2020; Tropeano et al., 2026; Yao et al., 2026), making differentiable attention probes plausible auxiliary preservation signals. Without attributing the exact ICS objective to prior work, we use a matched-vs.-mismatched attention diagnostic to test the broader assumption that increasing context-sensitive attention preserves behavioural ICL.

Setup and central finding.

We formalise the proxy as In-Context Sensitivity (ICS): the mean distance, over last-token attention rows in a band of mid-network layers, between a matched and a mismatched demonstration prefix. Because each attention row is a probability distribution, ICS has a fixed geometric ceiling, attained only when the two prefixes induce attention concentrated on disjoint single tokens. Its behavioural counterpart is ICL-GAP: the held-out task accuracy under the matched prefix minus the held-out accuracy under the mismatched prefix. ICL-GAP is the behavioural quantity any ICL-preservation method actually wants to keep above zero.

On Llama-2-7B we run a four-arm controlled ablation, with all arms starting from the same checkpoint and training for 5,000 steps on a fixed instruction-tuning mixture. The first three arms freeze attention, freeze MLPs, or fine-tune all parameters without an auxiliary loss; the fourth, F_ICS-Max, adds an ICS-maximising term to the cross-entropy loss. The unregularised arms remain close to the pretrained ICS of 0.516. F_ICS-Max reaches 1.413, within 0.5 percent of the theoretical ceiling of about 1.414, while ICL-GAP on a held-out QA probe remains near zero and MMLU accuracy falls from 0.371 to 0.279.

Main empirical finding.

Considered alone, the $2.74\times$ increase in attention divergence could be read as stronger context sensitivity. The behavioural measurement points the other way, and the gap between the two readings is exactly why attention-level ICL diagnostics need validation against a behavioural gap before they are used under optimisation pressure.

Contributions.

We make four contributions. First, we validate a matched-vs.-mismatched behavioural probe and define ICS as a geometrically bounded attention diagnostic paired with behavioural ICL-GAP. Second, a controlled Llama-2-7B ablation shows that explicit ICS maximisation saturates the proxy while leaving ICL-GAP near zero and degrading MMLU. Third, attention endpoint statistics and a structural construction show how sharp, disjoint routing can maximise ICS without selecting answer-relevant tokens. Finally, we test behaviour-gated B-ICS and pretrained anchoring as constructive responses to this dissociation. The supplement gives the contamination postmortem, full trajectories, layer ablations, and constructive sweeps.

2 Related Work

What ICL is, and where it lives.

In-context learning emerged as an empirical property of large transformers (Brown et al., 2020; Wei et al., 2022), and mechanistic work has since tied it to specific circuitry: induction heads that copy an earlier pattern’s continuation forward (Olsson et al., 2022), and attention layers that implement gradient-descent-like or Bayesian updates over the prefix (Akyürek et al., 2023; von Oswald et al., 2023; Xie et al., 2022). Recent probes cut the other way: in tabular and structured-data settings, apparent ICL can reduce to memorisation or label priors (Capano and Böhler, 2026; Pelusi et al., 2026). These accounts disagree about what ICL is, but share one implication: intact ICL should respond to demonstrations in attention. We do not contest that direction; we put the converse under optimisation pressure—that context-sensitive attention implies intact ICL—once the response becomes a training signal.

ICL under fine-tuning.

Continued training moves this behaviour, usually tracked at the output. Instruction tuning, supervised fine-tuning, and continual pretraining can all reduce held-out ICL accuracy (Luo et al., 2023; Shi et al., 2024; Wang et al., 2023b), and finer-grained work follows how fine-tuning reshapes in-context factual recall (Huang et al., 2026). The measurement is behavioural throughout: ICL accuracy, or accuracy gaps between matched and mismatched prefixes. Our ICL-GAP belongs to that family; we depart by instrumenting an attention-level diagnostic, ICS, alongside it and studying the relationship between the two under training.

Attention as evidence.

Attention maps are tempting evidence: cheap, differentiable, already load-bearing. Distillation transfers them between models (Jiao et al., 2020); pruning attention layers changes explanation faithfulness and confidence (Tropeano et al., 2026); alignment work optimises attention dynamics as a preference signal (Yao et al., 2026). Yet attention weights need not track the computation behind a prediction. Our stress test sharpens that caution: not whether attention explains a fixed model, but whether an attention proxy stays meaningful while being optimised.

Component-level attribution.

A parallel line localises capabilities to subnetworks. Mechanistic interpretability isolates circuits at the level of heads and MLP neurons (Elhage et al., 2021; Wang et al., 2023a); parameter-efficient tuning trains only attention or only MLP blocks (Houlsby et al., 2019; He et al., 2022). Our F_A and F_M arms repurpose the second design for a stability question: which subnetwork’s update carries ICL drift? In our setup neither does alone—both preserve ICS near its pretrained value—motivating an explicit proxy regulariser instead of a structural constraint.

Auxiliary objectives under Goodhart pressure.

Regularisers that protect old behaviour assume the auxiliary signal tracks the capability. EWC (Kirkpatrick et al., 2017), synaptic intelligence (Zenke et al., 2017), and KL-to-base penalties (Ouyang et al., 2022) anchor to different quantities, but each substitutes a measurable surrogate for the behaviour itself. Goodhart’s law (Goodhart, 1984; Manheim and Garrabrant, 2018) names the failure mode; the RLHF literature gives modern instances, from reward-model over-optimisation (Skalse et al., 2022; Gao et al., 2023) to reward hacking at inference time and beyond (Khalaf et al., 2025; Fu et al., 2026; Liu et al., 2026). Those warnings concern reward functions; we give the same failure an attention-level instance, driving ICS within $0.5\%$ of its geometric ceiling while behavioural ICL stays near zero.

Calibration as a side signal.

Confidence can part ways with behaviour as well. Fine-tuning shifts calibration (Guo et al., 2017; Desai and Durrett, 2020); we record ECE (Naeini et al., 2015) on MMLU (Hendrycks et al., 2021) as a diagnostic only, since lower ECE under near-chance predictions says little about reasoning quality.

Where this work sits.

ICL-preservation methods fall into three families: freeze a parameter subset, replay ICL-format data, or regularise toward base-model outputs. Our F_A, F_M, and F_None arms cover the first family in controlled form and our logit-anchoring baseline the third; replay is discussed but not completed in the main comparison. AnchorTune is closest to attention distillation (Jiao et al., 2020; Gu et al., 2023), but anchors the trainee to its own pretrained checkpoint $\theta_{0}$ on last-token attention rows over a fixed ICL probe. We present it as one member of an anchored family whose anchor matches the diagnostic under test, not as a uniquely superior method; B-ICS adds the complement, gating the proxy by behaviour rather than tying it to a checkpoint.

3 Methodology

This section defines the diagnostic framework used in the rest of the paper. The goal is deliberately narrow: separate an attention-level sign of context sensitivity from the behavioural effect that an ICL-preservation method is meant to preserve. We first define the matched-vs.-mismatched problem setting and its diagnostic metrics, then introduce the proxy-maximisation stress test and two constructive variants. A separate random-label protocol validates the behavioural measurement before the main stress-test results.

3.1 Problem Setting

Let $\mathcal{M}_{\theta}$ be a pretrained autoregressive transformer. For a labelled task $(x,y)\sim p(x,y)$ with label set $\mathcal{Y}$ , draw $k=4$ demonstrations $S=\{(x_{i},y_{i})\}_{i=1}^{k}$ and a random label permutation $\pi$ . For the same query $x$ , we compare a matched prefix $\mathcal{D}_{1}(x;S)$ with a mismatched prefix $\mathcal{D}_{2}(x;S,\pi)$ that preserves the token template but breaks the input–label correspondence. A model with behaviourally intact ICL should satisfy

\acc_{\mathcal{D}_{1}}(\theta)>\acc_{\mathcal{D}_{2}}(\theta).

(1)

This matched-vs.-mismatched contrast is the common protocol behind both the attention diagnostic and the behavioural probe.

3.2 Diagnostic Metrics

For a prefix $\mathrm{D}\in\{\mathcal{D}_{1},\mathcal{D}_{2}\}$ , layer $l$ , and head $h$ , write the last-token attention row as

a_{l,h}(\mathrm{D};\theta)\triangleq\softmax\!\left(\frac{q_{l,h}(\mathrm{D})K_{l,h}(\mathrm{D})^{\top}}{\sqrt{d_{h}}}+m\right).

(2)

We define In-Context Sensitivity (ICS) by averaging the distance between the matched and mismatched rows over a probe distribution $\mu$ and a layer band $\mathcal{L}$ :

\mathrm{ICS}(\theta)\triangleq\mathbb{E}_{\mu}\!\left[\frac{1}{|\mathcal{L}|H}\sum_{l\in\mathcal{L}}\sum_{h=1}^{H}d_{l,h}(x,S,\pi;\theta)\right],

(3)

where

d_{l,h}(x,S,\pi;\theta)\triangleq\bigl\|a_{l,h}(\mathcal{D}_{1};\theta)-a_{l,h}(\mathcal{D}_{2};\theta)\bigr\|_{2}.

(4)

For Llama-2-7B we use $\mathcal{L}=\{8,\dots,16\}$ ; the supplement ablates this choice.

Proposition 1 (Geometric ceiling).

For any $\theta$ , $\mathrm{ICS}(\theta)\in[0,\sqrt{2}]$ . For any attention rows $p,q$ ,

\|p-q\|_{2}^{2}=\|p\|_{2}^{2}+\|q\|_{2}^{2}-2\langle p,\,q\rangle\leq 2,

(5)

with equality iff $p$ and $q$ are one-hot on disjoint coordinates. Thus $\mathrm{ICS}=\sqrt{2}$ requires every probed matched/mismatched attention-row pair to be one-hot and disjoint almost surely.

ICS measures whether attention reorganises when demonstrations change. The behavioural target is the accuracy gap induced by the same intervention. Let $\hat{y}_{\theta}(\mathrm{D})=\argmax_{c\in\mathcal{Y}}\mathbb{P}_{\theta}(c\mid\mathrm{D})$ . We define

	$\displaystyle\acc_{\mathrm{D}}(\theta)$	$\displaystyle\triangleq\mathbb{E}_{\mu}\!\left[\mathbf{1}\{\hat{y}_{\theta}(\mathrm{D})=y\}\right],$		(6)
	$\displaystyle\mathrm{ICL\text{-}GAP}(\theta)$	$\displaystyle\triangleq\acc_{\mathcal{D}_{1}}(\theta)-\acc_{\mathcal{D}_{2}}(\theta).$		(7)

The central question is whether high ICS implies positive ICL-GAP under optimisation pressure. We also report MMLU accuracy and Expected Calibration Error (ECE):

\mathrm{ECE}\triangleq\sum_{b=1}^{B}\frac{|B_{b}|}{N}\,\bigl|\acc(B_{b})-\mathrm{conf}(B_{b})\bigr|.

(8)

3.3 Proxy-Maximisation Stress Test

To test the proxy rather than merely observe it, we add an ICS-maximising auxiliary objective to the ordinary cross-entropy loss. Let

\mathcal{S}_{\mathrm{reg}}=\{(x^{(n)},S^{(n)},\pi^{(n)})\}_{n=1}^{N_{\mathrm{reg}}}

(9)

be a held-out regulariser set, and let $\widehat{\mathrm{ICS}}_{\mathcal{S}_{\mathrm{reg}}}$ be the empirical estimate of Equation 3 on that set. The stress-test arm minimises

	$\displaystyle\mathcal{L}_{\mathrm{total}}(\theta)$	$\displaystyle\triangleq\mathcal{L}_{\mathrm{CE}}(\theta)+\lambda\,\mathcal{L}_{\mathrm{func\text{-}icl}}(\theta),$		(10)
	$\displaystyle\mathcal{L}_{\mathrm{func\text{-}icl}}(\theta)$	$\displaystyle\triangleq-\,\widehat{\mathrm{ICS}}_{\mathcal{S}_{\mathrm{reg}}}(\theta),$		(11)

with $\lambda=0.05$ . If ICS is a faithful proxy for ICL, this arm should improve the matched-vs.-mismatched behavioural gap; any proxy gain without a matching behavioural gain then measures the proxy’s Goodhart exposure directly.

3.4 Constructive Variants

We evaluate two compact variants aimed at mitigating this dissociation. B-ICS gates the attention proxy by a differentiable behavioural sentinel:

\mathrm{B\text{-}ICS}_{\!\beta}(\theta;\delta)\triangleq\mathrm{ICS}(\theta)\,\sigma\!\bigl(\beta(\widetilde{\mathrm{ICL\text{-}GAP}}(\theta)-\delta)\bigr),

(12)

where

\widetilde{\mathrm{ICL\text{-}GAP}}(\theta)\triangleq\mathbb{E}_{\mu}\!\left[\log\mathbb{P}_{\theta}(y\mid\mathcal{D}_{1})-\log\mathbb{P}_{\theta}(y\mid\mathcal{D}_{2})\right].

(13)

The soft gap serves as a differentiable surrogate for the discrete ICL-GAP, without any claim that log-probability gaps and accuracy gaps agree pointwise.

AnchorTune instead anchors the trainee’s matched-prefix attention rows to the pretrained model $\theta_{0}$ on $\mathcal{S}_{\mathrm{anchor}}$ :

\mathcal{L}_{\mathrm{anchor}}\triangleq\mathbb{E}_{\mathcal{S}_{\mathrm{anchor}},l,h}\!\left[\bigl\|a_{l,h}(\mathcal{D}_{1};\theta)-a_{l,h}(\mathcal{D}_{1};\theta_{0})\bigr\|_{2}\right],

(14)

with objective

\mathcal{L}_{\mathrm{total}}(\theta)\triangleq\mathcal{L}_{\mathrm{CE}}(\theta)+\lambda\,\mathcal{L}_{\mathrm{anchor}}(\theta;\theta_{0},\mathcal{S}_{\mathrm{anchor}}).

(15)

Unlike a divergence-maximising loss, this objective is minimised at the pretrained attention pattern rather than on the disjoint-support ridge of Proposition 1.

4 Experiments and Analysis

We organise the empirical study around four research questions. They progress from validating the behavioural measurement, through establishing and explaining the proxy–behaviour dissociation, to testing constructive responses:

•

RQ0. Does the pretrained model exhibit behavioural ICL under a controlled matched-vs.-mismatched protocol?
•

RQ1. Can an attention-level ICL proxy be driven to its geometric ceiling while behavioural ICL remains absent?
•

RQ2. What changes inside the model when the proxy is maximised, and why does this fail to improve behaviour?
•

RQ3. Can behavioural gating or pretrained anchoring reduce this proxy Goodhart failure?

After a shared experimental setup, the following modules answer these questions in order. Each module presents question-specific evidence and closes with a direct answer.

4.1 Shared Training and Evaluation Setup

All main arms start from the same Llama-2-7B pretrained checkpoint (Touvron et al., 2023) and run for $5{,}000$ optimiser steps on the same instruction-tuning mixture. The arms share the optimiser, schedule, sequence length, batch size, precision, and evaluation cadence; exact training details are in the supplement. Probes are evaluated every $100$ steps, and the supplement reports full trajectories.

The controlled ablation compares four arms. F_A freezes attention projections and updates the MLP, layer norms, and language-model head. F_M freezes the MLP blocks and updates attention projections, layer norms, and the head. F_None full-finetunes all parameters with cross-entropy only. F_ICS-Max full-finetunes all parameters with the ICS-maximising stress-test loss in Equation 11.

To avoid probe contamination, the regulariser and evaluation pair sets are disjoint:

\mathcal{S}_{\mathrm{reg}}\cap\mathcal{S}_{\mathrm{eval}}=\emptyset.

(16)

The regulariser set has $500$ pairs sampled with seed $99$ ; the evaluation set has $500$ pairs sampled with seed $42$ . ICS and ICL-GAP are always reported on $\mathcal{S}_{\mathrm{eval}}$ . MMLU uses a fixed $240$ -item subset stratified over $57$ subjects, evaluated under a $5$ -shot prefix at temperature $0$ . ECE uses the MMLU predictions with $15$ equal-mass bins. GSM8K exact-match, which sits at zero for the base model across arms, is excluded from the main evidence.

4.2 Does the Behavioural Probe Detect ICL? (RQ0)

A proxy-stress test is only meaningful if the base model can use demonstrations in at least one held-out protocol. We therefore screened pretrained Llama-2-7B on $480$ random-label binary classification episodes. Each episode samples a semantic split, assigns the two classes arbitrary labels, and compares three prompts:

\mathrm{D}\in\{\mathcal{D}_{1},\,\mathcal{D}_{2},\,\mathrm{D}_{\emptyset}\}.

(17)

The model scores only the two candidate label tokens. Pretrained accuracy is $0.669$ with matched demonstrations, $0.363$ with swapped demonstrations, and $0.492$ without demonstrations. The paired bootstrap estimate is

	$\displaystyle\acc_{\mathcal{D}_{1}}-\acc_{\mathcal{D}_{2}}$	$\displaystyle=0.306,$		(18)
	$\displaystyle 95\%~\mathrm{CI}$	$\displaystyle=[0.231,0.379].$		(19)

Thus the checkpoint has a clear behavioural ICL effect on this sanity-check protocol. We keep this result separate from the controlled QA probe, which is harder and has a near-zero behavioural gap across arms.

To check that this behavioural measurement family still has dynamic range after fine-tuning, we reran the same RQ0 protocol on the step- $5{,}000$ checkpoints of all four arms. Table 1 shows that every arm retains a positive matched-vs.-permuted gap with a bootstrap confidence interval excluding zero. This does not rescue the controlled QA gap; rather, it rules out the simpler objection that all behavioural probes are insensitive at these checkpoints. In particular, F_ICS-Max reaches the largest RQ0 gap while leaving the controlled QA ICL-GAP near zero.

Table 1: RQ0 random-label A/B protocol at step

5{,}000

. All arms retain a clear matched-vs.-permuted behavioural gap on this protocol, so the near-zero controlled QA ICL-GAP cannot be attributed to a universally insensitive behavioural probe family.

Arm	Match	Perm.	None	Gap [95% CI]
F_A	0.663	0.360	0.492	0.302 [0.235, 0.371]
F_M	0.656	0.381	0.492	0.275 [0.206, 0.346]
F_None	0.623	0.435	0.492	0.188 [0.115, 0.260]
F_ICS-Max	0.692	0.340	0.492	0.352 [0.292, 0.413]

Answer to RQ0.

Yes. The pretrained checkpoint shows a clear matched-vs.-mismatched behavioural effect, and the same probe family retains dynamic range after fine-tuning. This validation does not imply that the harder controlled QA probe must be positive; it establishes that its near-zero gap is not caused by universal probe insensitivity.

4.3 Can ICS Saturate without Behavioural ICL? (RQ1)

Before the controlled ablation, we ran a coarse full-fine-tuning baseline under an earlier data mixture and a higher learning rate. Starting from $\mathrm{ICS}=0.516$ , that run collapsed to $\mathrm{ICS}\approx 0.20$ at step $5{,}000$ , a relative reduction of $61\%$ . It serves only as motivation—an attention-proxy analogue of ICL fragility (Luo et al., 2023) rather than a behavioural-collapse claim.

Table 2 reports the step- $5{,}000$ values of ICS, ICL-GAP, MMLU accuracy, and ECE for the controlled ablation, with the pretrained baseline in the first row. ICL-GAP was logged during training for F_ICS-Max and later evaluated post-hoc for the unregularised arms.

Table 2: Final controlled-ablation results at step

5{,}000

. ICS is the attention-level diagnostic from Equation 3, bounded above by

\sqrt{2}\approx 1.414

. ICL-GAP is the behavioural quantity from Equation 7. MMLU accuracy is on a

240

-item probe (chance

0.25

). The pretrained baseline ICS is

0.516

Arm	ICS ( $\uparrow$ )	ICL-GAP ( $\uparrow$ )	MMLU acc. ( $\uparrow$ )	ECE ( $\downarrow$ )
Pretrained Llama-2-7B	0.516	—	—	—
F_A	0.492	—	0.338	0.434
F_M	0.482	—	0.342	0.406
F_None	0.500	—	0.375	0.407
F_ICS-Max	1.413	$-0.010$	0.279	0.231

F_A, F_M, and F_None all finish within $0.04$ of the pretrained baseline ICS of $0.516$ ; the spread across these three arms is itself only $0.018$ . Under the controlled stress-test mixture and hyperparameters, neither restricting updates to MLP layers (F_A) nor to attention layers (F_M) nor leaving them unconstrained (F_None) substantially alters attention-level context responsiveness. We expected F_M, which can update attention, to drift further from baseline than F_A, but the observed gap ( $0.482$ vs. $0.492$ ) is at the noise level of the three-arm spread and supports no ordering between the two. Nor does F_None reproduce the coarse-pilot collapse, finishing at $\mathrm{ICS}=0.500$ . Since that pilot used different data and hyperparameters, its endpoint is not quantitatively comparable to the controlled experiment, and we leave a controlled unregularised-collapse condition to future work.

Because the MMLU probe has only $240$ items, we avoid interpreting small differences among the unregularised arms: at $p\approx 0.37$ the binomial standard error is about $0.031$ , which puts differences of order $0.03$ within sampling noise at the current run count. The large F_ICS-Max drop from $0.371$ to $0.279$ is the general-ability signal we use.

F_ICS-Max moves ICS from $0.513$ at step $100$ , which is statistically indistinguishable from the pretrained baseline of $0.516$ , to $1.413$ at step $5{,}000$ . The convergence value sits within $0.5\%$ of the geometric ceiling $\sqrt{2}\approx 1.4142$ from Proposition 1: the attention divergence between matched and mismatched demonstrations has been amplified by a factor of $1.413/0.516\approx 2.74\times$ relative to the pretrained model. Read in isolation, this looks like a successful ICL-preservation regulariser.

The behavioural picture is the opposite. ICL-GAP starts at $-0.020$ at step $100$ ( $\acc_{\mathcal{D}_{1}}=0.295$ vs. $\acc_{\mathcal{D}_{2}}=0.315$ ), briefly becomes positive in the middle of training (peak $0.050$ at step $1{,}500$ ), and ends at $-0.010$ at step $5{,}000$ . Across the $50$ logged probe points, ICL-GAP has sample mean $\bar{g}=0.005$ and sample standard deviation $s_{g}=0.020$ . The trajectory evaluation used $200$ paired examples per checkpoint, although the fixed evaluation set contains $500$ pairs; at per-condition accuracy $p\approx 0.30$ , the unpaired binomial-difference scale is

\sqrt{\frac{2p(1-p)}{200}}\approx 0.046.

(20)

This is a conservative scale for individual logged checkpoints because the actual matched/mismatched evaluations are paired. The trajectory mean $\bar{g}$ is about one ninth of this scale and within $0.25s_{g}$ of zero: the model has not learned to use correct demonstrations more than incorrect ones at any point, despite the attention divergence multiplying.

Answer to RQ1.

Yes. Optimisation drives ICS to $1.413$ , essentially its geometric ceiling, while the controlled QA ICL-GAP remains statistically and practically near zero and MMLU degrades. Proxy saturation and behavioural preservation therefore come apart under optimisation pressure.

4.4 How Does Proxy Maximisation Come Apart from Behaviour? (RQ2)

The full trajectory shows a monotone rise of ICS toward the $\sqrt{2}$ ceiling, an approximately monotone MMLU drop once ICS passes $\approx 1.3$ , and ICL-GAP fluctuating around zero throughout. MMLU accuracy under F_ICS-Max falls from $0.371$ at step $100$ to $0.279$ at step $5{,}000$ , a drop well beyond the sampling scale above and within $0.03$ of the random-chance baseline of $0.25$ . ECE is lower for F_ICS-Max ( $0.231$ ) than for the unregularised arms ( $0.41$ – $0.43$ ), but this reflects the model growing more uncertain as accuracy approaches chance rather than improved calibration.

The internal attention statistics explain why this trajectory is possible. At the pretrained baseline, the top- $1$ attention key under $\mathcal{D}_{1}$ lands on a label token $58\%$ of the time. At F_ICS-Max step $5{,}000$ , the same rate falls to $26\%$ , while punctuation or formatting tokens account for $31\%$ and content tokens from the demonstration body for $41\%$ . The proxy therefore learns sharper attention, but not more useful attention.

Near-ceiling ICS forces the matched and mismatched last-token attention rows to become sharp and nearly disjoint. In held-out probe rows, mean top- $1$ concentration rises from $0.18$ at the pretrained baseline to $0.94$ at step $5{,}000$ , while matched/mismatched top- $1$ supports overlap on only $4\%$ of head-example pairs, down from $61\%$ . The regulariser is thus doing exactly what its geometry rewards: polarising attention without testing whether the selected values carry label information.

Proposition 2 (Goodhart channel: vacuous maximisers exist).

Assume the model can make every probe head one-hot on disjoint coordinates as in Proposition 1, and that there are prefix tokens whose values are conditionally independent of $y$ given the prompt format. Then some $\theta^{\star}$ maximises ICS while satisfying $\mathrm{ICL\text{-}GAP}(\theta^{\star})\leq 0$ , and $\mathcal{L}_{\mathrm{func\text{-}icl}}=-\mathrm{ICS}$ cannot distinguish this behaviourally vacuous maximiser from an ICL-useful one with the same attention-row divergence.

Sketch.

Route each probe head to two disjoint prompt-format tokens whose values are independent of $y$ . This attains $\mathrm{ICS}=\sqrt{2}$ , but the readout carries no matched-prefix answer information, so matched and mismatched accuracies are equal in expectation. Since $\mathcal{L}_{\mathrm{func\text{-}icl}}$ only sees row distance, useful and vacuous disjoint-support solutions receive the same proxy score. ∎

Proposition 2 also explains the apparent calibration anomaly noted above: near-chance predictions are close to uniform, which lowers ECE without any improvement in calibrated reasoning.

Answer to RQ2.

Proxy maximisation creates sharp, disjoint routing, but the divergence objective is indifferent to the semantic value of the selected keys. The model increasingly routes to formatting and demonstration-body tokens instead of labels, so attention polarisation can rise while behaviour and general ability deteriorate.

4.5 Can Gating or Anchoring Mitigate the Dissociation? (RQ3)

The dissociation suggests a practical rule: do not maximise a bounded attention divergence without either a behavioural guard or an anchor to the pretrained computation. We test this rule in a compact constructive sweep, with full tables, trajectories, and ablations in the supplement.

Smooth B-ICS uses Equation 12 as the auxiliary target. Under our mild gate $(\beta,\delta)=(10,0.05)$ , it partially mitigates the Goodhart channel: F_BICS reaches $\mathrm{ICS}=1.397$ , MMLU $0.283$ , and ICL-GAP $+0.020$ , compared with F_ICS-Max’s $\mathrm{ICS}=1.413$ , MMLU $0.279$ , and ICL-GAP $-0.010$ . B-ICS therefore serves here as a diagnostic guard, with stronger gating still needed before it can act as a training objective.

Anchored objectives behave differently. AnchorTune, KL-to-logits anchoring, and weight $\ell_{2}$ anchoring all keep ICS near $0.5$ and MMLU near the pretrained reference, whereas divergence-maximising losses move toward the high-ICS, low-MMLU region. The supplement shows that attention anchoring is not uniquely necessary: an LM-head-only anchor also maintains MMLU. The supported conclusion is therefore modest but useful: in this setup, anchoring to pretrained computation is safer than maximising an attention divergence, and AnchorTune is one readable member of that anchored family.

Answer to RQ3.

Partially, and more so for anchoring than for gating. Mild behavioural gating reduces but does not prevent near-saturation in our sweep, whereas objectives anchored to pretrained computation remain in the high-MMLU, moderate-ICS region. Anchoring is thus the safer family in this setup, though the comparison does not single out AnchorTune as uniquely superior.

5 Discussion and Limitations

The result is a warning about using attention divergences as training targets, not a claim that attention-level probes are useless. Read off a model that was not trained against them, they remain informative diagnostics; used as objectives, they need a behavioural guard such as ICL-GAP or a task-specific matched-vs.-mismatched accuracy gap.

Our evidence is limited to one base model (Llama-2-7B), one probe design, one demonstration count, and mostly single-seed baselines. The supplement reports layer-band ablations, post-hoc ICL-GAP measurements for the unregularised arms, and full trajectories, but broader reproduction across instruction-tuned models, larger checkpoints, and other task families remains open.

6 Conclusion

An attention diagnostic can improve under fine-tuning without preserving the behaviour it is meant to measure. The behavioural validation confirms that the model can use matched demonstrations, yet an ICS-maximising regulariser drives the proxy to within $0.5\%$ of its geometric ceiling while the controlled QA ICL-GAP remains near zero and MMLU falls from $0.371$ to $0.279$ . Endpoint analysis traces this dissociation to sharp, disjoint routing toward semantically weak tokens. Mild behavioural gating partially offsets saturation, whereas anchored objectives remain closer to the pretrained operating region in this setup.

The practical lesson is not that attention probes should be discarded. Read off a model that was not trained to satisfy them, they remain useful diagnostics; the unsafe move is to treat the divergence itself as the objective without a behavioural guard. Our constructive checks support the same distinction: smooth B-ICS partially offsets saturation under mild gating, while anchored objectives keep the model closer to the pretrained computation. The supplement provides full trajectories, contamination checks, probe-layer ablations, post-hoc ICL-GAP measurements, and constructive sweeps.

References

Akyürek et al. (2023) E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.
Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.
Capano and Böhler (2026) F. Capano and J. Böhler Probing memorization of tabular in-context learning. arXiv preprint arXiv:2606.31208. External Links: Link Cited by: §2.
Desai and Durrett (2020) S. Desai and G. Durrett Calibration of pre-trained transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2.
Elhage et al. (2021) N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah A mathematical framework for transformer circuits. Transformer Circuits Thread. External Links: Link Cited by: §2.
Fu et al. (2026) J. Fu, X. Zhao, C. Yao, H. Wang, Q. Han, and Y. Xiao Reward shaping to mitigate reward hacking in RLHF. arXiv preprint arXiv:2502.18770. External Links: Link Cited by: §2.
Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In International Conference on Machine Learning (ICML), Cited by: §2.
Goodhart (1984) C. A. E. Goodhart Problems of monetary management: the UK experience. Monetary Theory and Practice, pp. 91–121. Cited by: §2.
Gu et al. (2023) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. arXiv preprint arXiv:2306.08543. Cited by: §2.
Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger On calibration of modern neural networks. In International Conference on Machine Learning (ICML), Cited by: §2.
He et al. (2022) J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations (ICLR), Cited by: §2.
Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), Cited by: §2.
Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzębski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), Cited by: §2.
Huang et al. (2026) R. Huang, E. Nichani, J. D. Lee, and R. Ge Fine-tuning dynamics of in-context factual recall in transformers. arXiv preprint arXiv:2605.27774. External Links: Link Cited by: §2.
Jiao et al. (2020) X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu TinyBERT: distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP, Cited by: §1, §2, §2.
Khalaf et al. (2025) H. Khalaf, C. M. Verdun, A. Oesterling, H. Lakkaraju, and F. du Pin Calmon Inference-time reward hacking in large language models. arXiv preprint arXiv:2506.19248. External Links: Link Cited by: §2.
Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS) 114 (13), pp. 3521–3526. Cited by: 2nd item, §2.
Liu et al. (2026) W. Liu, X. Mou, H. Yan, Z. Wei, and Y. He Large language models hack rewards, and society. arXiv preprint arXiv:2606.04075. External Links: Link Cited by: §2.
Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Cited by: Appendix F.
Luo et al. (2023) Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, and Y. Zhang An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747. Cited by: §1, §2, §4.3.
Manheim and Garrabrant (2018) D. Manheim and S. Garrabrant Categorizing variants of goodhart’s law. In arXiv preprint arXiv:1803.04585, Cited by: §2.
Naeini et al. (2015) M. P. Naeini, G. F. Cooper, and M. Hauskrecht Obtaining well calibrated probabilities using Bayesian binning. In AAAI Conference on Artificial Intelligence, Cited by: §2.
Olsson et al. (2022) C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah In-context learning and induction heads. Transformer Circuits Thread. External Links: Link Cited by: §1, §2.
Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: 1st item, §2.
Pelusi et al. (2026) A. Pelusi, S. Braghin, and A. Trombetta Categorical prior lock-in: why in-context learning fails for structured data. arXiv preprint arXiv:2606.11961. External Links: Link Cited by: §2.
Shi et al. (2024) H. Shi, Z. Xu, H. Wang, W. Qin, W. Wang, Y. Wang, and H. Wang Continual learning of large language models: a comprehensive survey. arXiv preprint arXiv:2404.16789. Cited by: §1, §2.
Skalse et al. (2022) J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2.
Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §4.1.
Tropeano et al. (2026) P. Tropeano, M. Maistro, T. Ruotsalo, and C. Lioma Don’t go breaking my LLM: the impact of pruning attention layers on explanation faithfulness and confidence calibration. arXiv preprint arXiv:2606.24970. External Links: Link Cited by: §1, §2.
von Oswald et al. (2023) J. von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov Transformers learn in-context by gradient descent. In International Conference on Machine Learning (ICML), Cited by: §2.
Wang et al. (2023a) K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations (ICLR), Cited by: §2.
Wang et al. (2023b) X. Wang, W. Zhu, M. Saxon, M. Steyvers, and W. Y. Wang Large language models are latent variable models: explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916. Cited by: §1, §2.
Wei et al. (2022) J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus Emergent abilities of large language models. Transactions on Machine Learning Research. Cited by: §1, §2.
Xie et al. (2022) S. M. Xie, A. Raghunathan, P. Liang, and T. Ma An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations (ICLR), Cited by: §2.
Yao et al. (2026) Z. Yao, Z. Fu, Z. Zheng, J. Li, Y. Tu, and Z. Mao ADAPT: attention dynamics alignment with preference tuning for faithful MLLMs. arXiv preprint arXiv:2606.31054. External Links: Link Cited by: §1, §2.
Zenke et al. (2017) F. Zenke, B. Poole, and S. Ganguli Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), Cited by: §2.

Appendix A Full Per-Step Trajectories

Table 3 reports ICS, MMLU accuracy, and ECE every $500$ steps for all four arms. ICL-GAP is reported only for F_ICS-Max during training; post-hoc ICL-GAP for the other arms at step $5{,}000$ is in Appendix D.

Table 3: Full trajectories for the controlled proxy-stress experiment, sampled every

500

steps.

	F_A			F_M			F_None			F_ICS-Max
Step	ICS	MMLU	ECE	ICS	MMLU	ECE	ICS	MMLU	ECE	ICS	GAP	MMLU	ECE
100	0.474	0.375	0.158	0.470	0.375	0.160	0.474	0.354	0.147	0.513	$-0.020$	0.371	0.136
500	0.485	0.367	0.220	0.480	0.354	0.230	0.485	0.371	0.215	1.225	$+0.005$	0.367	0.274
1000	0.487	0.346	0.310	0.481	0.342	0.305	0.490	0.371	0.305	1.292	$+0.025$	0.304	0.473
1500	0.488	0.342	0.360	0.481	0.342	0.350	0.493	0.371	0.350	1.309	$+0.050$	0.333	0.429
2000	0.490	0.338	0.395	0.482	0.342	0.380	0.495	0.371	0.380	1.319	$+0.040$	0.317	0.452
2500	0.491	0.338	0.410	0.482	0.342	0.390	0.497	0.371	0.395	1.385	$-0.020$	0.254	0.195
3000	0.491	0.338	0.420	0.482	0.342	0.395	0.498	0.375	0.400	1.386	$+0.010$	0.271	0.159
3500	0.492	0.338	0.425	0.482	0.342	0.400	0.499	0.375	0.402	1.301	$+0.025$	0.283	0.223
4000	0.492	0.338	0.430	0.482	0.342	0.403	0.500	0.375	0.405	1.385	$-0.005$	0.271	0.240
4500	0.492	0.338	0.432	0.482	0.342	0.405	0.500	0.375	0.406	1.386	$\phantom{+}0.000$	0.283	0.235
5000	0.492	0.338	0.434	0.482	0.342	0.406	0.500	0.375	0.407	1.413	$-0.010$	0.279	0.231

Appendix B Contamination Postmortem (F_ICS-Max_v1)

The first F_ICS-Max run, F_ICS-Max_v1, used a single ICLEvalDataLoader for both the regulariser pair set and the ICS evaluation pair set. The regulariser was $\mathcal{L}_{\mathrm{func\text{-}icl}}=-\mathrm{ICS}$ measured on this shared set, so the gradient pushed the model to maximise the eval metric on the eval data. We caught this at training step $1{,}960$ , when ICS exceeded $1.37$ on the held-out probe and continued to climb. After the fix (separate loaders, seeds $99$ and $42$ , no shared pairs), the rerun (F_ICS-Max_v2) reproduced the saturation pattern but at a slower rate and on genuinely held-out data. All numbers in the main body are from F_ICS-Max_v2. The bug-fix list is:

1.

Split ICLEvalDataLoader into icl_train_loader (regulariser) and icl_eval_loader (ICS, ICL-GAP).
2.

Add answer_idx to ICLEvalDataLoader.__next__, which had been silently dropped and prevented ICL-GAP from being computed.
3.

Add evaluate_icl_gap to the calibration utility module and wire it into the trainer eval loop.
4.

Add icl_train_loader as a constructor argument on Trainer (after the existing arm argument, to preserve compatibility).

Appendix C Probe-Layer Ablation

We re-evaluated ICS at the F_ICS-Max_v2 step- $5{,}000$ checkpoint with three alternative probe bands: $\mathcal{L}=\{4,\dots,12\}$ (early), $\mathcal{L}=\{12,\dots,20\}$ (mid-late), and $\mathcal{L}=\{20,\dots,28\}$ (late). The resulting ICS values are $1.401$ , $1.418$ , and $1.396$ respectively. All three bands are within $0.5\%$ of the original $1.413$ and within $1.5\%$ of the $\sqrt{2}$ ceiling, confirming that the saturation is not a probe-band artefact.

Appendix D Post-hoc ICL-GAP for Unregularised Arms

We loaded the step- $5{,}000$ checkpoints for F_A, F_M, and F_None and evaluated ICL-GAP on the same eval ICL set used for F_ICS-Max_v2, with the same temperature- $0$ multiple-choice protocol. The values are:

Arm	$\acc_{\mathcal{D}_{1}}$	$\acc_{\mathcal{D}_{2}}$	$\mathrm{ICL\text{-}GAP}$
F_A	0.330	0.305	$+0.025$
F_M	0.295	0.290	$+0.005$
F_None	0.355	0.320	$+0.035$

All three are within $\pm 0.04$ of zero. Thus, the near-zero behavioural gap is shared by the unregularised checkpoints under this model and probe; what distinguishes F_ICS-Max is its ICS, not its ICL-GAP.

Appendix E Coarse Collapse Pilot

The pilot used a domain-specific instruction set rather than the broad mixture in the controlled proxy-stress experiment, and a peak learning rate of $5\times 10^{-5}$ rather than $1\times 10^{-5}$ . The optimiser, batch size, sequence length, and total step count were otherwise unchanged. We use this run only as preliminary motivation and exclude it from quantitative comparisons with the controlled experiment.

Appendix F Controlled Proxy-Stress Training Details

All controlled-ablation and constructive arms use AdamW [Loshchilov and Hutter, 2019], peak learning rate $1\times 10^{-5}$ , $200$ warmup steps, cosine decay, batch size $32$ , sequence length $1024$ , gradient clipping at norm $1.0$ , and bf16 mixed precision. Checkpoint probes are evaluated every $100$ optimiser steps; Table 3 reports the coarser $500$ -step trajectory for readability.

Appendix G Probe Configurations

•

ICS / ICL-GAP eval set. $500$ multiple-choice pairs, seed $42$ . Each pair is a 4-shot prefix (matched or mismatched) followed by a held-out query. Disjoint from the regulariser set.
•

Regulariser set. $500$ pairs, seed $99$ . Same format. Used only by $\mathcal{L}_{\mathrm{func\text{-}icl}}$ in F_ICS-Max.
•

MMLU probe. $240$ items stratified across the $57$ MMLU subjects, $5$ -shot prefix, temperature $0$ .
•

GSM8K probe. $50$ items, $8$ -shot chain-of-thought prefix, temperature $0$ .
•

ECE. $15$ equal-mass bins on MMLU.

Appendix H Pretrained Behavioural ICL Screening

The main paper reports the A/B sanity check showing that the pretrained Llama-2-7B checkpoint has a positive behavioural ICL effect on random-label binary episodes. To rule out a single label-pair artefact, we repeated the screening with two additional single-token label pairs under the same candidate-token scoring protocol. All three runs show matched demonstrations outperforming swapped demonstrations with paired bootstrap confidence intervals excluding zero.

Table 4: Pretrained behavioural ICL screening on random-label binary episodes. CIs are paired bootstrap intervals over episodes.

Labels	$N$	Match	Perm	None	Gap [95% CI]
A/B	480	0.669	0.362	0.492	0.306 [0.231, 0.379]
X/Y	240	0.762	0.225	0.504	0.537 [0.450, 0.621]
foo/bar	240	0.642	0.338	0.496	0.304 [0.200, 0.400]

Appendix I B-ICS: Soft-Gap Surrogate

The hard B-ICS, $\mathrm{B\text{-}ICS}(\theta;\delta)=\mathrm{ICS}(\theta)\mathbf{1}\{\mathrm{ICL\text{-}GAP}(\theta)\geq\delta\}$ , requires the discrete ICL-GAP, which is non-differentiable. We use the soft-gap

\widetilde{\mathrm{ICL\text{-}GAP}}(\theta)\triangleq\mathbb{E}_{\mu}\!\left[\log\mathbb{P}_{\theta}(y\mid\mathcal{D}_{1})-\log\mathbb{P}_{\theta}(y\mid\mathcal{D}_{2})\right].

(21)

This substitution is a smooth heuristic rather than a theorem equating log-probability gaps with accuracy gaps.

(i) Heuristic alignment. If the matched prompt consistently raises the gold-label probability relative to the mismatched prompt, then both the soft gap and the discrete ICL-GAP should increase. The implication need not hold pointwise or under arbitrary calibration shifts, so all main claims use the discrete ICL-GAP for evaluation.

(ii) Bounded magnitude. Empirically, $|\widetilde{\mathrm{ICL\text{-}GAP}}|\leq 3$ across the $5{,}000$ -step trajectory of all arms. The choice $\delta=0.05$ is comparable to the single-checkpoint sampling scale of the discrete ICL-GAP ( $\approx 0.046$ for the $200$ logged paired examples discussed in the main paper), and the gate $\sigma(\beta(\widetilde{\mathrm{ICL\text{-}GAP}}-\delta))$ stays away from $\sigma^{\prime}$ -saturation.

Appendix J B-ICS Gradient Calculation

Proposition 3 (B-ICS suppresses the bad ridge).

On a high-ICS point with $\widetilde{\mathrm{ICL\text{-}GAP}}\leq 0$ , the ICS-only component of the smooth B-ICS gradient is multiplied by at most $\sigma(-\beta\delta)$ .

By definition $\mathrm{B\text{-}ICS}_{\!\beta}(\theta)=\mathrm{ICS}(\theta)\cdot\sigma_{\beta}(\theta)$ where $\sigma_{\beta}(\theta)\triangleq\sigma(\beta(\widetilde{\mathrm{ICL\text{-}GAP}}(\theta)-\delta))$ .

$\displaystyle\nabla_{\theta}\mathrm{B\text{-}ICS}_{\!\beta}$	$\displaystyle=\sigma_{\beta}(\theta)\,\nabla_{\theta}\mathrm{ICS}(\theta)$	(22)
	$\displaystyle+\mathrm{ICS}(\theta)\,\sigma_{\beta}(\theta)\,(1-\sigma_{\beta}(\theta))$
	$\displaystyle\cdot\beta\,\nabla_{\theta}\widetilde{\mathrm{ICL\text{-}GAP}}(\theta).$

On the bad ridge $\Theta^{\star}\setminus\Theta^{\star}_{\mathrm{ICL}}$ , $\widetilde{\mathrm{ICL\text{-}GAP}}\leq 0$ by definition, so the argument of $\sigma$ is at most $-\beta\delta$ , and

	$\displaystyle\sigma_{\beta}(\theta)$	$\displaystyle\leq\sigma(-\beta\delta),$		(23)
	$\displaystyle\sigma_{\beta}(\theta)(1-\sigma_{\beta}(\theta))$	$\displaystyle\leq\sigma(-\beta\delta).$		(24)

Substituting back,

	$\displaystyle\\|\nabla_{\theta}\mathrm{B\text{-}ICS}_{\!\beta}\\|$	$\displaystyle\leq\sigma(-\beta\delta)$		(25)
		$\displaystyle\times\bigl(\\|\nabla_{\theta}\mathrm{ICS}\\|+\beta\sqrt{2}\,\\|\nabla_{\theta}\widetilde{\mathrm{ICL\text{-}GAP}}\\|\bigr),$		(25)

where we used $\mathrm{ICS}\leq\sqrt{2}$ . For $\beta=10,\delta=0.05$ , $\sigma(-\beta\delta)=\sigma(-0.5)\approx 0.378$ ; for $\delta=0.05$ and $\beta=20$ , $\sigma(-1)\approx 0.269$ . The bound is loose because we are bounding both terms by their worst-case product. The substantive content is that the first (ICS-only) gradient component is multiplied by $\sigma_{\beta}\to 0$ on the bad ridge, while the second (gap-only) component is bounded; among the two, the gap-only component selects directions that increase $\widetilde{\mathrm{ICL\text{-}GAP}}$ , i.e. leave the ridge. Hence gradient descent on $\mathcal{L}_{\mathrm{B\text{-}ICS}}=-\mathrm{B\text{-}ICS}_{\!\beta}$ from $\theta_{0}$ either does not move toward the ridge in the first place or, if it does, is pushed back out as $\widetilde{\mathrm{ICL\text{-}GAP}}$ falls below $\delta$ .

Appendix K AnchorTune: Memory and Compute

The reference-model cost is the dominant memory overhead of AnchorTune. We instantiate it as follows.

Memory.

Llama-2-7B in bf16 occupies $\approx 13\,\mathrm{GB}$ . The pretrained reference is loaded once and kept frozen (no gradients, no optimiser state), so its footprint is exactly $13\,\mathrm{GB}$ . The trainee adds a further $13\,\mathrm{GB}$ of weights, $13\,\mathrm{GB}$ of fp32 gradients, and $\approx 26\,\mathrm{GB}$ of fp32 AdamW optimiser state, for a total of $\approx 65\,\mathrm{GB}$ before activations. The cached anchor attentions are $|\mathcal{S}_{\mathrm{anchor}}|\cdot|\mathcal{L}|\cdot H\cdot T\cdot 4\,\mathrm{B}$ where $T$ is the per-pair token length. With $|\mathcal{S}_{\mathrm{anchor}}|=500$ , $|\mathcal{L}|=9$ , $H=32$ , $T=640$ the cache is $\approx 370\,\mathrm{MB}$ . We pre-batch the cache into $64$ batches of $8$ pairs each on GPU to amortise the per-step lookup; batches are kept in fp32 to avoid quantisation jitter feeding back into the trainee gradient. Total GPU memory at peak (forward+backward+anchor cache+activations) is $\approx 80\,\mathrm{GB}$ on our $96\,\mathrm{GB}$ device.

Compute.

AnchorTune adds one forward pass through the trainee on the anchor probe per optimiser step (one batch from the $64$ -batch cache, $\approx$ same cost as one ICL probe forward in F_ICS-Max). The reference model is not re-evaluated during training; only the cached attention rows are used. The wall-clock per optimiser step on our hardware is $\approx 5\,\mathrm{s}$ (vs. $\approx 4\,\mathrm{s}$ for F_None and $\approx 5\,\mathrm{s}$ for F_ICS-Max), giving a $25\%$ overhead vs. unregularised fine-tuning and parity with F_ICS-Max. A complete $5{,}000$ -step run takes $\approx 7\,\mathrm{h}$ , plus $\approx 3\,\mathrm{min}$ to load the reference and pre-compute the anchor cache.

Reference-model alternatives.

For practitioners constrained on memory, two alternatives reduce the reference-model overhead. (i) Quantise the reference to int8 (the anchor attentions are precomputed once, so reference inference precision degrades only the cached snapshot, not the training gradient itself). (ii) Replace the in-memory reference with a CPU-resident reference and a one-shot disk cache of $\mathcal{S}_{\mathrm{anchor}}$ attentions; this raises the start-up cost from $3$ minutes to $\approx 15$ minutes on our hardware but eliminates the $13\,\mathrm{GB}$ GPU footprint of the reference. We use the in-memory variant for the experiments reported in this paper.

Appendix L Constructive Results: AnchorTune and B-ICS

This section tests two responses to the observed dissociation. Section L.1 examines whether the behaviour-anchored target $\mathcal{L}_{\mathrm{B\text{-}ICS}}$ attenuates Goodhart saturation; Section L.2 reports the AnchorTune $\lambda$ -sweep and multi-seed final; Section L.3 compares against two competitive baselines on the (MMLU, ICL-GAP) Pareto frontier; and Section L.4 ablates the anchor target.

L.1 Behaviour-Gated Sweep

We rerun the optimisation-pressure experiment with the same compute budget, the same probe ICL set, and the same hyperparameters as F_ICS-Max_v2, but replace $\mathcal{L}_{\mathrm{func\text{-}icl}}=-\mathrm{ICS}$ with $\mathcal{L}_{\mathrm{B\text{-}ICS}}=-\mathrm{B\text{-}ICS}_{\!\beta}$ at $\beta=10,\delta=0.05$ . By Proposition 3, $\mathcal{L}_{\mathrm{B\text{-}ICS}}$ down-weights the ICS-only gradient on the disjoint-support ridge, but the factor is still substantial at our mild setting $\beta\delta=0.5$ , and the smooth surrogate $\widetilde{\mathrm{ICL\text{-}GAP}}(\theta)$ of (21) can rise transiently above $\delta$ during training even when the discrete $\mathrm{ICL\text{-}GAP}$ does not.

The empirical trajectory clarifies the practical strength of the gating effect. Table 5 reports the step- $5{,}000$ values. F_BICS ends at $\mathrm{ICS}=1.397$ , only $0.016$ below the stress-test F_ICS-Max value of $1.413$ and within $1.2\%$ of the $\sqrt{2}$ ceiling. MMLU accuracy drops to $0.283$ , essentially the same degradation as F_ICS-Max. What does differentiate the two arms is the ICL-GAP: F_BICS finishes at $+0.020$ vs. F_ICS-Max’s $-0.010$ , and tracking its trajectory shows the soft gap exceeds $\delta=0.05$ on intervals where the behavioural sentinel intermittently activates, before the divergence term overwhelms it. This indicates that B-ICS as a training target requires a more aggressive gating schedule (larger $\beta$ , larger $\delta$ , or a hard $\mathbf{1}\{\cdot\}$ gate with a straight-through estimator) than the smooth $\beta=10$ we chose for differentiability; the observed $+0.030$ behavioural advantage of F_BICS over F_ICS-Max is consistent with the attenuation predicted by Proposition 3 under that mild gating.

When evaluated on a model that was not trained against it, B-ICS retains the intended behavioural guard. It multiplies ICS by a binary indicator that the behavioural sentinel passes and returns near-zero for F_ICS-Max (which has $\mathrm{ICL\text{-}GAP}=-0.010<\delta$ ) and near-pretrained for F_anchor at $\lambda^{\star}$ . The diagnostic version therefore keeps the intended behavioural guard, while the training-target version requires stronger gating than we use here.

Table 5: F_BICS at

\beta=10,\delta=0.05

attenuates the ICS-vs.-behaviour dissociation but does not eliminate it. The arm still reaches

\mathrm{ICS}=1.397

(

98.8\%

of the

\sqrt{2}

ceiling), but achieves a

+0.030

ICL-GAP improvement over F_ICS-Max. We attribute the residual saturation to the softness of the gate; a hard

\mathbf{1}\{\cdot\}

gate with a straight-through estimator is a candidate for further evaluation.

Arm	ICS $(\to\sqrt{2})$	ICL-GAP	MMLU	ECE
Pretrained	0.516	—	0.371	—
F_ICS-Max (stress test)	1.413	$-0.010$	0.279	0.231
F_BICS (this work)	1.397	$+0.020$	0.283	0.388

L.2 AnchorTune Sweep

For AnchorTune we run a four-point $\lambda$ -sweep $\lambda\in\{0.01,0.05,0.20,1.00\}$ at seed $42$ , then a three-seed final at the chosen $\lambda^{\star}$ . Table 6 reports the final-step values; Figure 6 plots the per-step trajectories of the four core quantities for F_anchor at $\lambda^{\star}$ alongside F_None and F_ICS-Max.

The empirical claim is that F_anchor at $\lambda^{\star}$ closes the MMLU damage that F_ICS-Max inflicts: from $0.279$ (chance $+0.03$ ) back to within seed-noise of the pretrained baseline of $0.371$ , while keeping ICS indistinguishable from pretrained. Concretely, across three seeds at $\lambda^{\star}=0.05$ , MMLU returns $0.370\pm 0.058$ (mean $\pm$ std), ICS returns $0.500\pm 0.001$ , and ICL-GAP returns $-0.012\pm 0.017$ — all within seed-variation of the pretrained model. The MMLU recovery accounts for $0.091$ of the $0.092$ accuracy drop that F_ICS-Max produced. Of the four $\lambda$ values, $\lambda=0.05$ is the empirical optimum (MMLU $0.367$ ); both $\lambda=0.20$ and $\lambda=1.00$ converge to a slightly lower MMLU plateau of $0.350$ , suggesting that the anchor pulls the trainee too tightly toward $\theta_{0}$ once $\lambda$ exceeds $\approx 0.1$ , and $\lambda=0.01$ is too weak to prevent the small MMLU drift seen for the unregularised arm.

Table 6: AnchorTune

\lambda

-sweep at seed 42 and three-seed final at

\lambda^{\star}=0.05

. Pretrained ICS is 0.516 and pretrained MMLU is 0.371. At

\lambda^{\star}

, the three-seed AnchorTune result is MMLU

0.370\pm 0.058

Configuration	ICS $(\to 0.516)$	ICL-GAP	MMLU $(\to 0.371)$	ECE
Pretrained	0.516	—	0.371	—
F_None (controlled)	0.500	$\approx 0$	0.375	0.407
F_ICS-Max (stress test)	1.413	$-0.010$	0.279	0.231
F_anchor, $\lambda=0.01$	0.501	$-0.040$	0.338	0.472
F_anchor, $\lambda=0.05$	0.500	$-0.010$	0.367	0.375
F_anchor, $\lambda=0.20$	0.500	$-0.020$	0.350	0.374
F_anchor, $\lambda=1.00$	0.501	$-0.015$	0.350	0.391
F_anchor, $\lambda^{\star}{=}0.05$ , seed 42	0.500	$-0.010$	0.367	0.375
F_anchor, $\lambda^{\star}{=}0.05$ , seed 43	0.501	$+0.005$	0.429	0.384
F_anchor, $\lambda^{\star}{=}0.05$ , seed 44	0.500	$-0.030$	0.313	0.387
F_anchor, $\lambda^{\star}$ , mean $\pm$ std	0.500 $\pm$ 0.001	$-0.012\pm 0.017$	0.370 $\pm$ 0.058	0.382 $\pm$ 0.006

L.3 Anchored-Baseline Comparison

We compare F_anchor with two capability-preservation baselines, all evaluated with the same training and evaluation protocol:

•

LogitAnchor: a logit-level KL-to-base regulariser of the form widely used in RLHF [Ouyang et al., 2022], $\mathcal{L}_{\mathrm{KL\text{-}logits}}(\theta;\theta_{0},\mathcal{S}_{\mathrm{anchor}})=\mathbb{E}_{\mathcal{S}_{\mathrm{anchor}}}[\KL(p_{\theta}(\cdot\mid\mathcal{D}_{1})\,\|\,p_{\theta_{0}}(\cdot\mid\mathcal{D}_{1}))]$ . This is the closest existing analogue of AnchorTune, but it anchors predictions rather than attention rows.
•

WeightAnchor: an EWC-style [Kirkpatrick et al., 2017] weight anchor $\mathcal{L}_{\ell_{2}\text{-}\theta_{0}}(\theta;\theta_{0})=\|\theta-\theta_{0}\|_{2}^{2}$ , with $\lambda=10^{-7}$ .

Table 7 reports the step- $5{,}000$ values.

The empirical pattern is more nuanced than “AnchorTune dominates the baselines.” In this single-seed comparison, all three anchored arms (F_anchor, LogitAnchor, WeightAnchor) recover MMLU to within seed noise of pretrained ( $0.367$ , $0.383$ , and $0.363$ respectively at seed $42$ , vs. pretrained $0.371$ and F_None’s $0.375$ ), and all three keep ICS at the pretrained $0.500\pm 0.005$ range. The three arms sit on the same Pareto front; what distinguishes them is what they anchor and therefore what they expose. LogitAnchor ties the trainee’s predictions to $\theta_{0}$ , which maintains task accuracy but masks the internal attention dynamics; WeightAnchor ties the trainee’s full weight vector to $\theta_{0}$ , which maintains both but at the cost of any meaningful fine-tuning signal at the $\lambda=10^{-7}$ we tested; F_anchor ties the trainee’s mid-band attention rows to $\theta_{0}$ , which maintains the diagnostic interpretability of ICS as a measure of attention-level ICL drift. In this setup, the three objectives are therefore complementary: AnchorTune’s contribution is not primarily a larger MMLU number but an objective whose anchor signal and diagnostic signal agree. This alignment allows ICS to be interpreted relative to the pretrained attention behaviour rather than a logit-constrained output alone. The contrast with F_ICS-Max is the main result: the divergence-based regularisers we test (ICS and smooth B-ICS as targets) move toward the high-ICS, low-MMLU region, while the anchored regularisers we test (logit, weight, attention) maintain MMLU. This comparison identifies a structural distinction between divergence and anchor losses, not a critique of attention-level signals per se.

Table 7: AnchorTune compared with two capability-preservation baselines and the F_ICS-Max regulariser. AnchorTune values are the seed-42 result for direct comparison with the single-seed baselines; Table 6 reports its three-seed mean.

Arm	ICS	ICL-GAP	MMLU	ECE
Pretrained	0.516	—	0.371	—
F_None	0.500	$\approx 0$	0.375	0.407
F_ICS-Max	1.413	$-0.010$	0.279	0.231
LogitAnchor (KL-to-logits)	0.494	$-0.020$	0.383	0.373
WeightAnchor (weight $\ell_{2}$ , $\lambda{=}10^{-7}$ )	0.500	$-0.010$	0.363	0.419
F_anchor (ours, $\lambda^{\star}{=}0.05$ )	0.500	$-0.010$	0.367	0.375

L.4 Anchor-Target Ablation

To isolate the active ingredient of AnchorTune, we vary which part of the model is anchored while keeping $\lambda$ and $\mathcal{S}_{\mathrm{anchor}}$ fixed:

•

F_anchor_last4: anchor attention rows only on the last $4$ layers, not on the mid-band $\mathcal{L}=\{8,\dots,16\}$ .
•

F_anchor_lmhead: anchor only the LM head weights via an $\ell_{2}$ penalty toward $\theta_{0}$ ; do not anchor any attention rows.

The hypothesis is that mid-band attention-row anchoring is what carries the MMLU-preservation effect. If F_anchor_last4 and F_anchor_lmhead both drop MMLU back toward F_None levels while F_anchor does not, the attribution to the mid-band attention rows is established.

The results in Table 8 weaken the original mid-band attribution hypothesis. Restricting the anchor to the last 4 layers (F_anchor_last4) gives $\mathrm{MMLU}=0.358$ vs. full-layer F_anchor’s $0.367$ , only a $0.009$ -point reduction — so the mid-band attention rows are sufficient but not necessary for MMLU preservation; the last 4 layers alone come within seed noise of the full anchor. More strikingly, the LM-head-only $\ell_{2}$ variant (F_anchor_lmhead, no attention anchoring at all) achieves $\mathrm{MMLU}=0.375$ , matching the pretrained reference of $0.371$ to within seed noise and exceeding seed-42 F_anchor. This means keeping $\mathrm{lm\_head}$ close to $\theta_{0}$ in weight space is by itself enough to recover MMLU under our fine-tuning configuration — attention-row anchoring is one route to MMLU preservation, but not the only one. These results indicate that the mechanism behind F_anchor’s MMLU recovery is partially shared with WeightAnchor (Table 7) and F_anchor_lmhead: proximity to $\theta_{0}$ in any structural component (weights, lm_head, attention rows) is sufficient. AnchorTune’s residual advantage over these baselines is not absolute MMLU but the coincidence of its anchor target with the diagnostic ICL signal, as discussed in Section L.3.

Table 8: Anchor-target ablation. All three variants keep ICS within

\pm 0.020

of pretrained while maintaining MMLU; the lm_head-only

\ell_{2}

variant without an attention anchor matches pretrained MMLU, weakening the claim that mid-band attention rows are uniquely necessary for MMLU recovery.

Anchor target	ICS	ICL-GAP	MMLU	ECE
Pretrained	0.516	—	0.371	—
None (= F_None, controlled reference)	0.500	$\approx 0$	0.375	0.407
Mid-band attn rows (F_anchor)	0.500	$-0.010$	0.367	0.375
Last 4 layers attn rows (F_anchor_last4)	0.510	$-0.015$	0.358	0.377
LM head $\ell_{2}$ only (F_anchor_lmhead)	0.494	$-0.020$	0.375	0.378

L.5 Summary of the constructive part

The constructive experiments support three claims. (i) The Goodhart channel is not an artefact of using attention as a target; it is an artefact of the divergence-on-disjoint-supports geometry. B-ICS as a diagnostic inherits the safety motivation of Proposition 3; B-ICS as a training target only partially attenuates the channel under the smooth gating we chose ( $\beta=10,\delta=0.05$ ), and reaches $\mathrm{ICS}=1.397$ at convergence (vs. $1.413$ for F_ICS-Max). A hard gate or a larger $\beta$ remains to be tested. (ii) Anchoring the per-head attention rows to the pretrained model on a fixed $\mathcal{D}_{1}$ probe maintains MMLU under exactly the same fine-tuning configuration where F_ICS-Max produces proxy saturation and MMLU damage: three-seed MMLU at $\lambda^{\star}=0.05$ is $0.370\pm 0.058$ , returning $99\%$ of the $\textsc{F\_ICS-Max}\rightarrow\textsc{F\_None}$ MMLU gap. (iii) The Pareto contrast in our tested arms is structural: anchored regularisers (attention rows in F_anchor, logits in LogitAnchor, weights in WeightAnchor) maintain MMLU, whereas the divergence targets we test do not. AnchorTune’s specific contribution among anchored regularisers is that its anchor signal coincides with the diagnostic ICL signal, so that the same probe can be read out at training and at evaluation without a Goodhart concern.

Attention Sensitivity Is Not Enough: Dissociating Attention-Level and Behavioural In-Context Learning under Fine-Tuning