Attention Limited Reward Learning

Let $x\in\mathcal{X}$ denote a context, prompt, market state, or initial state, and let $\mathcal{Y}(x)$ be the feasible candidate set. A comparison query is a triple

and the observed label is $c\in\{0,1\}$, where $c=1$ records a choice of $y^{1}$ over $y^{0}$. Queries are drawn from a distribution $\rho$ determined by the data-collection policy.

The target is a deliberative reward $R^{*}(x,y)\in\mathbb{R}$, interpreted as the evaluation the principal wants the human to apply after processing all reward-relevant information under the intended criterion. Write

for the deliberative difference in query $z$. A standard reward learner chooses a score function $r\in\mathcal{R}$ by logistic maximum likelihood,

under the maintained hypothesis that $\mathbb{P}[c=1\mid z]=\sigma(d_{r}(z))$ for some reward aligned with $R^{*}$. The paper studies what comparison data reveal when human labels are instead attention-filtered measurements of deliberative reward.

Fix a query $z$ and let $\omega\in\Omega_{z}$ denote the evidence relevant to the comparison: facts in the context, latent consequences of the two candidates, safety implications, or other payoff-relevant information. For the rational-inattention benchmark in this subsection, assume $\Omega_{z}$ is finite and the prior $\mu_{z}$ has full support. A comparator chooses an attention-decision rule $\pi_{z}(a\mid\omega)$ over actions $a\in\{0,1\}$, where action $a=1$ means choosing $y^{1}$ and action $a=0$ means choosing $y^{0}$. The marginal action probability induced by $\pi_{z}$ is

Let $u_{z}(a,\omega)$ be the comparator’s payoff, encoding the evaluation criterion the principal intends. The comparator solves the Shannon rational-inattention problem

where $\kappa_{z}>0$ is the marginal cost of information and

The next lemma is the standard generalized-logit form of the Shannon solution. It is included to pin down where the attention multiplier and the endogenous marginal action tendency enter.

Equation (7) is a conditional first-order characterization. The payoff difference is multiplied by the query-specific inverse information cost $\beta_{z}$, while the endogenous marginal action tendency $\alpha_{z}$ enters additively. It does not by itself pin down the ex-ante marginal label probability obtained by integrating over evidence states, and when the payoff difference is deterministic and nonzero the optimum is a boundary case to which the interior logit formula does not apply. The rest of the paper therefore takes the following equation as a reduced-form marginal measurement channel, motivated by the conditional rational-inattention first-order condition rather than derived from it. The additive term $\eta_{z}$ in the reduced form should be read as standing in for the endogenous action tendency $\alpha_{z}$ together with genuinely exogenous order, format, and salience effects; Remark 1 gives one environment in which the channel holds exactly.

All graph-theoretic and information-theoretic results below are statements about the reduced-form channel in Definition 1. If $\beta_{z}=0$, the comparison label is independent of the deliberative reward difference after conditioning on the default term. Such a query may still produce random-looking labels, but it carries no information about the sign or magnitude of $\Delta_{z}^{*}$ through the reward channel.

With the channel in hand, the paper’s question can be stated. A learner observes the choice probabilities $q_{z}$, while the object of interest is $\Delta_{z}^{*}$; what does the former reveal about the latter? Section 3 answers for the standard Bradley–Terry reward learner, at the level of representation and identification. Section 4 then drops the estimator from the picture and answers at the level of information, bounding what any procedure could distinguish with any amount of data.

The first question is whether attention-filtered log odds can be represented by any scalar reward at all. This is a purely algebraic question about the structure of the log-odds field, and it has a complete answer. Fix one context and a finite candidate set $V=\{1,\ldots,m\}$. Let $G=(V,E)$ be the undirected comparison graph. Choose an arbitrary orientation for each edge $e=(i,j)\in E$ and let $q_{e}=q_{ij}$ be the probability that $i$ is chosen over $j$ in that oriented comparison. Define $\ell_{e}=\operatorname{logit}(q_{e})$, and use the antisymmetric extension $\ell_{ji}=-\ell_{ij}$ when an edge is traversed against its chosen orientation. Let $B\in\mathbb{R}^{E\times V}$ be the signed incidence matrix with $(Br)_{e}=r_{i}-r_{j}$ for $e=(i,j)$. A Bradley–Terry representation of the channel is exactly a potential representation $\ell=Br$. The only obstruction is circulation around cycles.

The proposition gives an immediate diagnostic for attention-filtered data. Homogeneous attention is harmless because it merely rescales the potential. Potential defaults are representable but still misaligned, because the fitted reward absorbs the default. Heterogeneous edge-specific attention generally creates nonzero cycle sums.

The second case illustrates a subtle failure mode. A default with potential structure, such as a systematic preference for familiar formats or concise answers, is perfectly representable by Bradley–Terry scores. It is nevertheless not deliberative reward; it reverses pairs whenever the default difference dominates the attended reward difference.

A failed cycle criterion does not stop anyone from fitting Bradley–Terry. The population likelihood remains strictly concave, so the fit converges regardless, and the question is what it converges to. The next proposition identifies the limit near the low-information regime as the weighted least-squares projection of the human log-odds field onto the space of potential fields. The fitted reward is therefore not a noisy copy of deliberative value but a graph-dependent compression of it, shaped by the comparison graph and the sampling policy as much as by human values.

Let $W=\operatorname{diag}(\rho_{e})$ collect the edge sampling probabilities, with $\rho_{e}>0$ and $\sum_{e}\rho_{e}=1$. Normalize scores by $\mathbf{1}^{\top}r=0$. For an oriented edge vector $\ell$, define the population objective

The projection in Equation (11) discards all cyclic components of the human log odds. The following example shows that the induced error is not merely cardinal; it can reverse the learned ranking.

Example 1 might suggest that the problem lies with the Bradley–Terry functional form, and that a richer model class, a Bayesian prior over rewards, or simply more data would recover the deliberative ranking. The deeper obstacle is that comparison probabilities alone cannot disentangle reward, attention, and defaults, so any method that consumes only passive labels inherits the same ambiguity. The next proposition makes this point in finite-dimensional form.

This proposition is the formal reason that a Bayesian prior, a larger neural reward model, or more passive samples cannot by itself recover deliberative reward. Without restrictions that distinguish reward from attention and defaults, the likelihood is flat along observationally equivalent explanations.

Non-identification says the likelihood is flat; it does not yet say why the flatness arises, or how much data it costs even where the parameters are partially informative. Both questions have exact answers once the label is treated as a communication channel and its information content is accounted for directly. That accounting is the subject of the next section.

A nearly even split of labels is often treated as the most informative region of a logistic model because the Bernoulli variance is largest near one half. That intuition is incomplete. If the split is even because attention is near zero, the label may be maximally random while revealing almost nothing about reward.

Consider a single pair and let $\Delta$ be an unknown deliberative reward difference with a finite prior distribution $\Pi$. Conditional on $\Delta$, the observed label satisfies

Let $\mathsf{H}(C)$ denote the entropy of the marginal label and $\mathrm{I}(\Delta;C)$ the mutual information between the reward gap and the label, measured in nats.

Thus a $50$–$50$ outcome has two distinct interpretations. It may mean the reward gap is close to zero, or it may mean the channel is low-transmission because $\beta$ is small. The entropy of the label alone cannot distinguish these cases.

Proposition 4 is a statement about a prior over the gap. The same distinction appears in local, estimation-theoretic form through Fisher information, and in that form it also exposes the mechanism behind the non-identification of Proposition 3. In the standard Bradley–Terry model, one label contains the most local information about the reward difference near a $50$–$50$ split. Under the attention-scaled channel, that maximum is multiplied by $\beta^{2}$.

The first claim gives a sample-complexity penalty, since low attention reduces per-label information about reward quadratically. The rank-one claims give a local version of Proposition 3. The likelihood identifies the log-odds index $\eta+\beta\Delta$, not the reward gap, the attention multiplier, and the default separately.

Fisher information measures local difficulty; the operational question is global. How many labels does it take to answer the coarsest question the data could settle, namely which of the two candidates is better? The next result shows that ordinal recovery, too, is bought only with attended information. To isolate the effect, consider the clean channel with no default and known attention,

The task is to decide whether $\Delta=\delta$ or $\Delta=-\delta$ for a fixed $\delta>0$.

The lower bound depends on the product $\beta\delta$. Bradley–Terry interprets a small product as a small reward gap. The attention-scaled model shows that it may instead be a large reward gap passing through a low-attention channel.

The same logic extends from one pair to many reward hypotheses. Let $\Theta$ be a finite set of possible reward functions and suppose the learner observes labels $C_{1},\ldots,C_{n}$, possibly under adaptively chosen queries. Let $H_{t-1}$ denote the history before label $t$, including previous labels, adaptively chosen queries, and any external randomization used by the query rule. Let $P_{\theta,t}(\cdot\mid h_{t-1})$ denote the conditional distribution of label $t$ under reward hypothesis $\theta$ and history $h_{t-1}$.

In attention-scaled logistic channels, the constants $d_{t}$ are of order $\beta_{t}^{2}$ for small attended reward separations. The bound therefore says that reward recovery is limited by total attended information $\sum_{t}d_{t}$, not simply by the number of labels. Repeating low-attention comparisons can be much less valuable than collecting fewer high-attention comparisons.

The bounds so far concern single pairs and finite hypothesis classes. Returning to the comparison graph closes the loop with Section 3. The cycle obstruction, similarly, has an information-theoretic meaning, and the projection of Proposition 2 turns out to be the object that discards it. A scalar reward can represent only potential fields on the comparison graph. The part of the observed log-odds field orthogonal to all potentials is cyclic comparison information, present in the human comparisons but impossible to encode in any scalar reward. The decomposition below is the combinatorial Hodge decomposition of Jiang et al. [15], applied to the weighted log-odds field generated by the attention-scaled channel.

Let the weighted inner product on edge fields be

Let $\mathcal{P}=\operatorname{im}(B)$ be the potential subspace. Define

The coefficient $r^{H}$ in (27) is computed by the same operator as the pseudo-true target in (11). To first order, the attention-blind Bradley–Terry limit is the potential component of the human log odds. Proposition 7 therefore prices what Proposition 2 discards. The cycle obstruction is not merely a consistency condition. It quantifies how much comparison information is irreducibly non-reward-like, and the cyclic energy measures the second-order information loss of compressing the human comparison channel to its potential component.

The theory predicts that the log-odds field of real comparison data need not be a potential field, and Proposition 7 prices what any scalar score then discards. We test this signature on a canonical human preference dataset, $57{,}477$ pairwise battles between $64$ language models collected on Chatbot Arena [6].¹¹1Publicly available at https://huggingface.co/datasets/lmarena-ai/arena-human-preference-55k. Items are models, edges are model pairs, and labels are human votes, so the empirical vote frequencies provide exactly the weight matrix $W$ of Section 3. We drop ties ($31\%$ of battles), keep pairs with at least $100$ decisive votes, and restrict to the largest connected component, which leaves $m=32$ models, $74$ pairs, $15{,}001$ votes, and a cycle space of dimension $74-31=43$. Edge log odds use the Haldane–Anscombe estimate $\hat{q}_{e}=(w_{e}+\tfrac{1}{2})/(n_{e}+1)$.

Sampling noise alone creates cyclic energy, so significance is judged against a parametric bootstrap null. We fit Bradley–Terry by weighted maximum likelihood, regenerate binomial votes at the observed $n_{e}$ two thousand times, and recompute the cyclic energy share of each regenerated field. The observed share is $4.2\%$, against a null mean of $2.5\%$ and a null $95$th percentile of $3.5\%$, for a bootstrap $p$-value of $0.008$ (Figure 2); the likelihood-ratio statistic gives the same verdict ($70.6$ on $43$ degrees of freedom, bootstrap $p=0.007$). The hypothesis that some scalar score generates these win rates is rejected at the one percent level.

The misfit is small but priced accurately. The best scalar fit loses $0.0034$ bits per comparison in weighted KL, and the prediction $\tfrac{1}{8}\|\hat{\ell}^{\mathrm{cyc}}\|_{W}^{2}$ of Proposition 7 is $0.0038$ bits, a ratio of $0.89$, even though the largest observed log odds is near $1.8$ and the proposition’s small-$\varepsilon$ hypothesis is far from satisfied. Part of the raw loss reflects sampling noise; the deviance in excess of its degrees of freedom gives a rough noise-corrected estimate of the population-level misfit near $0.0013$ bits per comparison. Individual triangle circulations reach at most $z\approx 3.0$, which is unremarkable given the number of triangles examined, so the evidence lies in the global excess rather than in any single cycle; we note, without attaching significance, that the top-ranked triangles involve closely related model versions such as claude-2.1, gpt-4-0314, and gpt-4-1106-preview.

The excess cyclic energy survives vote thresholds of $50$ and $200$ per pair (bootstrap $p=0.046$ and $0.008$). Counting ties as half wins shrinks every log odds toward zero and weakens the excess, to $p=0.06$ on the same graph and to insignificance when the threshold also admits sparsely compared pairs, consistent with dilution by a large mass of uninformative labels. Two caveats bound the interpretation. The votes aggregate over prompts and annotators, so cyclic energy can also arise from aggregation rather than per-query attention, and the analysis conditions on the pairs the platform chose to sample. What the case study establishes is a statistical rejection of every scalar score for this comparison data, at a misfit priced well by the one-eighth law; the finding is consistent with, though not unique evidence for, the attention mechanism. A self-contained script reproduces all numbers.

The second footprint concerns the channel itself, and it needs data in which the deliberative gap and the evaluator’s information acquisition are both measured. The perceptual comparison dataset of Tavares et al. [33] provides both.²²2Distributed with the aDDM toolbox, publicly available at https://github.com/goptavares/aDDM-Toolbox. Twenty-five participants made $31{,}854$ binary comparisons between two rotated bars, choosing the one closer to a target orientation, with angular distances set by the experimenter; the quality gap $\Delta\in\{-3,\ldots,3\}$ is therefore ground truth rather than a rating proxy, and every trial records the choice, the response time, and gaze fixations. Pooled choices follow a logistic in $\Delta$ with slope $1.14$, and per-participant slopes range from $0.45$ to $2.70$, a factor of six (Figure 3, left). The same objective gap passes through channels of very different gain across evaluators. The reduced form absorbs attention, effort, and acuity alike into $\beta$, so this is heterogeneous $\beta$ measured directly, whatever its source; and because preference datasets pool annotators over different pairs, evaluator-level heterogeneity of this size induces the edge-level heterogeneity that generates cyclic energy of the kind found in Section 5.1.

The headline is an information decomposition. Because the design is symmetric, a single label reveals which item is better but almost nothing about by how much; empirically the label carries $0.33$ bits about the sign of the gap and $0.0001$ bits about its magnitude. Response time carries $0.035$ bits per trial about the magnitude (permutation-corrected, $p<0.002$), while the label carries essentially none; mean response time falls from $2.2$ seconds on ties to $1.4$ seconds at the largest gap (Figure 3, right). Repeated labels do not close this gap, since they reveal only the confounded index $\beta\Delta$ of Proposition 5, while response time gives a reading from outside the label channel. The quantity an annotation protocol needs in order to separate a near-tie from a large-but-unresolved gap is absent from the label and present in the response time, which is exactly the measurement the theory recommends collecting.

Gaze is associated with an additive shift of the kind the default $\eta$ describes. At fixed $\Delta$, one second of relative dwell toward an item is associated with a $0.91$ increase in its choice log odds (standard error $0.03$), about as much as a full quality level, consistent with the gaze bias documented by Krajbich et al. [16]; the association is observational, and gaze may follow an emerging choice as well as shape it. Two further caveats apply. The task is perceptual rather than preferential, and response time is endogenous to difficulty, so these are the channel’s ingredients observed in a controlled comparison task rather than in RLHF annotation itself. Annotation platforms typically record decision latency and could release it; the analysis here is what that would enable.