The Answer Is Not the Argument

Humanity’s Last Exam [14] (HLE) contains 182 questions with a subject category of physics; including granular categories such as biophysics, atmospheric physics and quantum physics yields 228. Of these, 178 are text-only. We restricted the study to text-only items because not all monitor APIs accept images. To remove the possibility of a correct answer by guessing we included only exact-match questions, leaving 144. These were manually reviewed to eliminate items that did not require multi-step physical reasoning; factual-recall questions, such as those asking for the exponents on specific terms in little-known theories, were removed. This left 99 questions, of which 9 were used for an inter-rater reliability study and 90 for the main work. A further 11 questions were dropped from the 90 during labelling. Some were physically ill-posed or internally inconsistent. Three stipulated “pseudo-physics” scenarios in which the rules of an alternative universe were specified too loosely to determine a unique answer. Others admitted more than one defensible answer, for instance one question applying Basquin’s law for fatigue life requires a value for the maximum number of cycles, and although the reference answer assumes $10^{7}$ [7], $10^{6}$ [18] is an equally standard choice, so a model selecting the latter is marked incorrect for a reason that has nothing to do with its reasoning. Of the 99 questions reviewed closely, we found 16 where the reference answer supplied by HLE was either not physical or incorrect. These errors were reported to the organizers.

Three frontier models at the time of generation - GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro - each produced one solution per question, giving 237 traces over 79 questions. Each model was given the question and asked to return its working as a sequence of explicitly numbered steps with the final answer in a box. The prompts used to generate these traces are given in Appendix A.

Each trace received two independent labels: whether its final answer is correct, and the first numbered step containing a genuine error. Labelling these separately creates a critical set in which the final answer is correct but the trace contains a genuine error. A monitor that checks only the final answer classifies every one of these as successful. The errors studied here are genuine model mistakes rather than faults inserted by the authors.

We define the first error step as the earliest numbered step that commits to a false claim about the problem, about the relevant physics or mathematics, or about the trace’s own previous reasoning. An incorrect equation or numerical assertion is an error. Exploratory search is not - if a trace proposes a candidate approach, tests it, finds it inadequate and proceeds without using it, that is exploration. A false claim made while exploring is an error, so applying conservation of energy correctly and then discarding the approach is not an error, whereas applying it incorrectly is. An unsupported but plausible claim, or an omitted proof, is not labelled false merely because its justification is incomplete; if a trace elides a derivation and the following step is wrong, that is an error. We record the first false step rather than the step most responsible for the final answer. This avoids a second and less reliable judgement about downstream causal importance and directly identifies the point at which the written argument first becomes unsound. A full description of the annotation rules is given in Appendix B.

Deciding whether a trace reaches the correct answer is straightforward. Locating the first error is not as it requires subject knowledge and a close reading of the whole trace. We therefore used a two-arm review process. In the first arm, physicists with relevant subject expertise annotated 174 of the 237 traces to create a Human standard. Resource constraints prevented complete coverage. In the second, independent arm, Claude Opus 4.8 annotated every trace, GPT-5.5 responded adversarially, and Claude replied [10]; separately GPT-5.5 produced its own independent annotation, and a fifth round presented Claude with the full chain and the independent GPT annotation for a final verdict. This produced an AI standard, neither of the two debate models was itself evaluated as a monitor.

The final standard was produced by the lead author, combining both arms and resolving disagreements with the source of each proposed label masked. Claude Opus 5 and GPT-5.6 were consulted during this phase. Fable 5 was inadvertently consulted during one adjudication session on approximately three disputed traces because it remained selected from an earlier task. Fable 5 was subsequently included as a monitor, but excluding it does not alter the central conclusion (Sec. IV.2). We also include Kimi K3 [3], a frontier model held out entirely until after the remaining data had been collected and analysed.

The two arms agreed on final-answer correctness for 95.6% of jointly covered traces and on the presence of an error for 79.3%. Where both named a step, they agreed exactly 51.4% of the time, against a uniform chance rate of 15.1%. This disagreement is why localization targets were adjudicated explicitly rather than inherited from either arm. Of the 237 traces, 42 carry a label set by author adjudication; in the critical set the figure is 16 of 24. The resulting standard contains 237 traces: 130 with a correct answer and no error, 83 with a wrong answer, and 24 with a correct answer reached through flawed reasoning.

The annotation scheme was fixed through a pilot inter-rater reliability study. 9 questions were held out for pilot development. The lead author annotated the 9 traces generated for 3 of these questions as worked examples and recorded a training video; 6 annotators then independently scored the 18 traces generated for the remaining 6 questions. These annotations used a richer instrument than the one eventually adopted: final-answer correctness, first error step, a coherence judgement, and a categorical error type. Pairwise agreement was 0.930 on final-answer correctness and 0.859 on whether a trace contained an error, but only 0.434 on the exact step, rising to 0.643 within one step; 15 of the 18 traces were unanimous on correctness. The coherence field took its modal value on 92.2% of 103 judgements and carried almost no information, and the error-type field collapsed in practice to two of its categories on 74.5% of 98 judgements. Both were dropped, and the main study used only the two axes that reached usable agreement. The pilot also established the treatment of exploratory reasoning described above, which was the largest single source of disagreement in the discussion. Two further annotators were recruited but did not complete the pilot and are not included in these figures. For the main-study annotation pool, annotators returned 204 complete annotations covering 198 distinct traces, of which 24 fell on questions later excluded, leaving 174 of the 237 traces (73.4%) with a human label.

Given the rapid advancement of AI systems, we selected 8 monitor models spanning a wide range of capabilities, from comparatively weak systems such as Qwen3-32B to frontier systems such as Fable 5 (Table 1). Ultimately, for questions of scalable oversight [5], we wished to investigate whether a similar performance pattern was observed over the range of strengths of the monitors. Because GPT-5.5 and Claude Opus 4.8 were used in the AI-debate arm of the reference standard, we excluded them from the monitor pool. Gemini 3.1 Pro was retained because it was not used in that arm. Four conditions form an information ladder, each seeing everything the previous one saw:

• •

  BLIND: the problem and the complete numbered trace, with no reference answer.

• •

  HINT: BLIND condition with the reference answer in addition, framed as an unverified candidate the monitor is told not to trust.

• •

  CERT: BLIND condition with the reference answer in addition, framed as known to be correct, with an explicit warning that a correct final answer does not imply sound reasoning.

• •

  REVISE: the monitor’s own BLIND verdict is replayed to it, the certified answer is then revealed, and it may revise.

Two further conditions sit outside the ladder. RCTRL is identical to REVISE except that the answer is withheld and the monitor is simply invited to reconsider; it isolates the effect of certification from the effect of being asked twice. STEP shows the monitor one step at a time, with no view of what follows, and takes the trace verdict to be the union of the per-step decisions. STEP is a compound intervention rather than a further rung on the information ladder - it hides future context, segments the trace, and makes one decision per step, so its false-positive rate accumulates with trace length. We report it separately throughout.

In the whole-trace conditions, monitors returned has_error, first_error_step and error_confidence as JSON under the same error definition used for the reference standard. We call a trace flagged when the monitor returns has_error = 1, meaning that it judges the trace to contain at least one genuine error; has_error = 0 is a pass. For a flagged trace, first_error_step gives the predicted location. A trace can therefore be correctly flagged while the first error is localized incorrectly. Under STEP, each prefix instead received a step_has_error verdict; any positive verdict flagged the trace, and the earliest such verdict became its predicted localization. All monitors were run at temperature 0 with the exception of Kimi K3, whose API fixes temperature at 1.0 and for which thinking was set to low for cost reasons; its results are therefore a single draw rather than a near-deterministic one, and are not a capability claim about that model. Cell-restricted recall comparisons were made on a separate common subset for each monitor, retaining only traces with a valid parsed verdict under BLIND, HINT, CERT, and REVISE; sample sizes are reported in Appendix C. The prompts are given in Appendix D.

We report balanced accuracy as the primary detection metric. This is the arithmetic mean of sensitivity, the fraction of traces containing a genuine error that are flagged, and specificity, the fraction of clean traces that are not flagged. It therefore weights error detection and false-alarm avoidance equally, with chance performance at 0.5. We define exact localization $L_{0}$ as the fraction of all 107 error traces for which the monitor both flagged the trace and named the correct step. Missing the error and flagging the wrong step both count as failures. $L_{0}$ is our primary outcome because identifying the first false step is what the monitors were instructed to do. We also decompose every error trace into four outcomes: a flag placed before the true first error (early), on it (exact), after it (late), or absent (missed). Cell-restricted recall is the fraction of genuinely erroneous traces flagged, irrespective of whether the predicted error step is correct, reported separately for wrong-answer and critical traces. For each monitor, cell-restricted recall comparisons across BLIND, HINT, CERT, and REVISE use the subset of traces with valid verdicts in all four conditions, keeping the denominator fixed. Both cells contain only positive examples and therefore support recall but not specificity. Specificity requires clean, negative traces and is reported only for the full dataset.

Each question contributes three traces, one per generator, which share a problem statement, a reference answer and a difficulty. Resampling traces independently would treat those three as independent observations and understate the resulting intervals. We therefore report a cluster bootstrap: the 79 questions are resampled with replacement, all traces of a drawn question are carried with it, and every reported statistic is recomputed on each of 2000 resamples. Intervals are the 2.5th and 97.5th percentiles of the resulting distribution. Only questions are resampled. The 8 monitors are the panel we selected rather than a sample from a population of possible monitors, so these intervals describe sampling error over problems and carry no implication about how a further monitor would behave.

Consistency across monitors is reported separately using an exact sign test on each monitor’s difference-in-differences. For a given pair of conditions, this is the change in recall on wrong-answer traces minus the corresponding change on critical traces. The critical set receives the same intervention, but its conclusion carries no information about the error, so effects common to both cells cancel. A positive value means that answer access helps more where the conclusion is diagnostic. The sign test summarizes directional consistency within the selected panel. Because the monitors are not statistically independent draws from a population of models, its $p$-value should not be interpreted as generalizing to unseen monitors.

Mean balanced accuracy rises monotonically along the information ladder, from 0.637 (95% CI [0.605, 0.666]) under BLIND to 0.712 under HINT and 0.796 ([0.760, 0.832]) under CERT (FIG. 2(a)). The rise is driven by both components. Mean specificity rises from 0.655 to 0.757, and mean sensitivity from 0.619 to 0.836. Answer access therefore makes monitors flag more accurately in both directions rather than simply flagging more. Exact localization rises far less, from 0.261 ([0.212, 0.312]) under BLIND to 0.379 ([0.316, 0.436]) under CERT (FIG. 2(b)). The gap between the two panels is the central observation of this paper - the metric that reports whether a monitor raised an alarm improves by 0.637 $\rightarrow$ 0.796, while the metric that reports whether it found the error improves by 0.261 $\rightarrow$ 0.379. The distinction is clearest at CERT, where monitors flag a mean of 95.1% of traces whose final answer is wrong while identifying the erroneous step in 37.9% of error traces. Table 1 illustrates why recall alone is inadequate. Note deepseek-v4flash flags 94.5% of all traces and reaches a sensitivity of 0.972 at a specificity of 0.078; its AUC of 0.473 is below chance. Any evaluation ranking monitors by recall alone would place it first.

Monitor identity matters more than condition: several stronger monitors under BLIND outperform weaker monitors given certification. Within-condition spreads in balanced accuracy range from 0.272 under CERT to 0.358 under HINT, compared with the mean movement from 0.637 to 0.796 across the information ladder (FIG. 2(a)). Two monitors share a model family with a generator: gemini-3.1-pro reviewed its own traces, and fable-5 those of claude-opus-4.7. Recall on error traces from the monitor’s own family differed from recall on other traces by $-0.081$ and $+0.089$ for gemini-3.1-pro under BLIND and CERT, and $+0.071$ and $-0.121$ for fable-5. No difference reached significance (Fisher exact, smallest $p=0.105$), and both monitors reverse sign between conditions, which is not the pattern a consistent self-preference would produce. With 32-55 traces per comparison the test would only detect a difference of roughly 0.2, so we report this as the absence of a large effect rather than evidence of none. Critical and wrong-answer traces have median lengths of 10.0 and 12.5 steps (Mann-Whitney $p=0.142$). At these sample sizes the test detects only large differences, so this is the absence of a substantial length imbalance rather than evidence of none; the recall difference between the cells is unlikely to be explained by length alone.

Splitting recall by cell shows what produces the aggregate rise (FIG. 3). On traces whose final answer is wrong, moving from BLIND to CERT raises mean recall by +0.299, and under CERT all eight monitors sit between 0.855 and 0.988. Every monitor moves most of the way to the ceiling - on average certification closes 75.9% of the distance between a monitor’s blind recall and 1. This value is the comparable statistic because the raw gain is bounded by where a monitor started (deepseek-v4flash begins at 0.988 and can gain at most 0.012, while qwen-2.5-72b begins at 0.361).

On the critical set, the same intervention changes mean recall by -0.083 (question-bootstrap 95% CI [-0.196, +0.030]; FIG. 3(b)). The interval includes zero, so this result provides no evidence that certification improves critical-trace recall but does not establish a reliable decline. Recall declines for six monitors, is unchanged for one, and increases slightly for one, despite a prompt stating explicitly that a correct final answer does not imply sound reasoning. The difference between the wrong-answer and critical-trace gains is +0.382 (95% CI [+0.256, +0.506]) and is positive for 8 of 8 monitors (exact two-sided sign test, $p=$ 0.0078). Where the final answer is wrong, comparing it against the reference answer establishes that an error exists somewhere without reading a line of the derivation. Where the answer is correct, that comparison returns nothing, and finding the error requires checking the working. In essence, the answer is not the argument.

Under CERT, qwen3-32b reaches 0.962 on the wrong-answer cell against 0.988 for fable-5, a spread of under 0.05 despite their blind balanced accuracies differing by more than 0.3. In this setting, the much smaller model therefore nearly matches the frontier monitor when the conclusion reveals the error, further suggesting that this cell is dominated by conclusion comparison rather than reasoning verification. HINT sits between BLIND and CERT for most monitors on the wrong-answer cell. BLIND $\rightarrow$ HINT measures the effect of adding an unverified answer and HINT $\rightarrow$ CERT the effect of changing its epistemic status, thus the pattern indicates that both the presence of an answer and its certification contribute.

To test whether certification merely changed monitors’ propensity to flag answer-matching traces, we restricted the analysis to traces whose final answer was correct, treating critical traces as positives and clean traces as negatives. Mean balanced accuracy within this subset changed only slightly, from 0.586 under BLIND to 0.597 under CERT (+0.011). This panel-average change was heterogeneous across monitors and arose because the increase in clean-trace specificity slightly exceeded the fall in critical-trace recall. Certification therefore shifted monitors towards accepting answer-matching traces, helping on clean traces while harming detection of flawed reasoning that reached the correct answer.

The four-way decomposition of every error trace (FIG. 4) shows that the improvement in $L_{0}$ from BLIND to CERT (0.261 $\rightarrow$ 0.379) is accompanied by a fall in missed traces (0.383 $\rightarrow$ 0.165) and a rise in flags placed after the true first error (0.209 $\rightarrow$ 0.347). Flags placed before the error fall slightly (0.147 $\rightarrow$ 0.110). Monitors given the answer therefore place their flags further downstream. This pattern is consistent with working backwards from the conclusion, although the present design does not establish that mechanism. It is in any case invisible to a metric that treats a flag anywhere in an erroneous trace as a success.

Adjacent steps are not always cleanly separable - during the construction of the ground truth there would often be two adjacent steps that were both justifiable as the first error - we therefore repeated the analysis counting a flag within one step of the ground truth label as correct. Loosening the criterion is not simply a more forgiving version of the same measurement, because it also makes a lucky guess more likely to succeed. Consider a three-step trace. A monitor that flags step 2 is exactly right only if the error is at step 2, a one-in-three proposition; under a one-step margin it is counted correct wherever the error lies. Widening the window widens the target, and the wider target is easier to hit whether or not the monitor is reading the derivation. To quantify this, we ask what a monitor would score if it placed its flag at random. In a trace of $n$ numbered steps, a uniformly placed flag lands on the true step with probability $1/n$, and within one step of it with probability $w/n$, where $w$ is the width of the window after clipping at the ends of the trace ($w=3$ in the interior, $2$ if the error is at the first or last step). Only a trace the monitor actually flagged can be located at all, so we sum these probabilities over the traces each monitor flagged and divide by all error traces, giving a chance rate on the same denominator as $L_{0}$ itself. We call this the uniform-placement null: it is what the measurement would return from a monitor that decides whether to flag exactly as it does, but then chooses where to point with no information at all.

FIG. 5(a) shows the gain in error location through allowing a within-one-step window for the average of all monitors and for the chance rate. Moving from exact to within-one raises the chance rate by +0.136 at BLIND and +0.185 at CERT, against observed gains of +0.111 and +0.139. The apparent improvement from relaxing the tolerance is therefore not evidence that monitors are nearly right. Correcting for chance removes the difference between the two criteria (FIG. 5(b)). The skill score $(L-L_{\mathrm{chance}})/(1-L_{\mathrm{chance}})$ expresses how far a monitor has closed the distance between random placement and perfect localization: it is $0$ for a monitor performing at chance and $1$ for one that is always right. Under BLIND it is 0.199 at exact and 0.201 within one step; under CERT, 0.306 and 0.321. The two tolerances agree, and both rise monotonically along the ladder, so every conclusion drawn from exact localization holds under the looser criterion. Note that the chance rate rises from left to right on FIG. 5(a) because it scales with how often a monitor flags. A trace the monitor never flags cannot be located by accident; thus, a monitor can raise its raw localization simply by flagging more.

Exact localization may also be depressed by ambiguity in where an error first becomes explicit. We therefore repeated the analysis on the 19 traces for which the human arm, AI-debate arm, and final ground truth all selected the same first-error step. Absolute $L_{0}$ is higher in every condition on this cleaner subset (Table 2), but the information ladder ordering remains. In particular, localization rises from 0.382 under BLIND to 0.632 under CERT, so reference-label ambiguity affects the absolute scores but does not produce the certification gain. Because agreement preferentially selects clearer errors, these values are a robustness check rather than a noise-corrected estimate for the full dataset.

REVISE reaches the highest exact localization of any condition, 0.418, at a balanced accuracy of 0.783, slightly below CERT because it flags more overall (mean sensitivity 0.861 against specificity 0.705). Committing to a blind verdict before seeing the answer therefore preserves the localization gain that certification brings. Models will often change a judgement simply because they have been asked again, so REVISE is reported net of RCTRL, in which the monitor is invited to reconsider without being shown the answer. On that basis, revealing the answer causes monitors to withdraw prior flags at +0.414 on clean traces, where withdrawal is correct, and +0.189 on critical traces, where it retracts a correct judgement (FIG. 6(a)).

A monitor treating the answer as evidence rather than as instruction should withdraw on clean traces and hold on critical ones, so we take the difference between the two as a discrimination gap. The largest observed gap is obtained by fable-5, although both of its component estimates have fewer than ten observations. Among monitors with at least ten observations in both cells, gemini-3.1-pro shows the clearest separation while qwen3-32b returns -0.020 and llama-3.3-70b -0.047, withdrawing at effectively the same rate whether or not the trace contains an error. Across the eight monitors, the gap is descriptively associated with blind balanced accuracy (Spearman $\rho=0.64$). deepseek-v4flash sits apart for a reason visible in the figure - its bars are short on both sides because it withdraws almost nothing at all (+0.160 and +0.043, against a mean summed net withdrawal of 0.66 across the two cells from the other seven monitors), and a monitor that does not revise cannot revise wrongly. FIG. 6(b) shows that, averaged across monitors, revealing the answer adds a flag to 93.8% of previously passed wrong-answer traces and 18.0% of previously passed critical traces, respectively. Every monitor with an interpretable denominator shows the same asymmetry. For deepseek-v4flash, only one trace remained in each cell after BLIND, so neither rate is informative; qwen3-32b’s critical-cell rate rests on five traces. Certification provides an effective consistency check on the conclusion, but is a poor substitute for checking reasoning that happens to end at the correct answer.

The four ladder conditions all present a completed trace. By contrast, a monitor used as a runtime safety mechanism may need to interrupt, defer, or escalate a trajectory before the model reaches its conclusion or takes an associated action. STEP approximates this online constraint by exposing the reasoning one prefix at a time. It is also the weakest condition. Mean balanced accuracy is 0.614 against 0.637 under BLIND, exact localization is 0.196 against 0.261, and its miss rate of 0.422 is the highest of any condition.

Across STEP, BLIND, and CERT performance is ordered by the availability and reliability of the conclusion. STEP provides no conclusion, BLIND provides the trace’s own uncertified conclusion, and CERT provides a trusted one. This ordering is consistent with monitors relying on the endpoint as an important diagnostic cue rather than independently verifying each part of the argument. It also cautions against treating the gains from answer access as gains that would necessarily be available to a safety monitor operating before an answer or action exists.

STEP does not isolate the effect of withholding the conclusion. It combines several changes by hiding future context, segmenting the trace, and requesting one decision at every step. Its trace-level verdict is the union of those decisions, so false-positive risk accumulates with trace length. STEP is therefore neither cost- nor information-matched to the whole-trace conditions. The observed gap should be read as evidence that online monitoring can be substantially harder than retrospective whole-trace evaluation suggests, rather than as a calibrated estimate of the penalty imposed by online deployment.

The resulting failure mode is directly relevant to weak-to-strong control [6]. A monitor that is weaker than the system it oversees must balance missed errors against unnecessary interventions, and repeated prefix-level decisions can make the latter prohibitive. Under STEP, specificity ranges from 0.178 for deepseek-v4flash to 0.967 for fable-5, with most of the accumulated false-positive cost falling on the weaker monitors. Deepseek-v4flash produces at least one flag on 85.7% of traces and therefore flags approximately 82% of clean traces, while locating the first error exactly in only 28% of erroneous traces. Without substantial calibration or aggregation, a monitor with that operating profile could not serve as a selective gate on a live process.

For each of the 24 traces in which the final answer was correct despite flawed reasoning, we coded the error according to its relationship to the eventual conclusion. Eight errors occurred in an exploratory branch or were explicitly repaired before the conclusion; seven concerned supplementary claims; and seven were numerical or algebraic errors to which the requested conclusion was robust. Only two traces contained a substantive failure in the argument supporting the correct answer. No category showed a positive CERT-BLIND change in recall (Table 3), so the critical-set result is not driven solely by the repaired or exploratory cases. The substantive category contains only two traces and is not interpreted separately.

A representative self-repaired error occurred in a problem asking for the maximal Choi rank of a complementary quantum channel. The model temporarily claimed that the complement of an isometry has Choi rank one. Several steps later it explicitly calculated that its Choi matrix is $I_{d}$, and therefore has rank $d$, before returning to the correct bound $\min(n,rd)$. The false claim was consequently present in the written trace but did not survive into the final derivation. Note these self-repaired errors are particularly rubric-dependent because exploratory reasoning was the largest source of disagreement in the IRR pilot (see Section II.2).

Supplementary errors concerned claims that were not required for the requested quantity. For example, in calculating the amplitude of a wave transmitted through an impedance-matched, time-varying slab, one model incorrectly treated the transmitted field as a fixed-wavenumber mode with $k=\omega/c$. Spatial-interface matching instead fixes the frequency, and the exact boundary-driven field is not a single fixed-$k$ mode. This changes the description of the transmitted phase, but not the requested amplitude $Ae^{-\alpha L/c}$.

In the robust-conclusion group, an intermediate error could propagate algebraically without changing the requested result. In deriving the higher central charge of $U(1)_{2N_{1}}\times U(1)_{-2N_{2}}$, one model used the incorrect Jacobi factor $\genfrac{(}{)}{0.0pt}{}{2N}{n}$ rather than $\genfrac{(}{)}{0.0pt}{}{4N}{n}=\genfrac{(}{)}{0.0pt}{}{N_1N_2}{n}$ for each individual Gauss sum. The same extraneous factor $\left(2/n\right)$ appeared in both sectors and therefore squared to one, leaving the final expression $\left(N_{1}N_{2}/n\right)$ unchanged.

The remaining two traces reached the correct answer without providing a valid argument for it - they essentially got lucky. One question asked which violin-model parameter changed under a bridge mute, including whether the final member of the group $(\mu,a_{2},f_{2})$ moved up or down. The model justified “down” by arguing that the amplitude $a_{2}$ decreases, although the requested member was the resonance frequency $f_{2}$. The answer was nevertheless correct because the added mass also lowers $f_{2}$. The other asked how many spin-2 Kaluza-Klein eigenvalues lie below 14. The model incorrectly inferred exact double degeneracy of every non-zero eigenvalue from supersymmetric isospectrality and a translated partner potential, and then used an incorrect harmonic frequency. Numerically, the supposedly degenerate eigenvalues are distinct, approximately $11.79$ and $11.85$. Both nevertheless lie below the threshold, so the model’s count of the zero mode plus two non-zero modes was correct. These two cases constitute $2/237$ of all traces and are presented as illustrative failure modes rather than evidence for their population frequency. In every one of these cases the error is invisible from the final answer, which is why supplying the reference answer does not help a monitor find it.

Supplying a monitor with a trusted reference answer improves how well it distinguishes traces that contain an error from traces that do not. The improvement is concentrated almost entirely in cases where the final answer itself reveals that something went wrong. Where the conclusion is uninformative, the effect is flat or negative for seven of eight monitors and slightly positive for one - the large overall benefit of certification is not reproduced when the conclusion is held correct. A correct answer is evidence about the conclusion, not about the argument that produced it, and the monitors in our panel do not reliably separate the two.

This matters for how chain-of-thought monitoring is evaluated. Labelled evaluations are built on tasks with known answers, and it is natural to give the monitor that answer. Doing so measures a capability that combines process verification with conclusion checking, in proportions the aggregate metrics do not reveal. An evaluation such as our CERT condition can report a balanced accuracy of 0.796 alongside exact localization of 0.379 and critical-trace recall of 0.438. Indeed, a conclusion-only oracle that flags exactly those traces whose final answer disagrees with the certified answer would achieve a balanced accuracy of 0.888 on this dataset, despite having zero recall on critical traces and no localization capability.

The correction is procedural and inexpensive. An evaluation of process monitoring should include an answer-blind condition; it should report recall separately for traces whose conclusion exposes the error and for traces where it does not; and it should report localization alongside detection, since a monitor can be credited with a detection while attending to nothing in the derivation. Where a single figure of merit is required, critical-trace recall at fixed specificity measures the capability that process supervision exists to provide. None of this requires new data collection - every quantity is computable from the labels such evaluations already have.

We have observed the same behaviour in a different evaluator role. In earlier work on LLM-as-a-judge marking of physics assessments [19], supplying models with a deliberately corrupted reference solution caused them to mark fully correct student work as wrong, deferring to the reference rather than checking the physics. There the reference was false and the deference produced visible damage to accuracy. Here the reference is genuine and the same deference inflates apparent capability, which is harder to notice because no aggregate number looks wrong. Taken together, the two studies indicate that a reference standard shifts an LLM evaluator from verification toward consistency checking against that standard, across two task settings and two model generations.

The gap concerns the case that motivates process supervision: the conclusion provides no evidence that the reasoning is sound. Reward hacking and deliberate sabotage can share this structure, although the ordinary, mostly non-load-bearing errors studied here are not examples of either. A monitor evaluated with the answer in hand may therefore appear more capable than it is when the conclusion provides no diagnostic cue.

The dependence on monitor strength sharpens this. The strongest monitors in our panel show the clearest separation between clean and critical answer-matching traces, whereas weaker monitors incur the largest false-positive costs in STEP. In control settings the trusted monitor is by construction weaker than the model it oversees [6], so the failure mode concentrates where it is least affordable.

This experiment does not establish monitoring performance against adversarial reasoning. Deliberate concealment may make errors harder to detect, but most errors studied here were not load-bearing and none was optimized to evade scrutiny. The result should therefore be read as a warning about evaluation design rather than as a quantitative bound on sabotage detection.

The critical cell contains 24 traces, so consistency of direction across monitors is the unit of evidence rather than the magnitude of any single change. Localization targets in that cell are author-adjudicated for 16 of the 24 traces. This is unavoidable - it is where the independent arms disagreed - but it makes the headline cell the most adjudicator-dependent part of the standard. Human coverage of the reference standard is incomplete and uneven across cells. Annotation effort was unevenly distributed - one annotator contributed roughly a quarter of the human labels. Outside the pilot, only 6 traces were scored by more than one person, so annotator effects in the human arm cannot be estimated. Further, the reference standard was itself constructed with access to the reference answer. Answer access could therefore have biased both annotation arms against detecting errors in correct-answer traces, the same direction of anchoring studied in the monitors. The critical set may consequently omit less conspicuous errors, while some traces labelled clean may contain undetected errors. This would tend to make critical-trace recall optimistic and may favour answer-aware monitoring conditions in measured specificity. The multi-arm review and source-masked adjudication reduce dependence on any single judge but do not remove this shared conditioning.

Every error is an ordinary model mistake rather than an adversarially planted one, so the extrapolation to sabotage is by analogy. The study is also limited to physics questions. Several monitors emit near-constant or anti-calibrated confidence, limiting threshold-free comparison, and STEP emits no comparable graded score, so BLIND-versus-STEP comparisons are between two fixed operating points. Fable 5 was inadvertently consulted on a small number of disputed traces - we estimate three, though the count was not logged - during adjudication while also serving as a monitor. Repeating the central analysis without it leaves the difference-in-differences positive for all 7 remaining monitors ($p=0.0156{}$; the full panel gives $p=0.0078{}$, with the difference reflecting only the number of monitors). The mean increases from +0.382 (95% CI [+0.256, +0.506]) to +0.415 (95% CI [+0.288, +0.541]). Fable 5’s difference-in-differences was below the panel mean, so excluding it strengthens the effect rather than weakening it. Gemini Pro 3.1 generated one third of the traces it subsequently monitored. Kimi K3 was run at a different temperature and reasoning setting from the rest of the panel.

For extending the work further, the main challenge in the present work was the creation of the ground truth reference standard. Localizing a first error in a physics derivation requires specialized subject knowledge and building this for 237 traces through the multi-arm approach outlined in FIG. 1 took months. Step-level labelled examples already exist for mathematics in benchmarks such as ProcessBench [20]. The same information-ladder manipulation can be transferred to this benchmark unchanged to test whether the interaction reported here holds. Further, the localization results here describe where flags land but not why - modelling flag placement as a choice among a trace’s steps, with position and step length as competing predictors, would test whether monitors are attending to the reasoning or to structurally salient positions within it.