SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task

Every model receives the same system prompt. It casts the model as a scientific fact-checker, directs it to read axis labels, legends, sub-figure tags and exact cell values, to attend to comparison words (highest, outperforms, increases) and to metric direction, and to reply with only a JSON object carrying a brief reasoning field, a support score in $[0,1]$, and a label. No few-shot examples are given and the prompt is never tuned per model, so differences across the eleven systems reflect the models rather than the prompting. We disable the extended reasoning trace on the Qwen3.5 and Qwen3.6 models, which is three to four times faster and made a full development sweep feasible on shared hardware; all other models run in their default configuration. Images are resized where needed to keep memory within budget on shared GPUs (1536px longest side for the Transformers-based models, Qwen’s native min/max-pixel budget for the Qwen line, 1568px for Opus; GPT-5.5/GPT-5.6 Sol are sent unresized); large MoE models are sharded across two GPUs, with 8-bit quantisation where a model would otherwise not fit. Opus 4.8 runs through its CLI, reading images from disk, needing no local accelerator. Across the eleven models the score-threshold fallback ffres on well under one output in twenty. All models see identical inputs, and every raw response is stored, so fusion, routing, and the pair prior are computed afterwards from those stored responses rather than by re-querying the models. Reproducing our results requires neither GPUs nor API access.

Under a short organizer-granted extension (§5) we reran Subtask 1 with the structured-data channel removed for table-type claims and a newly available model, GPT-5.6 Sol (OpenAI, 2026), and submitted five true PNG-only runs (Table 2, rows 1.8–1.12; official test scores discussed in §5).

Opus 4.8 and Gemma-4-31B both exceed o4-mini; our Qwen3-VL-8B (71.5/45.2) reproduces its official baseline (72.1/46.9), validating the harness. The newest open model we tested, Qwen3.6-35B-A3B, is the second-strongest open system (82.5 F1 / 66.2 pair-acc), ahead of Qwen3.5-35B-A3B and surpassing o4-mini on Subtask 2 (92.0).

OpenAI’s GPT-5.x line (OpenAI, 2026) and Anthropic’s Claude Fable 5 (Anthropic, 2026) became available after our original runs (Runs 1.1–1.5, 2.1–2.2 in Table 2); we evaluated both on the full development set and submitted additional runs incorporating each (Runs 1.6–1.7, 2.3–2.4). GPT-5.5 reaches 84.5 F1 / 69.9 pair-acc on Subtask 1 and 97.4 on Subtask 2 (vs. 92.6 for Opus, our original Subtask-2 run), with a table-vs-figure gap of only $+2.5$ (Table 4), far below every model in our original runs ($+5$ to $+14$). Swapping it into the routed ensemble lifts the leak-free pair prior from 93.5 to 95.6 (Run 1.6).

Claude Fable 5 is stronger still (86.3 F1 / 74.4 pair-acc on Subtask 1, 97.7 on Subtask 2, Table 3) and has the smallest table-vs-figure gap of any model we tested ($+1.8$). Reaching this number required catching our own mistake: an initial pass, computed from our pipeline’s own prediction file, read an implausible 99.4 pair-acc; that file turned out to use the hidden claim_id_pair field for pair-forcing and to silently default failed API calls to a low-confidence label, rather than reflecting genuinely per-sample, per-model judgments. A model-specific usage cap also required an automated retry loop across several hours to reach full coverage. The corrected, fully-covered number is strong but in line with GPT-5.5 rather than an outlier. We view this episode as a useful, if uncomfortable, illustration of this paper’s broader warning (Section 5): even a solution built around honest, leak-free evaluation can be misled by its own tooling, and every number in this paper that could not be cross-checked this way deserves the same scrutiny. Adding Fable 5 to the GPT-5.5-augmented routed ensemble raises the leak-free pair prior from 95.6 to 96.2 on all 352 pairs, a smaller marginal gain than GPT-5.5’s own, consistent with diminishing returns as the ensemble’s members grow individually stronger and more correlated.

Every model is weaker on figures than on tables, by 1.8 to 14.4 points. The gap is by far the smallest for the two newest frontier models (Fable 5 $+1.8$, GPT-5.5 $+2.5$) and largest for the smaller open ones (Qwen3-VL-30B $+14.4$, Gemma-4-E4B $+11.4$), indicating that figure understanding is exactly where capability is still increasing—and where the latest models pull ahead. The per-type ranking also reshuffles—for example, GLM-4.6V-Flash reads figures nearly as well as much larger models despite a lower overall rank. Because the manipulation types that fool models (legend and category swaps; Section 6) occur in figures, this is also where most of our residual errors concentrate, and it is exactly the signal evidence-type routing exploits.

Score-weighted fusion of the three strongest models reaches 72.2 pair-acc, and evidence-type routing 73.0; pooling all 8 models available at the time of these runs is worse (66.2), as the weaker systems add noise rather than diversity. The choice of fusion rule matters little—majority vote, score-weighting, reliability-weighting, and an exhaustive subset search all land at 72.2, and only evidence-type routing improves on them. The legal pair prior is the decisive step, lifting pair-accuracy to 93.5 (leak-free; the leaky position-tie-break variant reads 94.9). The gain holds across domains (pair-acc 95.8 ML, 94.5 NLP, 89.4 PeerJ), with the biomedical split being the most challenging, consistent with its denser tables and figures. Subtask 2 is already a pairwise selection task; a three-model fusion edges out Opus alone by a small margin (93.5 vs. 92.9, both refreshed post-§5).

The gain is concentrated: 71.9% of pairs are already correct under honest prediction; in 27.3%, both members get the same absolute label (unresolvable by any threshold), and correcting only these recovers 94.6% pair-accuracy—indistinguishable from correcting all 352. A global threshold sweep instead gains at most 0.5 points (71.9$\to$72.4, oracle-tuned), ruling out simple recalibration. Errors here are biased—models default to “Supported” twice as often as “Refuted” when confused (65 vs. 31)—and the 19 pairs relative comparison cannot fix concentrate in the manipulation types already flagged as visually undetectable (§6): legend/category swaps fail three times more often than cell-value edits, mostly in figures.

The two subtasks reward different recipes. Subtask 2 already presents both evidence images and asks which supports the claim—structurally the same relative judgment that our pair prior reconstructs for Subtask 1. Here Opus 4.8 is strong on its own (92.9 accuracy, vs. 88.9 for the best open model and 85.2 for the o4-mini baseline); a three-model fusion improves on it by a small margin (93.5), a gain that postdates our original submission. We therefore submitted Opus directly for Subtask 2 and reserved fusion for Subtask 1, where the models are more complementary. The contrast also explains why our Subtask-1 pair prior works: it imports exactly the relative-comparison formulation that makes Subtask 2 tractable.

On the held-out fold, QLoRA fine-tuning lifts Qwen2.5-VL-7B from 77.8 to 84.7 pair-acc (+6.9 over zero-shot), a substantial gain for a few hundred training pairs and a single A100. Yet the tuned 7B still trails the training-free ensemble+prior (93.1 on the same pairs) by a wide margin. Fine-tuning is a viable, cheap route to a competent standalone model, but still does not provide the best results—the headroom lies in scale, error-diverse fusion, and the pairing structure, consistent with our broader finding that the residual difficulty is structural, not a matter of model adaptation.

Adding the checker’s score (Section 4.6) to the routed ensemble covers 184 of 352 pairs (the remainder have no paper excerpt with enough lexical/numeric overlap to retrieve) and lifts pair-accuracy from 93.5 to 93.6, or from 95.6 to 95.9 once GPT-5.5 is included. The gain is modest in aggregate because most residual errors are the visually-undetectable swaps identified in Section 6, where the source paper’s prose often does not reveal the true label-to-curve mapping either; but on failures where the paper text does state the relevant number, the checker recovers them by catching an explicit image/paper contradiction (e.g., a table cell reads 80.52% in the image while the paper states 85.52%). This turns our earlier negative result (retrieved context degraded accuracy when used to answer the claim directly) into a positive one, once the same text is used to check consistency instead.

On the held-out fold (Section 4.7), distillation raises Qwen3.5-9B from 51.4 to 62.5 pair-acc ($+11.1$) and Gemma-4-31B from 49.3 to 55.6 ($+6.3$), showing that some of the “cross-check the paper” behavior transfers to a small open model via a modest number of QLoRA examples. Both students still trail the Claude teacher (77.1 on the same pairs, 84.1 on the 41/72 it actually covers) by a wide margin, so this is a partial distillation rather than a drop-in replacement; closing the gap is left to future work.

The position bias identified in §4.5 also inflates the pair prior itself: with position-based tie-breaking, it reads 94.9 pair-acc, essentially the same $\sim$15-point leak as the gated-fusion probe. Removing position from the tie-break drops this to 93.5, which we report as our headline; the gated variant remains a separate, labeled run (§4.5).

Two “more information” fixes reduced accuracy: feeding retrieved paper context to the context-dependent pairs (92.2$\rightarrow$84.3) and a legend-mapping prompt that asks the model to transcribe the legend before judging. Both fail for the same reason—the manipulation is inside the evidence, while external/explicit text describes the un-tampered original and thereby masks the tampering. This is the empirical counterpart of the audit in Section 6: the hard cases are self-consistent edits with no in-image signal, so adding context cannot help and often actively misleads.