Rethinking Multi-Image Re-Representation in Multi-Image Understanding

Answering a multi-image question can require grounding visual evidence dispersed across images. Re-representation organises this evidence into intermediate records for reasoning. Let $x=(q,I_{1},\ldots,I_{n})$ contain a question and its ordered source images, and let $y$ denote an answer.

A solver $p_{\phi}(y\mid x,z)$ uses the record alongside the original input, giving the answer distribution

The re-representer and solver are usually the same model.

The textual record can describe visual content and express cross-image relations. Tool interaction additionally allows the model to construct new image views for inspection (Yao et al., 2022; Hu et al., 2024).

The trace ends with a stop action and an empty observation. Its distribution factorises as

The environment transforms or combines source images and retained intermediates. Each result is returned as a rendered image with a reusable reference and remains available for subsequent operations. The harness and tool interface are described in Sec. 3.2.

Both forms of re-representation organise evidence from the provided input. The following property characterises their information content relative to that input.

Closed-evidence re-representation. Let $X$, $Z$, and $Y^{\star}$ denote the random variables corresponding to the complete original input $x$ (question and source images before visual tokenisation), the intermediate record $z$, and the ground-truth answer, respectively. In the closed-evidence setting, the model and tools obtain no external evidence or additional observations. With model parameters and tool implementations fixed, write $Z=g(X,U)$, where $U$ collects the randomness used to generate the record and satisfies $U\perp Y^{\star}\mid X$. Thus $Y^{\star}\perp Z\mid X$, implying

This conditional-independence property follows from the stated assumptions; it is not an empirical finding. It concerns information about the answer beyond the complete input $X$. Cropping and re-encoding may expose details absent from the initial compressed visual tokens and improve their accessibility to the solver; conditional independence need not hold when conditioning only on those tokens.

Re-representation settings. We compare five settings that retain the original images and question and share the final-answer format.

A. No explicit re-representation (No-$\mathrm{Re}^{2}$). The model returns only the final answer, without an explicit intermediate reasoning trace.

B. Free-form textual re-representation (T-$\mathrm{Re}^{2}$). The model generates a free-form reasoning trace before answering, using standard CoT prompting (Wei et al., 2022).

C. Prompt-guided textual re-representation (PG-$\mathrm{Re}^{2}$). We extend T-$\mathrm{Re}^{2}$ with question-guided re-description instructions: describe relevant visual content, identify its source images, and organise cross-image comparisons (Kao et al., 2025).

D. Visual re-representation (V-$\mathrm{Re}^{2}$). The model performs online visual re-representation with Mosaic, following the interaction process in Eq. (2).

E. Prefabricated visual re-representation (PV-$\mathrm{Re}^{2}$). We provide visual intermediates constructed in advance alongside the original input, omitting the interaction traces used to produce them. The model uses the same answer-stage prompt as T-$\mathrm{Re}^{2}$ without tool access. This setting separates the use of visual intermediates from their construction.

Multi-image reasoning can require selecting and re-organising visual evidence within and across images. Mosaic provides a persistent workspace in which an MLLM transforms and combines source images and retained intermediates into task-driven visual representations (Fig. 1).

Persistent image workspace. Source images and derived views are stored as separate assets with stable references, such as [img1]. Operations create new assets without modifying their inputs, allowing the model to build on a result or revisit an earlier version. In the shared workspace, each image asset has its own transparent canvas that expands to accommodate transformed or composited content beyond its original boundaries. Regions without image content remain transparent for subsequent composition. Each asset has an editing view (RGBA image) for subsequent processing and an observation view (RGB image) with transparent checkered background as MLLM inputs.

Composable image operations. Mosaic provides ten deterministic image operations on the source images and their derivatives. The model can construct visual representations over successive calls, using the output of one operation as the input to another. Geometric operations select or transform individual views; collage and overlay combine content from multiple assets. Pixel differencing exposes intensity discrepancies between aligned images, while a coordinate grid provides spatial references. Each operation is invoked through a structured call specifying the input assets and parameters. The harness stores the output as a new asset and returns its rendered view and reference to the model. Full tool specifications and rendering details are provided in Appx. A.1.

Evaluation setup. We evaluate on BLINK (Fu et al., 2024), M4Bench (Ye et al., 2025) with fine-grained subtasks. We compare No-$\mathrm{Re}^{2}$, T-$\mathrm{Re}^{2}$, PG-$\mathrm{Re}^{2}$, PV-$\mathrm{Re}^{2}$, and V-$\mathrm{Re}^{2}$ for Qwen3-VL-8B, Qwen3-VL-32B, and GPT-5.4, which are representative models with basic tool-calling capabilities. The prefabricated visual intermediates for PV-$\mathrm{Re}^{2}$ are generated in advance by MosaicAgent-8B using Mosaic and are shared across all solvers.

The benefits vary across tasks. On M4Bench’s Detailed Difference task, Qwen3-VL-8B improves from 7.3% under No-$\mathrm{Re}^{2}$ to 46.1% under T-$\mathrm{Re}^{2}$ and 59.9% under V-$\mathrm{Re}^{2}$. Prompt-guided textual re-description reaches 45.7%. Visual re-representation also improves Detailed Difference over T-$\mathrm{Re}^{2}$ for Qwen3-VL-32B and GPT-5.4, by 6.0 percentage points each. On State Comparison, however, all three models perform worse under V-$\mathrm{Re}^{2}$ than under T-$\mathrm{Re}^{2}$. The effect of re-representation therefore depends on the subtask required, even within the same benchmark.

Visual inputs and online construction have different effects. On BLINK’s Relative Depth task, Qwen3-VL-8B reaches 82.3% with prefabricated visual intermediates, compared with 78.2% under T-$\mathrm{Re}^{2}$ and 74.2% under V-$\mathrm{Re}^{2}$. Thus, providing visual intermediates can improve performance on a task where online re-representation does not. This distinction motivates examining both the usefulness of visual representations and the policy that constructs and uses them.

These results motivate a more focused evaluation of tasks that require fine-grained visual recognition, geometric reasoning, and spatial localisation. We introduce MosaicBench next to examine these tasks in greater detail. The differences between PV-$\mathrm{Re}^{2}$ and V-$\mathrm{Re}^{2}$ motivate studying how visual intermediates are constructed during interaction. We next train a policy with Mosaic and examine changes in its performance and re-representation behaviour.

We introduce MosaicBench to evaluate re-representation on tasks requiring precise visual evidence and relations.

Task coverage. We construct 32 task types for training and evaluation, grouped by their core visual challenge: resolution, orientation, precision comparison, hypothesis testing, context interference, and spatial reference. Each example contains one to five images and a task-specific question. Fig. 3 illustrates representative tasks and their visual evidence.

Data construction. We draw images and annotations from 12 public datasets (VisDrone2019-MOT Wen et al. (2019), TT100K Zhu et al. (2016), CAMELYON16 Ehteshami Bejnordi et al. (2017), MVTec AD Bergmann et al. (2019), CARPK Hsieh et al. (2017), PubLayNet Zhong et al. (2019), LEVIR-CD Chen & Shi (2020), SmartDoc15-CH1 Burie et al. (2015), MSD Yang et al. (2019), Rico Deka et al. (2017), COCO Lin et al. (2014), and BDD100K Yu et al. (2020)) to construct multi-image visual tasks from annotation augmentation. Annotation-based selection identifies relevant objects, regions, and correspondences. Controlled transformations create related views through geometric changes, local edits, and rearrangements. Ground-truth answers follow source annotations or known construction parameters.

Evaluation. We construct MosaicBench from the same generated data, excluding overlap with the training suite at the sample, source, and image levels. The resulting benchmark contains 560 examples across 28 task types, with 20 examples per type. Detailed task-construction procedures are provided in Appx. B.5 and B.6. Of the 28 task types, 26 use four answer options with balanced correct-answer positions.

We initialise the policy with Qwen3-VL-8B-Instruct  (Bai et al., 2025) and train it using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), without supervised warm-up or demonstration trajectories. We use the training suite, introduced in Sec. 4.2; data selection and separation from MosaicBench are detailed in Appx. B.5. Further details on the training setup are provided in Sec. A.3.

Reward. For an interaction trace $\tau$ and final response $y$, we assign a reward based on answer accuracy and format compliance:

Both terms are binary. The accuracy term checks whether the extracted final answer exactly matches the ground truth. The format term requires a parseable answer within <answer>...</answer> tags and a minimum amount of preceding model-generated text. For each training prompt, we sample eight rollouts and compute advantages from group-normalised rewards. Each episode allows up to ten assistant turns.

Training data. We generate 400 examples for each of the 32 task types described in Sec. 4.2, using the source datasets listed there. This yields 12,800 examples. Using the initial policy, we perform eight tool-enabled rollouts with Mosaic and one no-tool run per example. We retain examples answered incorrectly in the no-tool run and correctly in two to seven of the eight tool-enabled rollouts. The resulting training suite contains 3,509 examples across 32 task types. Detailed selection protocols and overlap checks are provided in Appx. B.5.

Overall performance. MosaicAgent-8B achieves 63.3% on MosaicBench and 61.8% on M4Bench, exceeding the strongest evaluated open-weight baseline on each benchmark by 2.4 and 4.3 percentage points, respectively (Tab. 1) and even on par with closed-sourced models. On Mantis, it reaches 81.6%, close to the highest open-weight result of 81.9%. The largest advantage on MosaicBench is in hypothesis testing, where MosaicAgent-8B reaches 62.9%, compared with 31.7% for the strongest open-weight baseline. It also leads the open-weight baselines in precision comparison and orientation by 6.0 and 5.2 percentage points. On M4Bench, the largest lead is in D.Diff, at 74.4% versus 66.6%.

Training dynamics and tool-use distribution. Over 218 RL steps, the mean accuracy reward increases from 0.48 to 0.63 and the format reward from 0.91 to 0.99, comparing the first and last ten steps (Fig. 5). On MosaicBench, total tool calls increase from 1,830 before training to 3,037 afterwards, a $1.66\times$ increase. The distribution also shifts: cropping rises from 36.7% to 54.8% of all calls, and collage from 0.5% to 6.2%. Pixel differencing, by contrast, falls from 16.1% to 3.3%. The trained policy thus allocates a larger share of its calls to extracting image regions and composing views, alongside the increase in overall tool use.

Training with and without the harness. We compare the pre-RL backbone with MosaicAgent-8B and a CoT policy trained on the same training suite without visual harness (Tab. 2). RL training without harness (CoT RL) increases accuracy on MosaicBench from 45.2% to 53.0%, while M4Bench changes from 51.3% to 51.6%. With Mosaic available during both training and inference, MosaicAgent-8B reaches 63.3% and 61.8%, exceeding the CoT RL control by 10.3 and 10.2 percentage points, respectively. Thus, the performance gains are not attributable to RL alone; harness-enabled visual re-representation provides a substantial additional benefit beyond CoT RL on the same training data.

Fixed-prefix forced-answer probing. To measure what can be answered from an intermediate trajectory state, we reconstruct each recorded trajectory by deterministically replaying its tool calls in the same visual workspace. For question $i$, let $H^{p}_{i,t}$ denote the reconstructed context after $t$ tool transitions of a trajectory produced by checkpoint $p$. The context contains the original question and source images together with all intermediate model text, tool calls, tool feedback, and visual observations available up to that point. At $t=0$, it contains only the original input.

At each prefix, we freeze the trajectory: the solver cannot continue reasoning or invoke additional tools. Following early-answering interventions  (Lanham et al., 2023), we append the standard answer instruction and an <answer> prefix, then read the next-token logits for the valid candidate answers. For solver checkpoint $s$, we have

Prefix answerability is the mean of $v_{p,s}(i,t)$ over trajectories. Because probing does not alter or extend the recorded trajectory, changes in answerability reflect the information available at each prefix rather than additional inference performed by the solver.

We cross trajectories produced by the backbone (pre-RL) and MosaicAgent-8B (post-RL) with both solver checkpoints. Holding the solver fixed compares trajectory construction; holding the trajectory fixed compares answer readout from the same context. We retain trajectories with at least one tool call and a valid final response, and pair re-representer comparisons over the 441 questions meeting these criteria for both checkpoints.

Post-training trajectories improve final-prefix accuracy. With the solver fixed, trajectories produced after training increase final-prefix answerability by 11.0 percentage points with the backbone solver and 10.5 points with the trained solver as shown in Fig. 5. For fixed trajectories, switching to the trained solver changes the area under the normalised-progress curve by $+0.019$ on pre-training trajectories and $+0.014$ on post-training trajectories. The re-representer gains transfer to both solvers, supporting improved trajectory construction as the training benefit. The gains emerge later in the interaction. Post-training trajectories contain more tool steps on average (5.41 vs. 3.16). We therefore also compare answerability against absolute tool-step budgets. The re-representer advantage is absent over the first two steps.

We next inquire how agents with Mosaic proactively solve the multi-image tasks.

Answerability profiles. Among trajectories with a correct final-prefix probe, we identify two transition patterns. Progression starts incorrect and becomes correct without a later reversal, while self-correction contains at least one correct-to-incorrect transition followed by recovery. Trajectories that remain correct at every prefix are consistently correct. Separately, we measure post-stabilisation continuation: at least two additional tool steps after the probed answer stabilises, which may reflect verification or unnecessary tool use. Fig. 6 compares 210 final-prefix-correct trajectories before training and 335 after training.

Progression and self-correction. Progression captures the direct case in which successive visual operations make the available evidence sufficient for answering. Self-correction reveals a less monotonic process: an intermediate representation can move the model away from a correct answer before later operations recover it. Its increased prevalence after training suggests that successful visual reasoning can involve revising intermediate rather than only accumulating evidence.

Verification and overtooling. Post-stabilisation continuation captures a different behaviour: the agent keeps processing visual evidence after the probed answer has already stabilised. Such steps may verify an existing answer by inspecting additional evidence, or may constitute unnecessary tool use. Their increased frequency after training shows that the learned policy does not simply stop once a correct answer becomes available.

Training changes the mixture of problem-solving modes. Training does not simply produce more monotonic progression: although its absolute count increases, its share decreases from 41.9% to 35.5%, while self-correction and post-stabilisation continuation become substantially more frequent. The learned policy therefore exhibits a different mixture of progression, revision, and continued processing rather than converging to a single strategy.

In this work, we used generative AI tools to assist with translation, provide feedback on research methodology, and support qualitative and thematic data analysis.preprint We did not use generative AI to generate dataset examples or develop theoretical models. Additionally, we used generative AI tools to create artefacts, identify relevant literature, summarise or analyse existing literature, and create or modify scientific figures or images. We reviewed all AI-assisted work. All code was reviewed by at least three authors. The authors verified all scientific claims. AI-assisted figure generation was limited to visual styling and did not modify the underlying data. We take responsibility for the final content of this work, including text, claims or artefacts produced with the aid of generative AI.