Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgery

Vitreoretinal surgery involves high-precision microsurgical procedures that demand exceptional skill and real-time decision-making. Enhancing a surgeon’s contextual awareness through automated systems is critical for improving clinical outcomes, reducing cognitive load, and standardizing surgical training [15, 18]. Surgical phase recognition serves as a foundational capability for such context-aware systems; however, it remains challenging in vitreoretinal procedures due to subtle tool-tissue interactions and the scarcity of annotated multimodal datasets. We distinguish between two levels of surgical phases: macro-phases, corresponding to the conventional high-level stages of the procedure, and micro-phases, capturing fine-grained instrument states such as forceps jaw closure and tool-to-retina distance. Current AI-driven vitreoretinal surgery analysis often relies on single-modality video data, neglecting the depth-resolved structural insights provided by iOCT, which offers critical subsurface tissue information and tool-tissue proximity that video alone cannot capture.

Surgical narrations offer rich semantic supervision [27], yet datasets jointly containing narrations, microscope views, and iOCT B-scans are largely absent from the literature. Curating such a tri-modal dataset is itself a significant challenge, requiring synchronized acquisition across heterogeneous imaging systems in a demanding intraoperative setting. Moreover, clinical iOCT B-scans are not constrained to anatomically informative regions, and current technology does not support tool-following acquisition, which leaves much of the captured cross-sectional information contextually decoupled from the surgical procedure. To address this, we leverage a synthetic dataset of tool-aligned iOCT B-scans, ensuring that the acquired cross-sections are anchored to the surgical site and thus carry procedure-relevant context. Utilizing microscope views as a common anchor, we then bridge two complementary datasets, one of microscope–narration pairs and one of microscope–iOCT pairs, aligning all three modalities in a shared latent space without requiring a fully annotated tri-modal dataset.

Once the shared latent space is learned, supervised by surgical narrations and fine-grained annotations of the synthetic dataset, any single modality carries enough semantic structure to support inference about the others at inference time, as the joint supervision brings their representations into close proximity. For instance, a microscope view alone could in principle support an approximation of tool-to-retina distance without iOCT, to the extent that the two representations come to lie near each other in the latent space; we regard such cross-modal inference as an exploratory direction and evaluate it directly only where ground truth is available (Sec. 4).

The implementation is done using PyTorch 2.7.0 and Python 3.11. The codebase leverages GPUs with CUDA capabilities and standard deep learning libraries, including NumPy 2.2.7 for numerical computations, OpenCV 4.11.0 for video processing, and Pillow 11.2.1 for image handling.

The held-out set of five surgical videos is used to assess the trained PeskaVLP model in order to create a baseline for surgical phase recognition. Phase predictions were made without any temporal modeling, following a zero-shot procedure: for each of the five macro-phases, a descriptive text prompt is designed and projected into the latent space via Encoder 1, while each microscope frame is independently projected into the same latent space via Encoder 2. The predicted phase for a given frame is then assigned as the class whose text embedding is closest to the frame embedding, with closeness measured via cosine similarity. The baseline model’s performance is displayed in Table 2.

For macro-phase recognition, the dual-head MS-TCN++ model with label refinement outperformed the zero-shot baseline, achieving a mean F1 score of 53% with mean precision of 50.3% and mean recall of 57.9%. The performance breakdown by surgical phases is shown in Table 2. The model shows particularly strong gains for Anterior and Posterior Preparation, while Membrane Identification and Retina Stabilization remain comparatively harder to recognize. This configuration jointly introduces temporal modeling, label refinement, and multi-task learning relative to the Stage A baseline; disentangling their individual contributions requires dedicated ablations, which we leave to future work.

Micro-phase prediction was assessed quantitatively on the held-out synthetic ERM peeling sequences, where per-frame ground truth is available, and only qualitatively on real surgical videos. On synthetic data, the model achieved an MAE of 0.050 $\pm$ 0.001 for tool-to-retina distance and 0.167 $\pm$ 0.135 for forceps jaw closure. This ordering is counterintuitive, since the lower MAE for distance would ordinarily indicate the easier target. The two signals, however, have very different statistics. Tool-to-retina distance changes slowly and stays within a narrow band for most of the procedure, so a near-constant, over-smoothed prediction already yields a small absolute error while failing to follow the instrument’s true vertical motion, an expected consequence of the missing depth cues in 2D imagery. Forceps jaw closure instead spans the full $[0,1]$ range with frequent open/close transitions; the model recovers the timing and direction of these transitions but incurs larger errors and high per-sequence variance ($\pm$0.135) around them. In this sense forceps closure is predicted more informatively, since the clinically meaningful state changes are captured, even though its MAE is higher. On real videos the same qualitative pattern holds: forceps-state estimates vary meaningfully over time, whereas distance estimates stay stable but conservative. As no real-domain ground truth is available, these real-video micro-phase results should be read as exploratory rather than as validated measurements. Overall, discrete state changes appear learnable under weak cross-modal supervision, while continuous depth estimation remains challenging without explicit 3D priors.

Our results suggest that the surgical microscope image can serve as a common anchor to bridge otherwise disjoint datasets, allowing structural priors learned from synthetic iOCT to propagate into real surgical video without ever requiring synchronized tri-modal recordings. As synthetic data generation matures and the curation of real narrated surgical videos expands, both the fidelity of learned representations and the overall performance are expected to improve substantially. Rigorous evaluation on larger and more diverse datasets, including real surgical videos annotated with forceps properties to directly assess the method’s adaptability, will be essential to disentangle the contribution of each modality, inform more principled training strategies, and determine under which clinical conditions each modality provides the greatest benefit. Loss functions that more explicitly encourage cross-modal attention between the modalities, rather than only pulling matched pairs together, would also be a promising direction. This is a natural next step toward building richer, more robust multimodal representations for surgical AI, a form of cross-modal reasoning that surgeons perform implicitly throughout a procedure.

This work was supported by the Bavarian Research Foundation (BFS) under Grant AZ-1569-22. We gratefully acknowledge Hessam Roodaki and Ghazal Ghazaei for their support and helpful discussions.