SILICA: Repurposing Diffusion Priors for Joint Glass Segmentation and Depth Estimation

Traditional approaches to perceiving transparent surfaces often frame the problem as a 2D segmentation task. Boundary-centric models [22, 5] rely on the high visual contrast of glass edges, but fail on frame-less doors or complex glass panes where edge cues are scarce. Conversely, context-driven networks [14, 11] exploit high-level semantic co-occurrences, but also fail when presented with complex, cluttered backgrounds visible through the glass. Even recent, massive state-of-the-art models like SAM3 [2] struggle in the presence of salient features behind the glass surface.

Existing glass segmentation datasets [22, 14, 11] offer visual diversity but strictly provide 2D segmentation masks, lacking the metric depth annotation. While manipulation-focused 3D datasets [19, 3, 12] provide dense depth, they are restricted to small tabletop objects. Recent attempts to capture real-world planar glass depth, such as GWDepth [10], are severely limited in scale and diversity (comprising only $\sim$1,200 samples). This lack of paired, diverse, real-world 3D glass data fundamentally motivates SILICA’s training strategy, which bridges real-world segmentation masks with synthetic depth annotations.

Estimating depth “in the wild” refers to methods that are successful across a wide range of (possibly unfamiliar) settings. While early methods predicted relative depth, its limited geometric utility drove a shift toward affine-invariant depth, which recovers structure up to an unknown global scale and offset. Recently, foundational models [24, 23] have achieved exceptional affine-invariant zero-shot performance by training massive discriminative architectures on large-scale synthetic and pseudo-labeled datasets. However, despite this massive data coverage, these models still fundamentally fail to resolve depth for transparent surfaces in complex settings, incorrectly predicting the visible background instead.

Rather than training massive networks from scratch, state-of-the-art methods repurpose the encyclopedic spatial priors of text-to-image Latent Diffusion Models (LDMs) like Stable Diffusion. Operating in a compressed latent space, LDMs implicitly capture complex structural hierarchies from internet-scale data. Breakthroughs like Marigold [8] and GeoWizard [4] successfully adapt these generative priors for affine-invariant depth, achieving remarkable zero-shot generalization while enabling highly resource-efficient fine-tuning. However, unlike these previous methods, which systematically discard the multi-modal conditioning pathway, we leverage the model’s full architectural capacity. We propose a targeted fine-tuning protocol that explicitly re-engages this conditional alignment, achieving robust, in-the-wild performance for our joint task (as demonstrated in Fig. 5).

For diffusion training, there are two basic parameterizations of the diffusion loss function $L_{t}$:

1. 1.

   $\epsilon$-prediction: The model $f_{\theta}^{\epsilon}$ learns to predict the added noise $\epsilon$ [7]:

   $$L_{t}^{\epsilon}=\left\|\epsilon-f_{\theta}^{\epsilon}\!\left({z}_{t}^{y},{z}^{x},t\right)\right\|^{2}$$

2. 2.

   $x_{0}$-prediction: The model $f_{\theta}^{{z}}$ learns to directly predict the clean target sample ${z}^{y}$ [7]:

   $$L_{t}^{{z}}=\left\|{z}^{y}-f_{\theta}^{{z}}\!\left({z}_{t}^{y},{z}^{x},t\right)\right\|^{2}$$

Previous literature [1, 20] indicates that while $\epsilon$-prediction relies on scaling coefficients that severely magnify output variance at high noise levels, $x_{0}$-prediction directly estimates the clean target sample, thereby avoiding this instability and yielding much steadier predictions. Furthermore, fixing the timestep to $t=T$ eliminates the possibility of error propagation during the multi-step denoising process [6]. However, due to the stochastic nature of diffusion models, feeding random noise to the input latent still yields slightly different outputs for different random seeds, which was empirically shown by prior work [6].

For our robotics perception task, it is essential to guarantee consistent outputs regardless of external factors. Hence, we opt to remove the Gaussian noise addition ${z}^{y}_{t}$ entirely. Instead, we directly feed the image latent ${z}^{x}$ into the U-Net at the fixed timestep $t=T$:

This deterministic formulation ensures consistent predictions, and allows the pipeline to train and infer more efficiently, even with limited data.

Sanity check for using the same VAE and Decoder for joint task: While prior work [8] demonstrated that the frozen Stable Diffusion Variational Autoencoder (VAE) $\mathcal{E}$ can accurately reconstruct affine-invariant depth maps with low error(even though being pretrained on RGB), a similar re-use for segmentation or joint modeling of segmentation and depth estimation may not be assumed. Thus, we conduct an initial sanity check to verify if the frozen VAE $\mathcal{E}$ and Decoder $\mathcal{D}$ can accurately reconstruct binary glass segmentation. For this, we normalize the single-channel ground truth mask to the operational range of the VAE, i.e., $[-1,1]$, and replicate it across three channels to simulate an RGB input tensor $s$. We then pass it through the frozen VAE Encoder to obtain its latent representation $z^{s}=\mathcal{E}(s)$, and to recover latent binary segmentation $z^{s}$, it is immediately passed through the frozen Decoder $\hat{z}^{s}=\mathcal{D}(z^{s})$. Subsequently, averaging the three output channels yields an exceptional reconstruction mean Intersection over Union (mIoU) of 99.97%, demonstrating that latent compression and recovery can be done with minimal error. This enables us to use the same VAE and Decoder for training multi-modal data without using a domain-specific Encoder Decoder.

We not only have to deal with multi-modal data but also sim and real data mix. Due to the limitations of datasets containing both glass segmentation masks and accurate depth-to-glass annotations, our training configuration inevitably introduces a significant domain disparity: our segmentation pairs $({s},{x}_{s})$ consist entirely of real-world samples, while our depth samples $({d},{x}_{d})$ are entirely synthetic. To validate our core hypothesis that joint training enables a mutually beneficial information exchange capable of driving robust zero-shot real-world generalization for both our objectives, it is important that the shared U-Net $f_{\theta}$ effectively handles these modalities and domain disparity.

Taking inspiration from prior literature [4, 13], we initially evaluated a strategy that makes use of a single U-Net $f_{\theta}$ to handle multi-modal data via a task switcher, denoted as $s$. The switcher $s$, a one-dimensional vector that labels different tasks, $s\in\{s_{d},s_{s}\}$ (depth estimation and segmentation, respectively), is mapped through a low-dimensional positional encoder and added directly to the network’s time embeddings. Formulated as:

However, as detailed in Sec. V-D, this configuration quickly collapses. Hence, to establish a more robust task-routing mechanism, we re-examine the core architecture of our generative prior. Intuitively, for a text-to-image model such as Stable Diffusion, which was trained on internet-scale datasets to perform high-resolution image generation explicitly conditioned on text prompts encoded by CLIP [16], it must maintain a profound structural connection between its language prompts and its visual outputs. However, previous monocular depth estimators have largely ignored this prompt guidance pathway [8, 6, 13, 4], feeding empty text prompts to extract only visual features. In contrast, we propose a fine-tuning protocol to fully exploit this pre-trained text-image alignment. Since a visual synthesis model inherently possesses a deep structural understanding of scenes, leveraging CLIP’s shared embedding space for task conditioning provides a significantly more effective mechanism for routing cross-modal information.

We introduce the Mirage 18k Dataset, comprising 18,353 manually annotated images with paired binary segmentation masks, spanning 38 unique scenes. We hand-picked 12,420 samples strictly for segmentation training and 3,527 strictly for segmentation testing. To ensure there is no overlap between the two sets, we make sure that no data taken from one scene is part of both the test set and the training set. The remaining 2,406 samples include paired ground-truth metric depth to glass in addition to the binary segmentation masks, intended strictly for zero-shot depth and segmentation evaluation.

Our dataset captures a diverse distribution of glass structures in the wild: we have complex scenes in which multiple types of glass structures (e.g., a frosted door next to an indoor pane) are in the same image. Fig. 3 shows the distribution quantitatively.

We use the CVAT annotation tool to generate ground truth binary glass segmentation masks. Ground truth depth to glass is obtained by modeling each pane as a 3D planar surface. Temporary opaque paper markers were placed at the corners of each pane and annotated for in CVAT, and their median depth was extracted from a local $15\times 15$ pixel neighborhood. To ensure the frames line up perfectly, we record data twice: a completely stationary robot first records the glass adorned with the markers to capture reliable metric depth to glass surfaces, and then records a second sequence from the exact same pose after the markers are removed. By solving an overdetermined system via least-squares optimization in the inverse depth space, we reconstruct dense metric depth maps, as illustrated in Fig. 4. This ground truth depth is used strictly for depth evaluation, and not for training.

As presented in Fig. 5, we provide a qualitative comparison between our approach and the top-performing baselines across highly challenging environments typical of indoor robotic deployment. Notably, these complex, real-world scenarios are uniquely introduced by our novel Mirage 18k dataset to explicitly benchmark operational robustness.

①Crowded shopping mall (large glass panes): Our model successfully ignores dynamic obstacles (humans) and accurately segments the glass despite complex background details. In contrast, TransLab and SAM3 fail due to background and foreground saliency. For depth estimation, while DepthAnythingv2 incorrectly captures the depth of objects behind the glass, our model accurately predicts the glass depth, while Lotus-D also gave decent predictions but fails in consistency.

②Half-opened glass door: TransLab fails to predict the open door, and SAM3 misses the door structure entirely, segmenting only the isolated pane. Our model, however, extracts a highly accurate segmentation mask. In terms of depth, Lotus-D wrongly predicts a fully closed scene at an incorrect distance, and DepthAnythingV2 struggles significantly on the right side due to background saliency. In contrast, out model predicts accurate surface depth.

③Extreme close-up window (minimal context): Given highly constrained viewpoints, TransLab and SAM3 predict incorrect boundaries influenced by a background pillar. Our model precisely isolates the true, closer glass boundary. Furthermore, Lotus-D and DepthAnythingV2 completely miss the transparent surface, outputting background depth, while our model consistently resolves the correct glass depth.

These qualitative results provide clear evidence that our joint formulation effectively learns a robust spatial hierarchy, successfully resolving severe visual ambiguities in the presence of complex transparent structures.

Designed explicitly for robotics, SILICA is integrated into our custom autonomous wheelchair stack for real-world navigation. To operate in a new environment, we first collect a map by feeding the corrected glass depth points from our pipeline into RTAB-Map [9] to construct an accurate 3D map. From which, we obtain a 2D occupancy grid map where the glass areas are included as obstacles. These maps are then utilized for our MPPI-based 2D navigation and 3D localization, enabling the wheelchair to safely plan paths and navigate through glass-structured environments (as shown in Fig. LABEL:fig:fig_teaser). Crucially, multiple deployment runs verify that our model produces very rare false positives in glass-free scenes, ensuring highly reliable robotic operation. Running on a laptop with an NVIDIA RTX 3060 GPU, the pipeline achieves an inference rate of approximately 5 Hz for joint segmentation and depth estimation. Because glass structures are static, we directly inject these predictions into a spatio-temporal voxel grid with a decay mechanism to ensure smooth real-time performance. Finally, we also observe that our pipeline implicitly improves overall 3D mapping, as we provide RTAB-Map with stabilized depth points on glass surfaces results in significantly better and cleaner final maps.

CLIP-Conditioning: To justify our architectural design, we ablate our CLIP-based conditioning module against a naive positional encoder for task switching (Eq. 2). As established earlier, the severe scarcity of real-world glass depth annotations necessitates training on a complex mixture of synthetic depth and real-world segmentation data. Under these multi-modal conditions, simple positional encoding suffers from severe task interference. As shown in Fig. 6, the baseline fails to establish a proper spatial hierarchy: it wrongly predicts the glass surface as abnormally close in the depth map and also fails foreground-background disambiguation in the segmentation mask (e.g., misclassifies background furniture as foreground).

In contrast, our CLIP-conditioned approach (detailed in Sec. III-B) connects directly to all cross-attention layers of the U-Net, fully leveraging the robust priors of the diffusion model. This mechanism enables highly stable task switching and facilitates effective mutual information transfer between the depth and segmentation pathways. As seen in Fig. 6, this formulation successfully enforces a strict spatial hierarchy, resolving the visual ambiguity and dramatically improving the output quality of both tasks. This empirically proves that CLIP conditioning is essential to bridge the sim-to-real domain gap and successfully execute our joint-prediction framework.

Eliminating glass depth annotations: To test if explicit glass supervision is necessary, we filtered Hypersim, discarding scenes with $>10\%$ glass pixels. Retraining from scratch on 24,000 strictly glass-free images yielded only marginal metric differences (Tab. IV). This robust performance definitively validates our core intuition: explicit glass depth annotations are not strictly required. Instead, our CLIP-conditioned joint training framework teaches the network the correct spatial hierarchy for transparent surface perception.

Mutual Benefit of Joint Training: Ablating SILICA against single-task baselines proves their strict interdependence. A segmentation-only model suffers a $\sim 5.0\%$ mIoU drop with increased real-world false positives. Conversely, a depth-only model lacks the predicted segmentation mask essential for explicitly filtering corrupted raw sensor data during metric alignment, resulting in a $>80\%$ error spike on glass regions.