LISynSeg: Data-Centric Label-to-Image Synthesis for Cross-Modality Whole-Heart Segmentation

SynthSeg trains a segmentation network from label maps by applying spatial deformation and generating images with randomly sampled class intensities and acquisition effects [1]. Billot et al. evaluated this approach on the Multi-Modality Whole Heart Segmentation (MMWHS) challenge, which preceded CARE-WHS. Their experiment used 13 MRI label maps containing the same seven cardiac structures considered here. Because these labels did not represent all visible tissues, intensities from the corresponding MRI scans were clustered to divide each foreground label into two generation regions and the background into three to ten regions. These regions controlled image synthesis, while the segmentation target remained the original seven-class map [1, 13].

LISynSeg uses label-to-image synthesis to augment rather than replace real-image training. Let $(x_{i}^{a},y_{i}^{a})=D(x_{i},y_{i})$ denote a real image and label after standard nnU-Net augmentation, and let $y_{i}^{g}=G(y_{i}^{a};\eta_{i})$ denote the label after the Myo–LV operation described in Sec. 2.2. For each training example, we sample $z_{i}\sim\mathrm{Bernoulli}(\rho)$ with $\rho=0.10$ and use

Here, $z_{i}=1$ selects the synthetic branch, in which $y_{i}^{g}$ is used both to synthesize the image and as the segmentation target, whereas $z_{i}=0$ retains the augmented real image–label pair. For the synthesis ablation without boundary exchange, $G$ is the identity transformation. We do not apply an additional affine or elastic deformation within $S$ because nnU-Net has already spatially transformed the image and label.

The synthesis function operates directly on the seven cardiac labels and the background. For every label value present in the map, its Gaussian mean and standard deviation are sampled independently as $\mu_{l}\sim U(0,255)$ and $\sigma_{l}\sim U(0,12)$. We then simulate partial-volume effects, a bias field with standard deviation sampled from $U(0,0.35)$, gamma variation with $\log\gamma\sim\mathcal{N}(0,0.15^{2})$, and Gaussian noise with standard deviation sampled from $U(0,3)$. Resolution profiles are sampled from the native-to-target spacing ratios observed among training subjects in the current fold and are capped at a factor of three. Each synthesized image is finally standardized to zero mean and unit variance.

Myocardial thickness, determined by the Myo and LV boundaries, is one of the major sources of heterogeneity in the WHS datasets. Generic deformation models provide little direct control over this thickness; we therefore introduce an explicit Myo–LV boundary exchange step.

Let $M_{i}$ and $V_{i}$ denote the Myo and LV voxel sets in the augmented label map $y_{i}^{a}$. For any binary voxel set $Q$, let $\mathcal{D}_{1}(Q)$ denote its dilation using a $3\times 3\times 3$ neighborhood. The two interface bands are $B_{V\rightarrow M}=\mathcal{D}_{1}(M_{i})\cap V_{i}$ and $B_{M\rightarrow V}=\mathcal{D}_{1}(V_{i})\cap M_{i}$.

For each selected synthetic example, thickening or thinning is sampled with equal probability: thickening transfers voxels from $B_{V\rightarrow M}$ to Myo, whereas thinning transfers voxels from $B_{M\rightarrow V}$ to LV. The left panels of Fig. 3 show examples of this operation in CT and MRI, with the exchanged voxels indicated by white contours. Rather than transferring the complete interface band, we select a spatially smooth subset using a random field; the selected voxels are not required to form a single connected component.

The largest admissible exchange is determined by requiring the Myo and LV volume ratios to remain within $[0.72,1.35]$ and $[0.80,1.20]$, respectively, and the number of exchanged voxels is sampled as a fraction $\lambda\sim\mathcal{U}(0.35,0.85)$ of this limit. The transformation preserves $M_{i}^{g}\cup V_{i}^{g}=M_{i}\cup V_{i}$ and leaves $G(y_{i}^{a};\eta_{i})(v)=y_{i}^{a}(v)$ for $v\notin M_{i}\cup V_{i}$. If either class is absent from the sampled patch or no exchange satisfies the volume constraints, $G$ returns the original label map. This boundary exchange defines $G$ in Eq. 1 and is applied only to examples selected for synthesis.

The CARE protocol defines the intended AO and PA extents, but the released annotations vary at distal endpoints that are not always marked by visible anatomical boundaries. Standard Dice and cross-entropy losses nevertheless treat every voxel as a fixed target, including voxels near these endpoints; this can produce conflicting supervision when annotations encode different endpoint extents.

For vessel $c\in\{\mathrm{AO},\mathrm{PA}\}$, we associate the AO with the LV and the PA with the RV. Using the dilation operator from Sec. 2.2, the proximal region $R_{c}$ contains vessel voxels in contact with $\mathcal{D}_{1}(\mathrm{LV})$ for the AO or $\mathcal{D}_{1}(\mathrm{RV})$ for the PA. If no ventricular contact is present in the sampled patch, no endpoint mask is applied. The right panels of Fig. 3 show the corresponding ventricle used to locate the proximal vessel region and the distal voxels excluded from the segmentation loss.

Let $\mathbf{s}$ denote the nnU-Net target spacing and $p(v)=\mathbf{s}\odot v$ the physical coordinate of voxel $v$. The proximal reference point is the mean coordinate $r_{c}=|R_{c}|^{-1}\sum_{v\in R_{c}}p(v)$, and the distance of vessel voxel $v$ from this point is $d_{c}(v)=\lVert p(v)-r_{c}\rVert_{2}$.

The closest 90% of annotated vessel voxels retain their original supervision, whereas the remaining distal voxels are assigned to the ignore set $U_{c}$. The distal cap is defined by the 96th percentile of $d_{c}$, and $U_{c}$ additionally includes background within a four-voxel dilation of this cap when its distance from $r_{c}$ is at least the cap distance, allowing a tolerance of half the smallest voxel spacing. If tied distance values cause the distal selection to exceed 40% of the vessel, masking is skipped.

For either branch of Eq. 1, the mask is computed from the resulting target $y_{i}^{\prime}$ immediately before loss computation. Writing $U_{i}=U_{\mathrm{AO}}\cup U_{\mathrm{PA}}$ and denoting the segmentation network by $f_{\phi}$, the objective is

The operation changes which voxels contribute to the loss without generating an extended or truncated vessel label. It is applied to every training example, regardless of whether the input is real or synthetic, and is not used during validation or inference. The masking rule therefore does not explicitly favor either extension or truncation of the annotated vessel endpoint.

Each training batch first undergoes the spatial and intensity augmentations used by nnU-Net. For an example selected with probability $\rho=0.10$, the Myo–LV operation is applied to the transformed label map, and the augmented real image is replaced by an image synthesized from the resulting map. In the complete method, the AO and PA ignore regions defined in Sec. 2.3 are excluded from the Dice and cross-entropy losses immediately before loss computation. For each selected example, we independently sample the class intensity distributions, bias field, gamma transformation, noise level, and resolution profile described in Sec. 2.1, as well as whether the myocardium is thickened or thinned. At inference, the network receives one preprocessed image channel without a modality indicator or synthesis step. The same 3D full-resolution configuration is used for CT and MRI; no 2D model or slice-wise inference is used. We otherwise use the standard nnU-Net inference procedure.

The released CARE-WHS training set contains 106 labeled volumes from five site groups: 60 CT and 46 MRI scans. All images and annotations used in this study were released through the CARE 2026 Whole Heart track [3, 14, 12, 5]; no private images or annotations were used. Every model is trained with both CT and MRI cases, and the two modalities are separated only when validation results are reported. We use the same partition into five folds as the nnU-Net baseline. Tables 1 and 2 report one development split containing 85 training cases and 21 validation cases, of which 12 are CT and 9 are MRI. With the exception of the “Only synthetic” experiment listed in Table 1, the nnU-Net comparisons use the same augmentation, optimization schedule, and 200-epoch budget. The 3D U-Net, UNETR, and SwinUNETR baselines use the same development split, preprocessed inputs, and evaluation procedure.

For the model comparisons, we report the Dice similarity coefficient (DSC) separately for CT and MRI. Within each reported group, DSC is first averaged across cases for each of the seven structures. The overall WHS score is the weighted mean of these seven structure averages, using the total reference-annotation volume of each structure as its weight. Each label in each model is postprocessed by extracting its largest connected component before evaluation.

Table 1 compares four segmentation models trained on only real images and four training settings based on nnU-Net.

Among the four models trained on only real images, nnU-Net [9] achieved the highest DSC on this development split. The two Transformer models, UNETR [7] and SwinUNETR [6], did not outperform nnU-Net under the evaluated settings. These results compare the specific implementations tested here and do not establish a general advantage of one architecture type over another.

Training nnU-Net with only synthetic images performed worse than training with only real images, with the largest decrease observed on MRI ($-5.87$ percentage points). This comparison shows that synthetic images cannot replace real images under the current setting and are instead used alongside them during training. Using real and synthetic images increased MRI and overall DSC by 0.22 and 0.05 percentage points, respectively, while CT DSC decreased by 0.14 percentage points. The full LISynSeg setting, which additionally includes the two cardiac label operations, achieved the highest DSC in all three columns. These gains were obtained without making the segmentation model larger or changing the inference procedure.

Table 2 evaluates the Myo–LV boundary operation and AO/PA loss masking individually and together using the same base setting with real and synthetic images.

The Myo–LV boundary operation alone produced a small increase on CT but reduced MRI DSC by 2.40 percentage points and also lowered the overall DSC. The operation therefore produced different responses on CT and MRI. AO/PA loss masking alone remained close to the base setting across all three columns. When both operations were enabled, the model achieved the highest numerical DSC on CT, MRI, and overall. Although the differences from the base setting were small, they were positive in all three columns.

Although the quantitative results summarize overall performance, they do not fully reveal whether LISynSeg mitigates the specific segmentation discrepancies targeted by our design. Figure 4 therefore visually compares the real-image nnU-Net baseline with LISynSeg on one CT case and two MRI cases from the validation set.

For the CT Site-B case (top row), LISynSeg more closely follows the endocardial Myo–LV interface and reduces the LV under-segmentation observed in the baseline. Accordingly, the LV Dice score increases from 0.83 to 0.86, while the Myo Dice score increases from 0.75 to 0.79 for this subject. In the MRI Site-C and Site-D cases, LISynSeg produces segmentations that are more consistent with the released ground-truth annotations. Specifically, it captures less of the AO annotation in the Site-D case and a larger portion of the PA annotation in the Site-C case, thereby reducing the AO over-segmentation and PA under-segmentation observed in the baseline. In the middle row, the AO Dice score increases from 0.73 to 0.75, while in the bottom row, the PA Dice score increases from 0.67 to 0.71.

These examples localize the improvements to the anatomical structures targeted by the cardiac-specific label operations and complement the aggregate quantitative results reported in Table 2. Because the annotated endpoints of the great vessels vary across the released labels, the AO and PA panels illustrate agreement with the released annotations rather than the standardized vessel-length evaluation used by the challenge.

Our official validation submission combines nnU-Net models trained with only real images, with real and synthetic images, and with the data augmentation (DA5) setting. In the public validation leaderboard, the ensemble achieved DSCs of 0.9360 for CT, 0.8640 for MRI, and 0.9072 for the combined track. The submission ranked among the top five on the combined track at the time of writing. These results report the performance of the submitted ensemble, whereas Tables 1 and 2 isolate the contribution of the data augmentation strategy.