NeoRed: A Knowledge-Logic-Alignment MLLM for Neonatal Respiratory Disease Diagnosis

General MLLMs have advanced cross-modal representation learning and vision–language alignment (Bai et al. 2025b; Glm et al. 2024; Chen et al. 2024b; Bai et al. 2023; Liu et al. 2024b). Early models, including Flamingo (Alayrac et al. 2022) and the BLIP series (Li et al. 2022; Dai et al. 2023; Li et al. 2023b), established strong multimodal representations through large-scale image–text pretraining. Subsequent models such as LLaVA (Liu et al. 2024a), Qwen-VL (Bai et al. 2025b; Bai et al. 2025a), and InternVL (Chen et al. 2024b) project visual features into the language space and adopt instruction tuning for multimodal interaction. LLaVA-NeXT (Liu et al. 2024b) and LLaVA-OneVision (Li et al. 2024) further improve visual alignment and reasoning through enhanced visual encoding and multi-stage training. However, their limited domain-specific medical knowledge constrains their reliability in medical diagnosis.

Recent studies have extended MLLMs to radiology report generation and medical visual question answering (Chen et al. 2024a; Li et al. 2023a; Chaves et al. 2024; Guo et al. 2024; Xu et al. 2025; Huang et al. 2025). General medical models, including BiomedGPT (Zhang et al. 2024), LLaVA-Med (Li et al. 2023a), UMIT (Yu et al. 2025b), HuatuoGPT-Vision (Chen et al. 2024a), and Lingshu (Xu et al. 2025), use large-scale multi-stage training to enhance multimodal alignment and diagnostic reasoning. Task-specific models such as LLaVA-Ultra (Guo et al. 2024), LLaVA-Rad (Chaves et al. 2024), and RadFM (Wu et al. 2025) further improve radiological understanding across targeted 2D and 3D settings. However, most existing medical MLLMs are trained primarily on adult data and remain limited in modeling neonatal-specific disease patterns and diagnostic processes.

To support precise diagnosis of neonatal respiratory diseases, we construct NeoCXR and NeoCXR-EV datasets, which are collected from two independent hospitals. We designed a heterogeneous data pipeline to handle the distinct data from each hospital, as shown in Fig. 2. Hospital A provided ready-to-use Anteroposterior (AP) CXRs and metadata, from which we directly extract the relevant data. While hospital B provided raw data, including multi-view CXRs and PDF-formatted clinical records.

We first employed Zhipu GLM-4-Flash (Glm et al. 2024) to automatically identify AP views from multi-view images. Text was first extracted from the PDF documents using PaddleOCR (Cui et al. 2025), followed by extraction of structured metadata. Then data from both hospitals were structured and translated into English by Tencent Cloud API. To ensure data quality, we performed iterative quality control. In each round, a new random 20% subset of each dataset was independently reviewed by two neonatologists for AP-view identification, OCR, translation, and disease annotation. Errors were corrected based on expert feedback, and the process was repeated until no further errors were identified.

We finally obtain two datasets. NeoCXR, collected from Hospital A, contains 6,278 samples from 2,466 patients and is split at the patient level into training, validation, and internal test sets with a ratio of 7:1:2. NeoCXR-EV, collected from Hospital B, includes 1,089 samples from 590 patients and serves as an external validation set. Details are summarized in Tab. 1. Both datasets were approved by the relevant institutional ethics committees, and all patient data were de-identified before analysis. Each sample in the datasets is a multimodal tuple: the inputs consist of a chest X-ray paired with 21 clinical factors, while the outputs are structured reports comprising both imaging conclusion and disease diagnosis. These datasets cover 7 neonatal respiratory diseases (NRDS, neonatal pneumonia, TTN, pneumothorax, BPD, pleural effusion, atelectasis) and a normal category. Disease distribution is provided in supplementary material.

To better model the clinical diagnostic logic, we structure complex clinical factors into three categories, according to the causal progression of neonatal diseases and the diagnostic logic of neonatologists. The factors are organized as follows:

1) Developmental factors, which capture neonatal maturity and growth, including gestational age, birth weight, body length, head circumference, multiple pregnancy, fetal growth restriction (FGR) or small for gestational age (SGA).

2) Perinatal risks, which capture maternal and obstetric conditions that may adversely affect the neonate, including delivery mode, premature rupture of membranes (PROM), preeclampsia, gestational hypertension, gestational diabetes, antenatal dexamethasone and magnesium sulfate.

3) Physiological status, which represents the newborn’s immediate postnatal condition, encompassing Apgar scores, body temperature, respiration rate, pulse, blood gas, and blood glucose. The distribution of clinical context on both datasets is shown in Fig. 3. Clinical factors are prevalent in both datasets, while notable distributional differences between NeoCXR and NeoCXR-EV.

To provide the model with explicit boundaries across clinical categories, each category is enclosed within relevant tokens: "<dev>…</dev>", "<peri>…</peri>", and "<phys>…</phys>". Missing categories are replaced with "not provided" for input consistency.

As illustrated in Fig. 4, NeoRed takes a neonatal CXR and clinical context as input, and generates a structured report comprising an imaging conclusion and a disease diagnosis.

Formally, the input CXR is denoted as $\mathbf{x}^{\text{cxr}}\in\mathbb{R}^{C\times H\times W}$ and the structured clinical context is denoted as $\mathbf{x}^{\text{txt}}$. Vision encoder $\mathcal{E}_{v}$ and a projector $\mathcal{P}$ extract and map visual features into a sequence of visual tokens $\mathbf{V}=\{v_{1},v_{2},\ldots,v_{N_{v}}\}$:

The $\mathbf{x}^{\text{txt}}$ is tokenized into a sequence of textual tokens $\mathbf{T}=\{t_{1},t_{2},\ldots,t_{N_{t}}\}$. The two token sequences are concatenated into a joint multimodal input $\mathbf{U}=[\mathbf{V},\mathbf{T}]$, which is fed into the LLM backbone to autoregressively generate target report $\mathcal{R}=\{r_{1},r_{2},\ldots,r_{K}\}$. K denotes the length of generated report in tokens. The model is optimized by minimizing the autoregressive cross-entropy loss over the ground-truth report $\mathcal{R}^{*}=\{r_{1}^{*},r_{2}^{*},\ldots,r_{K}^{*}\}$:

where $r_{<i}=\{r_{1},\ldots,r_{i-1}\}$ represents all previously generated tokens before step $i$, $\Theta$ denotes the learnable parameters.

To emulate the diagnostic logic of experienced neonatologists and improve joint multimodal diagnosis, we design a novel KLA framework comprising KPI, DLC and VSA.

Neonatal respiratory disease diagnosis relies on both clinical context and CXRs, yet existing methods overlook the exploration of disease-specific modality dependencies and fail to effectively leverage this multidimensional information. To address this issue, we propose KPI to inject neonatologist-inspired priors into multimodal representations, implicitly guiding disease-specific attention of MLLM.

Formally, for each modality $m\in\mathcal{M}=\{\text{dev, peri, phys, cxr}\}$, we first extract the input embeddings $\boldsymbol{h}^{m}\in\mathbb{R}^{B\times L\times D}$ from the corresponding developmental, perinatal, physiological, and image tokens.

Here, $L$ represents sequence length (i.e., number of tokens for each modality), and $D$ is embedding dimension. After applying average pooling along token dimension, we obtain modality-specific features $\bar{\boldsymbol{h}}^{m}\in\mathbb{R}^{B\times D}$. These features are then stacked to form a unified multimodal representation $\bar{\boldsymbol{H}}\in\mathbb{R}^{B\times|\mathcal{M}|\times D}$. To model dependencies between diseases and modalities, we introduce a learnable disease-specific prior matrix $\boldsymbol{W}_{prior}\in\mathbb{R}^{N\times|\mathcal{M}|}$, initialized using neonatologist-inspired priors (see Tab. 2). The prior matrix $\boldsymbol{W}_{prior}$ is normalized via softmax and then used to weight modality-specific features $\bar{\boldsymbol{H}}$, yielding disease-specific multimodal representations $\boldsymbol{F}^{diag}\in\mathbb{R}^{B\times N\times D}$, where $N$ denotes the number of disease categories.

Then, $\mathbf{F}^{diag}$, $\bar{\mathbf{h}}^{cxr}$, and $\boldsymbol{F}^{cli}$ are passed through a dedicated classifier and supervised with binary cross-entropy (BCE) loss against disease labels, independently. This process yields prior classification loss $\mathcal{L}_{\text{prior}}$, CXR classification loss $\mathcal{L}_{\text{cxr}}$, and clinical classification loss $\mathcal{L}_{\text{cli}}$. The total optimization objective of KPI is formulated as:

where $\lambda_{1}=\lambda_{2}=0.1$ are weighting coefficients.

Existing MLLMs generate reports in an autoregressive manner without explicit constraints ensuring that the generated text remains consistent with the underlying diagnostic decision. This may result in linguistically fluent yet diagnostically inconsistent reports. To address this issue, we propose DLC to ensure semantic consistency between generated text and clinical diagnostic logic.

To address this issue, we seek a global representation to regulate autoregressive generation. The BOS hidden state naturally serves this role in causal LLMs, as all subsequent tokens can access it through self-attention. Our attention analysis further shows strong token-level dependency on BOS, motivating us to inject diagnostic supervision into BOS and transform it into a global diagnostic anchor for guiding report generation. Specifically, we impose diagnostic supervision on BOS token and align its diagnostic distribution with multimodal diagnostic representations. We adopt multimodally fused features as local diagnostic representations, following the same formulation as $\boldsymbol{F}^{diag}$ in KPI (as defined in Eq. 4), while features are derived from the last-layer hidden states of LLM. The hidden state of <BOS> token and $\boldsymbol{F}^{diag}$ are passed through dedicated classifiers and supervised with BCE loss, yielding global semantic loss $\mathcal{L}_{\text{gl}}$ and local classification loss $\mathcal{L}_{\text{lc}}$. To enforce semantic consistency between report generation and diagnostic logic, we compute Jensen-Shannon divergence between the two predictive distributions of global and local views as diagnostic consistency loss ${\mathcal{L}_{\text{dc}}}$.

where $\boldsymbol{p}$ and $\boldsymbol{q}$ denote the predicted probabilities from global and local views, and $\boldsymbol{m}$ denotes their mean.

The optimization objective of DLC is defined as:

where weighting coefficients $\alpha=1.0$, $\beta=0.5$, $\gamma=0.1$.

To further align visual evidence with its clinical textual interpretation, we design VSA, which establishes bidirectional semantic correspondence between image features and imaging conclusions, encouraging the generated imaging conclusions to be supported by visual evidence.

We extract the last-layer hidden states of image tokens by the LLM, denoted as $\boldsymbol{h}^{cxr}\in\mathbb{R}^{B\times L\times D}$, and apply average pooling along the token dimension to obtain image representation $\bar{\boldsymbol{h}}^{cxr}$. We extract the textual representation of imaging conclusion from the ground-truth report $R^{*}=\{r^{*}_{1},r^{*}_{2},\dots,r^{*}_{K}\}$ by applying a binary mask over token-level hidden states, where tokens in the imaging conclusion field are set to 1 and others to 0. Textual representation $\bar{\boldsymbol{h}}^{con}$ is obtained via average pooling over masked tokens. We employ a bidirectional contrastive objective to align image and text representations:

where $s_{ij}$ denotes the cosine similarity, $\tau$ is set to $0.07$.

The total training objective $\mathcal{L}_{\text{total}}$ is formulated as a weighted sum of language modeling loss and three auxiliary losses:

where $w_{\text{lm}}=1.0$ and $w_{\text{kpi}}$ = $w_{\text{dlc}}$ = $w_{\text{vsa}}$ = $0.5$.

We evaluate our model on two benchmarks. 1) The neonatal benchmark comprises internal test set of NeoCXR (1434 samples) and the full NeoCXR-EV (1089 samples), as we mentioned in Tab. 1. 2) The adult benchmarksinclude the official test sets of MIMIC-CXR (Johnson et al. 2019) and IU-Xray (Demner-Fushman et al. 2015). After removing samples with empty findings or impression sections, 2,737 and 3,193 samples remain for MIMIC-CXR and IU-Xray. We evaluate generated reports using natural language generation (NLG) metrics, including ROUGE-L (Lin 2004), BLEU-1 (Papineni et al. 2002), METEOR (Banerjee and Lavie 2005), RaTE (Zhao et al. 2024), to assess linguistic quality and medical factuality. For neonatal benchmarks, we evaluate diagnostic consistency using an unified label extraction protocol, reporting micro precision, recall, and F1 scores. Details and validation are provided in supplementary material. For adult benchmarks, we follow previous report generation studies (Xu et al. 2025; Wang et al. 2026) and report standard NLG together with RaTE for factuality evaluation. Implementation details provided in supplementary material.

We compare NeoRed against 8 state-of-the-art (SOTA) MLLMs across two domains. 1) Generalist models: LLaVA-NeXT-7B (Liu et al. 2024b), Qwen2.5-VL-7B (Bai et al. 2025b), InternVL-2.5-8B (Chen et al. 2024b), and Qwen3-VL-8B (Bai et al. 2025a). 2) Medical models: LLaVA-Med-7B (Li et al. 2023a), LLaVA-Rad-7B (Chaves et al. 2024), HuatuoGPT-V-7B (Chen et al. 2024a), Lingshu-7B (Xu et al. 2025).

Following the common practice in prior works (Xu et al. 2025; Deria et al. 2026; Li et al. 2025), we evaluate existing models in a zero-shot setting by directly using their publicly released weights under a unified prompt, without any additional fine-tuning. Detailed prompt templates are provided in supplementary material.

1) Performance on neonatal benchmark. Tab. 3 reports results on the in-domain NeoCXR test set and the external NeoCXR-EV set. On NeoCXR, NeoRed substantially outperforms both the best generalist model, Qwen3-VL-8B (24.25% F1), and the strongest medical model, HuatuoGPT-V-7B (9.73% F1). NeoRed also remains the best-performing model on NeoCXR-EV, demonstrating generalization under substantial disease distribution shift (see supplementary material). All pair-wise comparisons of the average performance achieve statistical significance ($p<0.05$).

To demonstrate the contribution of neonatal-domain adaptation, we fine-tune Qwen3-VL and LLaVA-Rad under the same training settings as NeoRed. Fine-tuning on NeoCXR substantially enhances their performance, achieving average gains of 35.12% and 39.45% on NeoCXR, and 6.45% and 21.12% on NeoCXR-EV for Qwen3-VL-8B and LLaVA-Rad-7B, respectively, highlighting the importance of neonatal-specific data adaptation. Furthermore, equipped with the proposed KLA framework, NeoRed consistently outperforms fine-tuned LLaVA-Rad on NeoCXR by 2.97% and Qwen3-VL on NeoCXR-EV by 4.52%.

2) Generalization to adult benchmark. Tab. 7 evaluates NeoRed on adult benchmarksMIMIC-CXR and IU-Xray. On MIMIC-CXR, NeoRed achieves an average score of 26.73%, second only to LLaVA-Rad-7B, which is specifically optimized for adult chest X-rays. On IU-Xray, NeoRed achieves an average score of 33.01%, second only to the best model, Lingshu. These results indicate that NeoRed retains generalization on adult report generation.

See comparative case studies in supplementary material.

To comprehensively evaluate the effectiveness of each module in the proposed KLA framework, we conduct extensive ablation studies on NeoCXR dataset as follows:

1) Ablation of KLA framework. Our baseline is LLaVA-Rad-7B fine-tuned on NeoCXR without KLA. We adopt a leave-one-out ablation strategy to isolate each module’s contribution, while comparison with baseline demonstrates their joint effectiveness. As shown in Tab. 4, removing KPI or DLC module causes a larger drop in CE metrics, confirming the importance of knowledge prior injection and diagnostic consistency constraint in neonatal disease diagnosis. Removing VSA primarily hurts NLG performance, confirming significance of vision alignment.

2) Internal Ablation of KPI and DLC. Tab. 5 evaluates the three KPI components. Removing $\mathcal{L}_{\text{prior}}$ degrades both NLG and CE performance, confirming the importance of prior knowledge injection. Excluding $\mathcal{L}*\text{cli}$ or $\mathcal{L}*\text{cxr}$ also degrades performance, confirming the benefit of modality-specific supervision. Tab. 6 analyzes the three DLC components. Removing $\mathcal{L}_{\text{gl}}$ consistently harms both NLG and CE metrics, while excluding $\mathcal{L}_{\text{dc}}$ or $\mathcal{L}_{\text{lc}}$ also causes performance drops, with $\mathcal{L}_{\text{lc}}$ having the largest overall impact.

3) Ablation of priors. The prior matrix is constructed based on disease–modality relevance determined through expert consensus between two neonatologists, with any disagreements resolved through discussion until consensus is reached. Comparison results of random, all-one, all-zero initializations confirm effectiveness of our priors (see Tab. 8).

4) Attention analysis of generated tokens. Following (Lin et al. 2025), we sample 100 NeoCXR cases and compute the attention ratios of generated tokens to image and clinical tokens across decoder layers, averaged over heads and tokens. During the learning (layers 5–10) and diagnostic stages (layers 25–31), NeoRed achieves more balanced multimodal fusion than the vision-dominated LLaVA-Rad (see Fig. 5), providing visual evidence for the effectiveness of our proposed KLA framework. Detailed token-level analysis is provided in supplementary material.

5) Sensitivity analysis of loss weight. Following prior work (Liu et al. 2025), we adopt commonly used auxiliary-loss weights and validate them through sensitivity analysis. A representative analysis for weighting coefficient in overall objective ($w_{\text{kpi}}=w_{\text{dlc}}=w_{\text{vsa}}$=0.5) is shown in Tab. 9.

6) Ablation of clinical context. As shown in Fig. 6, removing all clinical context produces the worst results, demonstrating the necessity of clinical context. Removing any clinical category degrades performance, confirming contribution of each. Developmental factors have the greatest impact on CE metrics, followed by perinatal risks. A diagnostic case with and without clinical text is shown in Fig. 7. Image-only input leads to a pneumonia misdiagnosis, while incorporating clinical context enables correct NRDS diagnosis.