HeteroPROMPT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework

Collaborative perception (CP) enables autonomous vehicles to improve environmental awareness by sharing perceptual information, particularly under occlusions and limited field-of-view [1]. Early CP approaches perform early fusion by transmitting raw sensor data, which can achieve strong performance but incurs substantial bandwidth cost. To reduce communication load, late fusion shares only high-level detection outputs, offering low bandwidth cost but often being limited by local prediction errors. More recent methods adopt intermediate fusion, where agents exchange intermediate features to balance bandwidth and detection accuracy. Beyond fusion strategies, prior work has investigated communication-efficient feature compression and bandwidth allocation [2, 3], robustness to latency, and communication [4, 5] and detection noise/uncertainty [6, 7]. However, performance degradation caused by feature-domain shifts among heterogeneous agents (e.g., different sensors, models, or training domains) remains comparatively underexplored, despite being critical for safe and reliable real-world CP.

Aligning heterogeneous intermediate features is critical for collaborative perception when agents employ different sensors, models, datasets, or training domains; otherwise, feature misalignment can significantly degrade fusion performance. Existing approaches generally fall into two categories. The first category reduces the domain gap by retraining core components, such as the fusion/detection modules [8, 9, 10, 11] and/or the joining agent’s encoder. For example, HEAL [12] constructs an ego-centric unified space and then adapts heterogeneous encoders to match it. Although effective, such retraining-based solutions are computationally expensive for newly emerging modalities and raise privacy concerns because they require access to the joining agent’s model parameters for optimization. The second category introduces model interpreters that align heterogeneous features while freezing the main collaborative pipeline and agent encoders [13, 14, 15]. These methods can better preserve privacy and enable faster adaptation, but often introduce additional complexity and may incur non-trivial inference latency. Recent prompt-based variants reduce this cost using lightweight prompts. For example, Faster-HEAL [16] uses low-rank prompts with a learnable aligner, while PEARL [17] performs anonymous real-time selection using modality-specific compressors/interpreters and domain-invariant feature similarity. In contrast, HeteroPROMPT performs metadata-free routing through a compact AutoEncoder (AE)-based latent space and a lightweight classifier, using a shared resizer before modality inference. This routes communicated BEV features to the corresponding prompt-conditioned FiLM/DRIFT modules without requiring proprietary metadata or modality-specific learnable compression for the routing stage.

A practical strategy for domain adaptation is to modulate intermediate activations with lightweight, conditional transformations, instead of retraining large perception backbones. Conditional normalization methods achieve this by predicting per-channel affine parameters conditioned on an external signal, e.g., (CBN) [19]. Related channel-wise recalibration mechanisms, such as squeeze-and-excitation (SE) blocks [20], gate feature channels to improve robustness and representation capacity with minimal additional parameters. These approaches demonstrate that channel-wise modulation can efficiently reshape feature distributions while keeping the main network largely unchanged. Building on this family, Feature-wise Linear Modulation (FiLM) [18] provides a general conditioning layer that applies learnable, feature-wise affine transformations to intermediate representations, making it a natural fit for parameter-efficient feature alignment. In parallel, prompt-based learning adapts models by optimizing small learnable visual prompts while freezing the backbone, as in low-rank prompting [21]. Prompting has also been explored for detection and domain shift, including visual prompt-driven domain generalization/adaptation and detection-oriented settings [22, 23, 24]. Motivated by these advances, we combine visual prompts with FiLM-style channel-wise modulation to obtain a compact interpreter for aligning heterogeneous intermediate BEV features, while keeping the main collaborative perception pipeline frozen.

The purpose of Stage 1 is to learn an ego-centric unified feature space under a homogeneous setting, where all agents share the same encoder. Let $\mathcal{A}=\{1,\dots,N\}$ denote the agent set including the ego agent $e$ and neighbors $j\in\{\mathcal{A}\backslash\{e\}\}$. Each agent $i\in\mathcal{A}$ encodes its observation $O_{i}$ into a BEV feature map $F_{i}$ using the shared encoder $f_{m_{1}(ego)}$, which would be transmitted to the ego agent through a communication operator $\Gamma_{i\rightarrow e}$. Here, modality represents the collective configuration of an agent’s sensors, encoder, and detection architecture/model. Hence, $m_{1}$ represents the ego agent modality, and $f_{m_{1}(ego)}$ is the ego agent unified feature space encoder function. The ego agent aggregates the received features via pyramid fusion to obtain the unified feature representation $H_{m_{1}}$, which is then fed into the detection head to predict final detections $B$, described in Algorithm 1.

HeteroPROMPT adopts a pyramid fusion scheme for efficient and accurate fusion, following [12]. We perform fine-to-coarse fusion by downsampling the received features using a ResNeXt [25] block to obtain $L$ scales. At each scale $l$, a foreground estimator $f_{\text{fg}}^{(l)}$ produces an occupancy map to highlight informative regions. Fusion weights are computed by applying a softmax across agents. The fused features are then upsampled to a common resolution and concatenated to produce $H_{m_{1}}$, described in Algorithm 2.

Besides the final detection loss $\mathcal{L}_{\text{det}}$, which includes focal classification loss $\mathcal{L}_{\text{focal}}$, regression loss $\mathcal{L}_{\text{smooth-L1}}$, and direction loss $\mathcal{L}_{\text{dir}}$, Stage 1 additionally supervises the foreground estimator at each pyramid level using a focal loss:

where $Y$ is the ground-truth, $B$ denotes the predictions, $OCC_{i}^{(l)}$ is the predicted occupancy map, and $Y_{i}^{(l)}$ is the foreground mask with $\alpha_{l}$ as weighting contribution of scales.

After Stage 1 learns an ego-centric unified feature space, new agents may join the collaboration group. Since the unified space is learned under the ego agent’s sensor/model configuration, a distribution mismatch in BEV features arises when a new agent employs different sensors, perception backbones, or datasets, which can significantly degrade the detection performance. Therefore, the new agent’s features must be aligned to the unified feature space to minimize performance loss. In practice, retraining the fusion and detection modules for every newly joining agent (or each new agent backbone for each base) is often infeasible. This motivates a lightweight and privacy-preserving adaptation mechanism that can efficiently align heterogeneous features under constraints such as limited computation/resources and the proprietary nature of companies’ sensors and models.

HeteroPROMPT addresses these constraints by freezing the ego fusion module and detection head, thereby preserving the Stage 1 unified feature space. We also freeze the newly joining agent’s encoder, confining adaptation to a small set of prompts operating on shared intermediate features; this reduces computational overhead and better preserves the new agent’s privacy. By significantly reducing complexity and by preserving critical aspects of new agent’s metadata (i.e., sensor type, encoder, detection head model, etc.), our approach represents a major departure from prior CP frameworks, including HEAL [12]. To align heterogeneous features to the unified space, we introduce Decomposed-Rank Interpreter for FiLM Tuning (DRIFT) at two points in the detection pipeline containing lightweight and modality-specific visual prompts, and fine-tune them. For a newly joining agent of modality $m_{2}$, we use a set of agents $\mathcal{A}_{m_{2}}$ and a frozen encoder $f^{**}_{enc_{m_{2}}}$. The alignment procedure is described in Algorithm 3:

Here, $F_{k,m_{2}}$ is the BEV feature produced by the new agent k, $F_{k,m_{2}\rightarrow e}$ denotes the communicated BEV feature after synchronization and ego-frame transformation at the current collaboration timestamp, following the benchmark CP pipeline, and $P_{m_{2},i}$ is the learnable prompt for modality $m_{2}$ at step $i$. $f_{\text{resizer}}$ maps the new agent feature to the common unified resolution (e.g., spatial resizing and lightweight channel projection). The superscripts $*$ (Stage 1) and $**$ (new agent) indicate frozen pretrained modules during Stage 2. We use Feature-wise Linear Modulation (FiLM) [18], defined as:

where $\gamma_{m,i}(\cdot)$ and $\beta_{m,i}(\cdot)$ are produced from the prompt $P_{m,i}$ for the modality $m$ at step $i$ using a lightweight MLP or CNN, and are broadcasted to match the feature dimensions.

To ensure compatibility, the first prompt $P_{m_{2},1}$ matches the shape of the communicated feature, i.e., $(C_{m_{2}},H_{m_{2}},W_{m_{2}})$, while the second prompt $P_{m_{2},2}$ matches the unified resolution after resizing, i.e., $(C,H,W)$. Directly optimizing dense prompts of size $C\times H\times W$ can introduce a large number of trainable parameters, increasing optimization difficulty and slowing convergence. To reduce the trainable parameter count, we apply PARAFAC (CP) decomposition [26] to represent each prompt with a low-rank factorization of rank $R$. Specifically, we learn three factor matrices $A\in\mathbb{R}^{R\times C}$, $B\in\mathbb{R}^{R\times H}$, and $D\in\mathbb{R}^{R\times W}$, and reconstruct the prompt as

where $P_{c,h,w}$ denotes the prompt element at channel $c$ and spatial location $(h,w)$. This decomposition reduces the number of trainable parameters from $C\times H\times W$ to $R(C+H+W)$ with $R\ll\min(C,H,W)$, lowering training cost and accelerating convergence while maintaining comparable performance.

FiLM [18] is used to modulate intermediate BEV features using the learnable prompts, thereby aligning heterogeneous features to the ego agent unified space. During Stage 2, we jointly optimize the lightweight prompt factors and FiLM in DRIFT , while keeping the new agent encoder and the ego fusion/head frozen. This design reduces the feature-domain gap without requiring access to raw sensor data or proprietary model metadata. Moreover, only the low-rank prompt factors and FiLM weights need to be stored for each modality, notably reducing storage compared to full perception stacks.

To support generalization, Stage 2 adaptation is performed offline for a set of common sensor/model configurations (referred to as modalities). The ego agent stores the corresponding prompt–FiLM pair in DRIFT for each modality. During real-time deployment, when a new agent joins, the ego must select the appropriate prompt–FiLM pair; an incorrect selection can lead to substantial performance degradation.

Many existing approaches perform this selection using sensitive sensor/model metadata, which may be unavailable due to privacy or proprietary constraints. Moreover, metadata alone can be insufficient. Even with the same sensor and backbone, a model trained on a different dataset exhibits a shifted feature distribution, causing a domain gap and degrading performance by selecting wrong DRIFT modules. Hence, we developed a real-time, privacy-preserving mechanism to detect the joining agent’s modality and select the appropriate modules without revealing sensitive metadata.

To identify the modality of a newly joining agent at runtime, HeteroPROMPT extracts modality-discriminative cues from communicated BEV features using an AutoEncoder (AE). After completing Stage 1, we introduce an intermediate Stage 1.5 to train the AE on ego domain features and construct a compact unified representation. In this stage, we keep the encoder, pyramid fusion, and detection head frozen, and train only the AE, as described in Algorithm 4.

The AutoEncoder consists of an encoder $g_{AE}$ that maps the BEV features to a compact feature space, and a decoder $d_{AE}$ that maps the compact features back to the original size. $f^{*}$ is for frozen modules. Stage 1.5 is trained using the Stage 1 objective in (1) together with a reconstruction loss:

where $\lambda$ balances detection and reconstruction. Therefore, the AE is guided by both the frozen Stage 1 detection pipeline and the reconstruction objective, encouraging the compact representation to preserve detection-relevant unified BEV features for real-time modality selection.

For a newly joining agent, its communicated BEV feature is first resized to the ego agent’s unified resolution to ensure compatibility with the AE encoder. The AE encoder then maps the resized feature into a compact latent space, which captures modality-discriminative cues for identifying the agent’s feature domain. We train a lightweight classifier on a collection of such compact latent codes obtained from a set of common modalities (sensor/model configurations).

During deployment, when a new agent joins, the ego agent applies the resizer, AE encoder, and classifier to infer the agent’s modality as:

Based on the predicted modality ($\hat{m}$), the ego selects the corresponding DRIFT  modules, which enables real-time module selection without requiring any sensitive sensor or model metadata. After choosing the appropriate DRIFTs, the ego agent performs collaborative fusion and detection.

We evaluated HeteroPROMPT on two large-scale CP datasets: OPV2V-H [12] for the main results and ablations, and V2XSet [27] for further validation of the performance of our proposed framework. OPV2V-H [12] is built upon OPV2V [28] and is designed specifically for heterogeneous CP. Different perception encoders have been used both for LiDAR (PointPillar [29] and SECOND [30]) and camera (Lift-Splat-Shoot [31] to lift image features to BEV features, with EfficientNet [32] and ResNet-50 [33] as image encoders) to evaluate the generalization capability of HeteroPROMPT under heterogeneous CP settings.

For fair comparison with prior work, we define heterogeneous modalities following HEAL [12]. Table I summarizes the four modalities used in our experiments. Modality $m_{1}$ is used as the ego agent to construct the ego-centric unified feature space with feature size $(64,128,256)$ using 64-channel LiDAR. We then progressively add three heterogeneous agents with modalities $m_{2}$, $m_{3}$, and $m_{4}$. All agents use pretrained, frozen single-agent encoders and backbones with $[0.4\,\mathrm{m},0.4\,\mathrm{m}]$ BEV grid. The detection range is $[-102.4\,\mathrm{m},+102.4\,\mathrm{m}]$ and $[-51.2\,\mathrm{m},+51.2\,\mathrm{m}]$ along $x$ and $y$.

We compare HeteroPROMPT with state-of-the-art intermediate-fusion frameworks for heterogeneous collaborative perception under progressively added heterogeneous agents. As summarized in Table II, HeteroPROMPT consistently improves AP50 over HEAL [12] on both datasets while using orders-of-magnitude fewer trainable parameters. On OPV2V-H, HeteroPROMPT outperforms HEAL by $+1.8\%$, $+1.0\%$ and $+1.2\%$ at AP50, for $+m_{2}$, $+m_{3}$, and $+m_{4}$, respectively, while maintaining a comparable AP70. Additionally, HeteroPROMPT on average improves $+4.7\%/10.4\%$ and $+5.8\%/10.8\%$ at AP50/AP70, for CoBEVT [34] and HM-ViT [9], respectively. On V2XSet, HeteroPROMPT similarly improves AP50 over HEAL by $+1.4\%$ for $+m_{2}$, $+1.5\%$ for $+m_{3}$ and $+1.1\%$ for $+m_{4}$, while maintaining a comparable AP70. Importantly, HeteroPROMPT reduces the trainable parameters to only $0.064$M across all settings, yielding $94.1$–$99.6\%$ parameter reduction relative to HEAL, dramatically lower overhead than retraining baselines (CoBEVT [34], HM-ViT [9]), supporting fast and privacy-preserving feature adaptation. We further evaluate training efficiency by comparing HeteroPROMPT with HEAL. As reported in Table III, HeteroPROMPT reduces peak GPU memory usage by $45.7\%$, while achieving end-to-end training throughput of $30.74$ samples/sec. ($2.5\times$ higher than HEAL) and $2.27\times$ higher computational throughput in TFLOPs/sec. on an NVIDIA RTX A6000 (batch size 1), demonstrating its computational efficiency.

After Stage 1, we train an AutoEncoder (AE) to construct a compact unified feature space using the frozen ego encoder, fusion, and detection modules in an intermediate training stage. The AE encoder/decoder each use three CNN layers. We evaluate four latent sizes: $(64,8,16)$, $(64,16,32)$, $(128,8,16)$, and $(128,16,32)$. For $m_{2}$, $m_{3}$, and $m_{4}$, we extract compact features using 140/30/30 train/val/test samples per modality using OPV2V-H and V2XSet datasets. To visualize the geometry of the learned latent space, we apply t-SNE [35] (perplexity 40–50) on flattened compact tensors. Fig. 2 shows clear modality-dependent clustering in the compact space.

Table IV reports the classification test loss for different latent sizes. Since $(64,8,16)$ yields the lowest loss, we use it for classification with all samples. For modality classification, we apply average pooling on the compact features followed by a three-layer MLP classifier. To evaluate scalability, we further extend the modality set with $m_{5}$ (SECOND [30]) and $m_{6}$ (PointPillar [29]) with different voxel sizes than $m_{3}$ and $m_{1}$. Fig. 3 reports the test confusion matrix over all modalities. The classifier achieves near-perfect accuracy ($>99.99\%$), with one confusion between $m_{5}$ and $m_{2}$.

We study the contribution of each DRIFT module in Stage 2 heterogeneous adaptation of HeteroPROMPT. We used a CNN-based FiLM block with PARAFAC prompts of rank $R=16$, and evaluated three variants: using only DRIFT 1 (pre-resizer), only DRIFT 2 (post-resizer in the unified resolution), or both. Table V shows that combining both DRIFT modules is important for effectively closing the feature-domain gap. Removing DRIFT 2 yields a drop of $2.9\%$ at AP50 and $3.4\%$ at AP70, while removing DRIFT 1 reduces performance by $4.6\%$/$4.8\%$ at AP50/AP70, indicating that DRIFT 1 contributes more strongly in this setting.

We also evaluated the effect of the FiLM architecture (MLP vs. CNN) and the PARAFAC prompt rank $R$ on the accuracy–complexity trade-off under the $m_{1}+m_{2}+m_{3}$ setting. Fig. 4 reports AP50 (circles) and AP70 (triangles) results versus the number of trainable parameters. Overall, CNN-based FiLM with $R=16$ (brown) provides the best AP70 performance while maintaining low complexity, and achieves competitive AP50 precision. Among the MLP architectures, a competitive option is MLP-based FiLM with $R=64$, which achieves the best AP50 performance. Increasing $R$ further increases model capacity and complexity with diminishing gains, while MLP-based FiLM generally yields lower complexity but weaker accuracy. As highlighted above, it is important to note that the HEAL AP results come at a high cost in complexity that is almost two orders of magnitude higher than HeteroPROMPT’s overall complexity. For $R=16$, varying the CNN width (squares and upside-down triangles) shows that $32$ channels provides the best trade-off, whereas smaller widths reduce capacity and larger widths increase complexity with marginal gains.