异构协作感知中的模态隔离问题
Linking Modality Isolation in Heterogeneous Collaborative Perception
AI总结：
CodeAlign通过跨模态特征-代码-特征翻译有效解决异构协作感知中的模态隔离问题，显著降低训练参数和通信负载，提升感知性能。
摘要：
协作感知利用多个智能体之间的数据交换来增强整体感知能力。然而，智能体之间的异构性引入了领域差距，阻碍了协作，并且这一问题进一步被一个未被充分研究的问题所加剧：模态隔离。当具有不同模态的多个智能体在任何训练数据帧中都不同时出现时，就会出现模态隔离，扩大跨模态领域差距。现有对齐方法依赖于空间重叠观察的监督，因此无法处理模态隔离。为了解决这一挑战，我们提出了CodeAlign，这是首个高效的、无需共现的对齐框架，通过跨模态特征-代码-特征（FCF）翻译来平滑对齐模态。关键思想是通过代码本显式地识别表示一致性，并直接学习不同模态特征空间之间的映射，从而消除对空间对应关系的依赖。代码本将特征空间规范到代码空间，提供紧凑且具有表达力的表示。通过为每个模态准备一个代码空间，CodeAlign学习FCF翻译，将特征映射到其他模态的相应代码，然后在目标代码空间中解码回特征，从而实现有效的对齐。实验显示，当整合三种模态时，CodeAlign仅需先前对齐方法训练参数的8%，减少通信负载1024倍，并在OPV2V和DAIR-V2X数据集上实现了最先进的感知性能。代码将在https://github.com/cxliu0314/CodeAlign上发布。
英文摘要：
Collaborative perception leverages data exchange among multiple agents to enhance overall perception capabilities. However, heterogeneity across agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X dataset. Code will be released on https://github.com/cxliu0314/CodeAlign.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
