使用多模态球面图像的单帧语义分割
Single Frame Semantic Segmentation Using Multi-Modal Spherical Images
AI总结：
针对全景多模态语义分割，本文提出基于transformer的跨模态融合架构，结合畸变感知模块与融合前跨模态交互，在三个室内全景数据集上取得最先进mIoU。
摘要：
近年来，研究界对提供360度方向视角的全景图像表现出浓厚兴趣。为充分发挥其潜力，可以输入多种数据模态，并利用互补特性，基于语义分割实现更鲁棒、更丰富的场景解释。然而，现有研究主要集中于针孔RGB-X语义分割。在本研究中，我们提出一种基于transformer的跨模态融合架构，以弥合多模态融合与全向场景感知之间的差距。我们采用畸变感知模块，以解决由等距柱状表示导致的极端物体形变和全景畸变。此外，在融合特征之前，我们进行跨模态交互以实现特征校正和信息交换，从而为双模态和三模态特征流传递长程上下文。在三个室内全景视图数据集上，使用四种不同模态类型的组合进行的全面测试中，我们的方法取得了最先进的mIoU性能：在Stanford2D3DS（RGB-HHA）上为60.60%，在Structured3D（RGB-D-N）上为71.97%，在Matterport3D（RGB-D）上为35.92%。我们计划很快发布所有代码和训练好的模型。
英文摘要：
In recent years, the research community has shown a lot of interest to panoramic images that offer a 360-degree directional perspective. Multiple data modalities can be fed, and complimentary characteristics can be utilized for more robust and rich scene interpretation based on semantic segmentation, to fully realize the potential. Existing research, however, mostly concentrated on pinhole RGB-X semantic segmentation. In this study, we propose a transformer-based cross-modal fusion architecture to bridge the gap between multi-modal fusion and omnidirectional scene perception. We employ distortion-aware modules to address extreme object deformations and panorama distortions that result from equirectangular representation. Additionally, we conduct cross-modal interactions for feature rectification and information exchange before merging the features in order to communicate long-range contexts for bi-modal and tri-modal feature streams. In thorough tests using combinations of four different modality types in three indoor panoramic-view datasets, our technique achieved state-of-the-art mIoU performance: 60.60% on Stanford2D3DS (RGB-HHA), 71.97% Structured3D (RGB-D-N), and 35.92% Matterport3D (RGB-D). We plan to release all codes and trained models soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
