AsymFormer：面向移动平台实时RGB-D语义分割的非对称跨模态表征学习
AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentation
AI总结：
针对室内场景实时语义分割的精度与效率平衡需求，提出采用非对称骨干、LAFS和CMA模块的AsymFormer网络，在NYUv2等数据集上取得有竞争力的精度，且在RTX3090上实现高速推理。
摘要：
理解室内场景对城市研究至关重要。考虑到室内环境的动态特性，有效的语义分割需要同时兼顾实时运行能力与高精度。为解决这一问题，我们提出AsymFormer，这是一种新型网络，它利用RGB-D多模态信息提升实时语义分割精度，且不会大幅增加网络复杂度。AsymFormer采用非对称骨干网络进行多模态特征提取，通过优化计算资源分布减少冗余参数。为融合非对称多模态特征，该网络使用局部注意力引导的特征选择（Local Attention-Guided Feature Selection，LAFS）模块，利用不同模态特征的依赖关系有选择地融合特征。随后，引入跨模态注意力引导的特征关联嵌入（Cross-Modal Attention-Guided Feature Correlation Embedding，CMA）模块，进一步提取跨模态表征。AsymFormer在NYUv2数据集上取得54.1%的mIoU，在SUNRGBD数据集上取得49.1%的mIoU，结果具有竞争力。值得注意的是，AsymFormer在RTX3090上的推理速度达到65 FPS（启用混合精度量化后为79 FPS），表明其能够在高精度与效率之间取得平衡。
英文摘要：
Understanding indoor scenes is crucial for urban studies. Considering the dynamic nature of indoor environments, effective semantic segmentation requires both real-time operation and high accuracy.To address this, we propose AsymFormer, a novel network that improves real-time semantic segmentation accuracy using RGB-D multi-modal information without substantially increasing network complexity. AsymFormer uses an asymmetrical backbone for multimodal feature extraction, reducing redundant parameters by optimizing computational resource distribution. To fuse asymmetric multimodal features, a Local Attention-Guided Feature Selection (LAFS) module is used to selectively fuse features from different modalities by leveraging their dependencies. Subsequently, a Cross-Modal Attention-Guided Feature Correlation Embedding (CMA) module is introduced to further extract cross-modal representations. The AsymFormer demonstrates competitive results with 54.1% mIoU on NYUv2 and 49.1% mIoU on SUNRGBD. Notably, AsymFormer achieves an inference speed of 65 FPS (79 FPS after implementing mixed precision quantization) on RTX3090, demonstrating that AsymFormer can strike a balance between high accuracy and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
