GeminiFusion：面向Vision Transformer的高效像素级多模态融合
GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer
AI总结：
本文提出像素级多模态融合方法GeminiFusion，通过层自适应噪声结合模态内与模态间注意力，在保持线性复杂度的同时实现高效且优于前沿技术的跨模态特征融合。
摘要：
跨模态transformers通过有效整合不同模态，在各种视觉任务中展现出了优越性。本文首先批判了先前的token交换方法，该方法用模态间特征替换信息量较少的token，并证明基于交换的方法表现不如cross-attention机制，而后者的计算需求不可避免地限制了其在更长序列中的使用。为克服计算挑战，我们提出了GeminiFusion，这是一种利用对齐的跨模态表示的像素级融合方法。GeminiFusion优雅地结合了模态内和模态间注意力，动态整合跨模态的互补信息。我们采用层自适应噪声在逐层基础上自适应地控制它们的相互作用，从而实现和谐的融合过程。值得注意的是，GeminiFusion保持了相对于输入token数量的线性复杂度，确保该多模态框架的运行效率可与单模态网络相媲美。在多模态图像到图像翻译、3D目标检测和任意模态语义分割任务（包括RGB、depth、LiDAR、event data等）上的全面评估，证明了我们的GeminiFusion相对于前沿技术的优越性能。PyTorch代码可在https://github.com/JiaDingCN/GeminiFusion获取
英文摘要：
Cross-modal transformers have demonstrated superiority in various vision tasks by effectively integrating different modalities. This paper first critiques prior token exchange methods which replace less informative tokens with inter-modal features, and demonstrate exchange based methods underperform cross-attention mechanisms, while the computational demand of the latter inevitably restricts its use with longer sequences. To surmount the computational challenges, we propose GeminiFusion, a pixel-wise fusion approach that capitalizes on aligned cross-modal representations. GeminiFusion elegantly combines intra-modal and inter-modal attentions, dynamically integrating complementary information across modalities. We employ a layer-adaptive noise to adaptively control their interplay on a per-layer basis, thereby achieving a harmonized fusion process. Notably, GeminiFusion maintains linear complexity with respect to the number of input tokens, ensuring this multimodal framework operates with efficiency comparable to unimodal networks. Comprehensive evaluations across multimodal image-to-image translation, 3D object detection and arbitrary-modal semantic segmentation tasks, including RGB, depth, LiDAR, event data, etc. demonstrate the superior performance of our GeminiFusion against leading-edge techniques. The PyTorch code is available at https://github.com/JiaDingCN/GeminiFusion
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
