arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2011.05005cs.CVcs.LG
收藏

基于信道交换的深度多模态融合

Deep Multimodal Fusion by Channel Exchanging

Beijing National Research Center for Information Science and Technology (BNRist)(北京信息科学与技术国家研究中心(BNRist));Tsinghua University(清华大学);Tencent AI Lab(腾讯AI实验室)

Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, Junzhou Huang

摘要:

通过使用多个数据源进行分类或回归的深度多模态融合在各种应用中表现出明显优于单峰对应物的优势。然而,目前的方法,包括聚合为基础的融合和基于融合的模态间融合和模态内处理之间的平衡仍然不足,招致性能提高的瓶颈。为此,本文提出了一种无参数的多模态融合框架--通道交换网络(CEN),它可以在不同模态的子网络之间动态地交换通道。具体地,信道交换过程是通过训练期间由批归一化(BN)缩放因子的大小测量的个体信道重要性来自我引导的。这种交换过程的有效性也通过共享卷积滤波器来保证,但在模态之间保持单独的BN层,这作为一个附加的好处,使我们的多模态架构几乎与单峰网络一样紧凑。通过RGB-D数据进行语义分割和通过多域输入进行图像翻译的广泛实验验证了我们的CEN与当前最先进方法相比的有效性。也进行了详细的消融研究,这证明了我们提出的每个组件的优势。我们的代码可在https://github.com/yikaiw/CEN上获得。

英文摘要:

Deep multimodal fusion by using multiple sources of data for classification or regression has exhibited a clear advantage over the unimodal counterpart on various applications. Yet, current methods including aggregation-based and alignment-based fusion are still inadequate in balancing the trade-off between inter-modal fusion and intra-modal processing, incurring a bottleneck of performance improvement. To this end, this paper proposes Channel-Exchanging-Network (CEN), a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of our CEN compared to current state-of-the-art methods. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑