arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.02252cs.CV
收藏

用于多模态多任务稠密图像预测的通道交换网络

Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction

Tsinghua University(清华大学);Renmin University of China(中国人民大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);JD.com Inc.(京东集团);JD Explore Academy(京东探索研究院)

Yikai Wang, Fuchun Sun, Wenbing Huang, Fengxiang He, Dacheng Tao

摘要:

多模态融合和多任务学习是机器学习中的两个重要课题。尽管取得了富有成效的进展,但针对这两个问题的现有方法仍然容易受到同样的挑战-任务),同时保留每种模态的特定模式(分别为任务)。此外,虽然它们实际上是密切相关的,多模态融合和多任务学习很少在同一方法框架内探索。在本文中,我们提出了自适应,无参数,更重要的是,适用于多模态和多任务密集图像预测的交换网络(CEN)。在其核心,CEN自适应地交换不同模态的子网之间的信道。具体地,信道交换过程是通过训练期间由批归一化(BN)缩放因子的大小测量的个体信道重要性来自我引导的。对于密集图像预测的应用,通过四种不同的场景测试CEN的有效性:多模态融合,循环多模态融合,多任务学习,多模式多任务学习。通过RGB-D数据进行语义分割和通过多域输入进行图像翻译的大量实验验证了CEN与最先进方法相比的有效性。详细的消融研究也已进行,这表明我们提出的每个组件的优点。我们的代码可在https://github.com/yikaiw/CEN上获得。

英文摘要:

Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge -- it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for multimodal and multitask dense image prediction. At its core, CEN adaptively exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of CEN compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrate the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑