arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.12070cs.CVcs.CLcs.LG
收藏

深度多模态神经架构搜索

Deep Multimodal Neural Architecture Search

Hangzhou Dianzi University(杭州电子科技大学);Hefei University of Technology(合肥工业大学);The University of Sydney(悉尼大学);Huawei Cloud & AI(华为云与人工智能)

Zhou Yu, Yuhao Cui, Jun Yu, Meng Wang, Dacheng Tao, Qi Tian

摘要:

设计有效的神经网络在深度多模态学习中至关重要。现有的大多数工作集中在一个单一的任务和人工设计的神经架构,这是高度特定于任务,很难推广到不同的任务。在本文中,我们设计了一个广义的深度多模态神经架构搜索(MMnas)框架,用于各种多模态学习任务。给定多模态输入,我们首先定义一组原语操作,然后构建一个基于深度编码器-解码器的统一主干,其中每个编码器或解码器块对应于从预定义操作池中搜索的操作。在统一的主干之上,我们附加了特定任务的头来处理不同的多模态学习任务。通过使用基于梯度的NAS算法,有效地学习不同任务的最佳架构。广泛的消融研究,综合分析和比较实验结果表明,所获得的MMnasNet显着优于现有的国家的最先进的方法在三个多模态学习任务(超过五个数据集),包括视觉问答,图像-文本匹配,视觉接地。

英文摘要:

Designing effective neural networks is fundamentally important in deep multimodal learning. Most existing works focus on a single task and design neural architectures manually, which are highly task-specific and hard to generalize to different tasks. In this paper, we devise a generalized deep multimodal neural architecture search (MMnas) framework for various multimodal learning tasks. Given multimodal input, we first define a set of primitive operations, and then construct a deep encoder-decoder based unified backbone, where each encoder or decoder block corresponds to an operation searched from a predefined operation pool. On top of the unified backbone, we attach task-specific heads to tackle different multimodal learning tasks. By using a gradient-based NAS algorithm, the optimal architectures for different tasks are learned efficiently. Extensive ablation studies, comprehensive analysis, and comparative experimental results show that the obtained MMnasNet significantly outperforms existing state-of-the-art approaches across three multimodal learning tasks (over five datasets), including visual question answering, image-text matching, and visual grounding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑