arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1809.01943cs.IRcs.AIcs.CLcs.CV
收藏

用于视觉推理的级联互调制

Cascaded Mutual Modulation for Visual Reasoning

Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Research Center for Brain-inspired Intelligence, CASIA(中国科学院自动化研究所类脑智能研究中心);University of Chinese Academy of Sciences(中国科学院大学);Center for Excellence in Brain Science and Intelligence Technology, CAS(中国科学院脑科学与智能技术卓越创新中心)

Yiqun Yao, Jiaming Xu, Feng Wang, Bo Xu

摘要:

视觉推理是一种特殊的视觉问答问题,其本质上是多步骤和组合的,并且还需要密集的文本-视觉交互。我们提出CMM:级联互调制作为一种新的端到端的视觉推理模型。CMM包括对问题和图像的多步骤理解过程。在每一步中,我们使用了一个逐行线性调制(Film)技术,使文本/视觉管道相互控制。实验表明,CMM显着优于大多数相关的模型,并达到国家的最先进的视觉推理基准:CLEVR和NLVR,从合成和自然语言收集。消融研究证实,我们的多步骤框架和我们的视觉引导的语言调节是至关重要的任务。我们的代码可在https://github.com/FlamingHorizon/CMM-VR上获得。

英文摘要:

Visual reasoning is a special visual question answering problem that is multi-step and compositional by nature, and also requires intensive text-vision interactions. We propose CMM: Cascaded Mutual Modulation as a novel end-to-end visual reasoning model. CMM includes a multi-step comprehension process for both question and image. In each step, we use a Feature-wise Linear Modulation (FiLM) technique to enable textual/visual pipeline to mutually control each other. Experiments show that CMM significantly outperforms most related models, and reach state-of-the-arts on two visual reasoning benchmarks: CLEVR and NLVR, collected from both synthetic and natural languages. Ablation studies confirm that both our multistep framework and our visual-guided language modulation are critical to the task. Our code is available at https://github.com/FlamingHorizon/CMM-VR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑