基于多跳特征调制的视觉推理
Visual Reasoning with Multi-hop Feature Modulation
摘要：
计算机视觉和自然语言处理的最新突破激发了人们对具有挑战性的多模态任务（如视觉问答和视觉对话）的兴趣。对于这样的任务，一种成功的方法是通过逐行线性调制（FiLM）层将基于图像的卷积网络计算条件化到语言上，即，每通道缩放和移位。我们建议以多跳方式生成Film层的参数，而不是像以前的工作那样一次生成。通过在关注语言输入和生成电影层参数之间交替，这种方法能够更好地扩展到具有较长输入序列（例如对话）的设置。我们证明，多跳电影生成实现了最先进的短输入序列任务参考-与单跳电影生成-同时也显着优于现有的最先进的单跳电影生成的GuessWhat？！视觉对话任务。
英文摘要：
Recent breakthroughs in computer vision and natural language processing have spurred interest in challenging multi-modal tasks such as visual question-answering and visual dialogue. For such tasks, one successful approach is to condition image-based convolutional network computation on language via Feature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling and shifting. We propose to generate the parameters of FiLM layers going up the hierarchy of a convolutional network in a multi-hop fashion rather than all at once, as in prior work. By alternating between attending to the language input and generating FiLM layer parameters, this approach is better able to scale to settings with longer input sequences such as dialogue. We demonstrate that multi-hop FiLM generation achieves state-of-the-art for the short input sequence task ReferIt --- on-par with single-hop FiLM generation --- while also significantly outperforming prior state-of-the-art and single-hop FiLM generation on the GuessWhat?! visual dialogue task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
