细节中的可视化QA魔鬼:早期融合和批处理规范对CLEVR的影响
The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR
摘要:
视觉QA是高级推理的关键挑战,需要理解语言,视觉以及场景中许多对象之间的关系。虽然像CLEVR这样的数据集被设计成没有这种复杂的关系推理就无法解决,但一些令人惊讶的简单前馈,“整体”模型最近在这个数据集上表现出了强大的性能。这些模型缺乏任何一种明确的迭代,符号推理过程,这被假设为是必要的计数对象,缩小了一组相关的对象的基础上的几个属性等,这种强大的性能的原因是知之甚少。因此,我们的工作分析这样的模型,并发现小的架构元素是至关重要的性能。特别是,我们发现语言和视觉的\textit{early fusion}提供了很大的性能改进。这与Visual QA初期流行的后期融合方法形成了鲜明对比。我们提出了一个简单的模块,我们称之为多模态核心,我们假设执行多模态任务的基本操作。我们相信,了解为什么这些元素是如此重要的复杂的问答将有助于设计性能更好的算法的视觉QA,同时尽量减少手工工程的努力。
英文摘要:
Visual QA is a pivotal challenge for higher-level reasoning, requiring understanding language, vision, and relationships between many objects in a scene. Although datasets like CLEVR are designed to be unsolvable without such complex relational reasoning, some surprisingly simple feed-forward, "holistic" models have recently shown strong performance on this dataset. These models lack any kind of explicit iterative, symbolic reasoning procedure, which are hypothesized to be necessary for counting objects, narrowing down the set of relevant objects based on several attributes, etc. The reason for this strong performance is poorly understood. Hence, our work analyzes such models, and finds that minor architectural elements are crucial to performance. In particular, we find that \textit{early fusion} of language and vision provides large performance improvements. This contrasts with the late fusion approaches popular at the dawn of Visual QA. We propose a simple module we call Multimodal Core, which we hypothesize performs the fundamental operations for multimodal tasks. We believe that understanding why these elements are so important to complex question answering will aid the design of better-performing algorithms for Visual QA while minimizing hand-engineering effort.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。