arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.06858cs.CV
收藏

VQA中视觉瓶颈的实验研究

An experimental study of the vision-bottleneck in VQA

Orange Innovation(奥兰治创新);LIRIS, INSA Lyon(里昂国立应用科学学院LIRIS实验室)

Pierre Marza, Corentin Kervadec, Grigory Antipov, Moez Baccouche, Christian Wolf

摘要:

在许多结合视觉和语言的任务中,视觉问答(VQA)中视觉和语言都起着至关重要的作用。为了正确地解决这个问题,一个给定的模型应该理解所提出的图像的内容和问题的本质。虽然模态之间的融合,这是问题的另一个明显重要的部分,已经得到了高度的研究,视觉部分在最近的工作中得到的关注较少。当前用于VQA的最先进的方法主要依赖于现成的对象检测器提供一组对象边界框和嵌入,然后通过推理模块将其与问题词嵌入相结合。在本文中,我们提出了一个深入的研究VQA中的视觉瓶颈,实验从图像中提取的视觉对象的数量和质量。我们还研究了两种方法的影响,将回答一个问题所需的对象的信息,在推理模块直接,并在对象选择阶段的早期。这项工作突出了视觉的VQA的背景下,和定制的视觉方法在VQA手头的任务的重要性。

英文摘要:

As in many tasks combining vision and language, both modalities play a crucial role in Visual Question Answering (VQA). To properly solve the task, a given model should both understand the content of the proposed image and the nature of the question. While the fusion between modalities, which is another obviously important part of the problem, has been highly studied, the vision part has received less attention in recent work. Current state-of-the-art methods for VQA mainly rely on off-the-shelf object detectors delivering a set of object bounding boxes and embeddings, which are then combined with question word embeddings through a reasoning module. In this paper, we propose an in-depth study of the vision-bottleneck in VQA, experimenting with both the quantity and quality of visual objects extracted from images. We also study the impact of two methods to incorporate the information about objects necessary for answering a question, in the reasoning module directly, and earlier in the object selection stage. This work highlights the importance of vision in the context of VQA, and the interest of tailoring vision methods used in VQA to the task at hand.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑