arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1705.00601cs.CVcs.CL
收藏

预设的承诺:利用视觉问答中的问题前提

The Promise of Premise: Harnessing Question Premises in Visual Question Answering

Virginia Tech(弗吉尼亚理工大学);Georgia Institute of Technology(佐治亚理工学院)

Aroma Mahendru, Viraj Prabhu, Akrit Mohapatra, Dhruv Batra, Stefan Lee

摘要:

在本文中,我们做了一个简单的观察,关于图像的问题往往包含前提-对象和问题所暗示的关系-和推理的前提可以帮助视觉问答(VQA)模型更智能地响应无关或以前看不见的问题。当提出与图像无关的问题时,最先进的VQA模型仍然会纯粹基于学习的语言偏见来回答,导致无意义甚至误导性的答案。我们注意到,如果一个视觉问题的前提中至少有一个是假的(即没有在图像中描述),那么它与图像无关。我们利用这一观察,通过搜索错误的前提来构建问题相关性预测和解释(QRPE)的数据集。我们训练新的问题相关性检测模型,并表明,模型的原因前提始终优于模型,不。我们还发现,在训练过程中强制标准VQA模型对前提进行推理,可以改善需要组合推理的任务。

英文摘要:

In this paper, we make a simple observation that questions about images often contain premises - objects and relationships implied by the question - and that reasoning about premises can help Visual Question Answering (VQA) models respond more intelligently to irrelevant or previously unseen questions. When presented with a question that is irrelevant to an image, state-of-the-art VQA models will still answer purely based on learned language biases, resulting in non-sensical or even misleading answers. We note that a visual question is irrelevant to an image if at least one of its premises is false (i.e. not depicted in the image). We leverage this observation to construct a dataset for Question Relevance Prediction and Explanation (QRPE) by searching for false premises. We train novel question relevance detection models and show that models that reason about premises consistently outperform models that do not. We also find that forcing standard VQA models to reason about premises during training can lead to improvements on tasks requiring compositional reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑