arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.07192cs.CV
收藏

Knowledge-Routed Visual Question Reasoning:深度表示嵌入的挑战

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding

Sun Yat-sen University(中山大学);DarkMatter AI Research(DarkMatter AI研究院);University of California, Los Angeles(加州大学洛杉矶分校)

Qingxing Cao, Bailin Li, Xiaodan Liang, Keze Wang, Liang Lin

摘要:

虽然有利于鼓励视觉问答(VQA)模型通过利用图像和文本上下文之外的输入-输出相关性来发现底层知识,但现有的知识VQA数据集大多以众包方式进行注释,例如,通过互联网收集来自不同用户的问题和外部原因。除了知识推理的挑战之外,如何处理注释者偏见也仍然没有解决,这往往导致问题和答案之间的表面过拟合相关性。为了解决这个问题,我们提出了一个新的数据集命名为知识路由视觉问题推理VQA模型评估。考虑到理想的VQA模型应该正确感知图像上下文,理解问题并融入其学习的知识,我们提出的数据集旨在切断当前深度嵌入模型所利用的捷径学习,并推动基于知识的视觉问题推理的研究边界。具体来说,我们生成的问题-答案对的基础上的视觉基因组场景图和外部知识库与控制程序解开其他偏见的知识。该程序可以从场景图或知识库中选择一个或两个三元组,以推动多步推理,避免答案歧义,并平衡答案分布。与现有的VQA数据集相比,我们进一步暗示了以下两个主要的约束程序,以纳入知识推理:i)多个知识三元组可以与问题相关,但只有一个知识与图像对象相关。这可以强制VQA模型正确地感知图像,而不是仅仅基于给定的问题猜测知识; ii)所有问题都基于不同的知识,但是候选答案对于训练集和测试集都是相同的。

英文摘要:

Though beneficial for encouraging the Visual Question Answering (VQA) models to discover the underlying knowledge by exploiting the input-output correlation beyond image and text contexts, the existing knowledge VQA datasets are mostly annotated in a crowdsource way, e.g., collecting questions and external reasons from different users via the internet. In addition to the challenge of knowledge reasoning, how to deal with the annotator bias also remains unsolved, which often leads to superficial over-fitted correlations between questions and answers. To address this issue, we propose a novel dataset named Knowledge-Routed Visual Question Reasoning for VQA model evaluation. Considering that a desirable VQA model should correctly perceive the image context, understand the question, and incorporate its learned knowledge, our proposed dataset aims to cutoff the shortcut learning exploited by the current deep embedding models and push the research boundary of the knowledge-based visual question reasoning. Specifically, we generate the question-answer pair based on both the Visual Genome scene graph and an external knowledge base with controlled programs to disentangle the knowledge from other biases. The programs can select one or two triplets from the scene graph or knowledge base to push multi-step reasoning, avoid answer ambiguity, and balanced the answer distribution. In contrast to the existing VQA datasets, we further imply the following two major constraints on the programs to incorporate knowledge reasoning: i) multiple knowledge triplets can be related to the question, but only one knowledge relates to the image object. This can enforce the VQA model to correctly perceive the image instead of guessing the knowledge based on the given question solely; ii) all questions are based on different knowledge, but the candidate answers are the same for both the training and test sets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑