arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.08148cs.LG
收藏

开放集基于知识的视觉问答与推理路径

Open-Set Knowledge-Based Visual Question Answering with Inference Paths

Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(中国科学院计算技术研究所智能信息处理重点实验室);University of Chinese Academy of Sciences(中国科学院大学)

Jingru Gan, Xinzhe Han, Shuhui Wang, Qingming Huang

AI总结:

针对开放集基于知识的视觉问答,提出GATHER检索器-排序器框架,通过图构建、剪枝和路径级排序,实现开放集问答并提供可解释推理路径,在OK-VQA重构的ConceptVQA上验证有效性。

摘要:

给定一张图像和一个相关的文本问题,基于知识的视觉问答(KB-VQA)旨在借助外部知识库为问题提供正确答案。先前的KB-VQA模型通常被构建为检索器-分类器框架,其中预训练的检索器从知识图谱中提取文本或视觉信息,然后在候选答案中进行预测。尽管取得了有希望的进展,现有模型存在两个缺点。首先,将问答建模为多类分类限制了答案空间为预设语料库,缺乏灵活推理的能力。其次,分类器仅考虑“答案是什么”而不考虑“如何得到答案”,无法将答案锚定到明确的推理路径上。在本文中,我们直面可解释的开放集KB-VQA的挑战,其中系统需要在开放实体条件下回答问题并保留可解释的推理路径。为解决上述问题,我们提出了一种新的KB-VQA检索器-排序器范式,即图路径排序器(简称GATHER)。具体而言,它包含图构建、剪枝和路径级排序,不仅检索准确的答案,还提供解释推理过程的推理路径。为全面评估我们的模型,我们重新构建了基准数据集OK-VQA,并手动修正了实体级标注,将其发布为ConceptVQA。在真实世界问题上的大量实验表明,我们的框架不仅能够在整个知识库上进行开放集问答,还能提供明确的推理路径。

英文摘要:

Given an image and an associated textual question, the purpose of Knowledge-Based Visual Question Answering (KB-VQA) is to provide a correct answer to the question with the aid of external knowledge bases. Prior KB-VQA models are usually formulated as a retriever-classifier framework, where a pre-trained retriever extracts textual or visual information from knowledge graphs and then makes a prediction among the candidates. Despite promising progress, there are two drawbacks with existing models. Firstly, modeling question-answering as multi-class classification limits the answer space to a preset corpus and lacks the ability of flexible reasoning. Secondly, the classifier merely consider "what is the answer" without "how to get the answer", which cannot ground the answer to explicit reasoning paths. In this paper, we confront the challenge of \emph{explainable open-set} KB-VQA, where the system is required to answer questions with entities at wild and retain an explainable reasoning path. To resolve the aforementioned issues, we propose a new retriever-ranker paradigm of KB-VQA, Graph pATH rankER (GATHER for brevity). Specifically, it contains graph constructing, pruning, and path-level ranking, which not only retrieves accurate answers but also provides inference paths that explain the reasoning process. To comprehensively evaluate our model, we reformulate the benchmark dataset OK-VQA with manually corrected entity-level annotations and release it as ConceptVQA. Extensive experiments on real-world questions demonstrate that our framework is not only able to perform open-set question answering across the whole knowledge base but provide explicit reasoning path.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑