arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.15484cs.CLcs.LG
收藏

眼见为实!基于知识图谱嵌入的事实可视化问答

Seeing is Knowing! Fact-based Visual Question Answering using Knowledge Graph Embeddings

University of Illinois, Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Kiran Ramnath, Mark Hasegawa-Johnson

摘要:

基于事实的可视化问答(FVQA)是VQA的一个具有挑战性的变体,它要求QA系统在其推理过程中包括来自不同知识图谱(KG)的事实以产生答案。大型幼儿园,特别是常识幼儿园,已知是不完整的,即,并不是所有不存在的事实都是不正确的。因此,能够对不完整的KG进行QA推理是现实世界应用中的一个关键要求,在文献中尚未广泛解决。我们开发了一种新的QA架构,使我们能够对不完整的KG进行推理,这是目前FVQA最先进的方法(SOTA)所缺乏的,因为它们对事实检索的关键依赖。我们使用KG嵌入,一种广泛用于KG完成的技术,用于FVQA的下游任务。我们还采用了一种新的图像表示技术,我们称之为“图像作为知识”,以实现这一能力,以及一个简单的一步CoAttention机制,以参加文本和图像在QA。我们的FVQA架构在推理时间内更快,为O(m),而现有的FVQA SOTA方法为O(N log N),其中m =顶点数,N =边数= O(m^2)。KG嵌入显示持有补充信息词嵌入:这两个指标的组合允许性能媲美SOTA方法在标准答案检索任务,并显着更好(26%绝对),在建议的缺失边缘推理任务。

英文摘要:

Fact-based Visual Question Answering (FVQA), a challenging variant of VQA, requires a QA-system to include facts from a diverse knowledge graph (KG) in its reasoning process to produce an answer. Large KGs, especially common-sense KGs, are known to be incomplete, i.e., not all non-existent facts are always incorrect. Therefore, being able to reason over incomplete KGs for QA is a critical requirement in real-world applications that has not been addressed extensively in the literature. We develop a novel QA architecture that allows us to reason over incomplete KGs, something current FVQA state-of-the-art (SOTA) approaches lack due to their critical reliance on fact retrieval. We use KG Embeddings, a technique widely used for KG completion, for the downstream task of FVQA. We also employ a new image representation technique we call 'Image-as-Knowledge' to enable this capability, alongside a simple one-step CoAttention mechanism to attend to text and image during QA. Our FVQA architecture is faster during inference time, being O(m), as opposed to existing FVQA SOTA methods which are O(N log N), where m = number of vertices, N = number of edges = O(m^2). KG embeddings are shown to hold complementary information to word embeddings: a combination of both metrics permits performance comparable to SOTA methods in the standard answer retrieval task, and significantly better (26% absolute) in the proposed missing-edge reasoning task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑