arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.04785cs.CVcs.AIcs.CL
收藏

基于形式逻辑的可视化问答

Visual Question Answering based on Formal Logic

Georgia Institute of Technology(佐治亚理工学院);Cisco(思科);Georgia Tech Research Institute(佐治亚理工研究所)

Muralikrishnna G. Sethuraman, Ali Payani, Faramarz Fekri, J. Clayton Kerce

摘要:

视觉问答(VQA)近年来在机器学习社区中获得了很大的吸引力,这是由于在理解来自多个模态(即,图像、语言)。在VQA中,基于一组图像提出一系列问题,手头的任务是找到答案。为了实现这一点,我们采取了符号推理为基础的方法,使用形式逻辑的框架。图像和问题被转换成符号表示,在符号表示上执行明确的推理。我们提出了一个形式逻辑框架,其中(i)图像在场景图的帮助下转换为逻辑背景事实,(ii)使用基于Transformer的深度学习模型将问题转换为一阶谓词逻辑子句,以及(iii)通过使用背景知识和谓词子句的基础来执行可满足性检查,以获得答案。我们提出的方法是高度可解释的,管道中的每一步都可以很容易地由人类分析。我们在CLEVR和GQA数据集上验证了我们的方法。我们在CLEVR数据集上实现了接近完美的99.6%的准确率,与最先进的模型相当,这表明形式逻辑是解决视觉问答的可行工具。我们的模型也是数据高效的,在CLEVR数据集上实现了99.1%的准确率,仅用10%的训练数据进行训练。

英文摘要:

Visual question answering (VQA) has been gaining a lot of traction in the machine learning community in the recent years due to the challenges posed in understanding information coming from multiple modalities (i.e., images, language). In VQA, a series of questions are posed based on a set of images and the task at hand is to arrive at the answer. To achieve this, we take a symbolic reasoning based approach using the framework of formal logic. The image and the questions are converted into symbolic representations on which explicit reasoning is performed. We propose a formal logic framework where (i) images are converted to logical background facts with the help of scene graphs, (ii) the questions are translated to first-order predicate logic clauses using a transformer based deep learning model, and (iii) perform satisfiability checks, by using the background knowledge and the grounding of predicate clauses, to obtain the answer. Our proposed method is highly interpretable and each step in the pipeline can be easily analyzed by a human. We validate our approach on the CLEVR and the GQA dataset. We achieve near perfect accuracy of 99.6% on the CLEVR dataset comparable to the state of art models, showcasing that formal logic is a viable tool to tackle visual question answering. Our model is also data efficient, achieving 99.1% accuracy on CLEVR dataset when trained on just 10% of the training data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑