arXiv 2205.075482022-05-16cs.AIcs.CV

一种用于视觉问答的神经符号ASP管道

A Neuro-Symbolic ASP Pipeline for Visual Question Answering

Institute of Logic and Computation（逻辑与计算研究所）

Thomas Eiter, Nelson Higuera, Johannes Oetsch, Michael Pritz

备注：: Paper presented at the 38th International Conference on Logic Programming (ICLP 2022), 15 pages

摘要：

我们提出了一个神经符号视觉问答（VQA）管道CLEVR，这是一个众所周知的数据集，由图片显示的场景与对象和与它们相关的问题。我们的管道包括（i）训练神经网络进行CLEVR场景的对象分类和边界框预测，（ii）对神经网络的预测值分布进行统计分析，以确定高置信度预测的阈值，以及（iii）将CLEVR问题和网络预测转换为逻辑程序，以便我们可以使用ASP求解器计算答案。通过利用选择规则，我们考虑确定性和非确定性场景编码。我们的实验表明，非确定性的场景编码取得了良好的效果，即使神经网络的训练，而确定性的方法相比，相当差。如果网络预测不够完美，这对于构建强大的VQA系统非常重要。此外，我们表明，限制非确定性的合理选择，允许更有效的实现与相关的神经符号的方法相比，而不会失去太多的准确性。这项工作正在考虑接受在TPLP。

英文摘要：

We present a neuro-symbolic visual question answering (VQA) pipeline for CLEVR, which is a well-known dataset that consists of pictures showing scenes with objects and questions related to them. Our pipeline covers (i) training neural networks for object classification and bounding-box prediction of the CLEVR scenes, (ii) statistical analysis on the distribution of prediction values of the neural networks to determine a threshold for high-confidence predictions, and (iii) a translation of CLEVR questions and network predictions that pass confidence thresholds into logic programs so that we can compute the answers using an ASP solver. By exploiting choice rules, we consider deterministic and non-deterministic scene encodings. Our experiments show that the non-deterministic scene encoding achieves good results even if the neural networks are trained rather poorly in comparison with the deterministic approach. This is important for building robust VQA systems if network predictions are less-than perfect. Furthermore, we show that restricting non-determinism to reasonable choices allows for more efficient implementations in comparison with related neuro-symbolic approaches without loosing much accuracy. This work is under consideration for acceptance in TPLP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文