One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pre-training and supervised training data with transformer-based models. Second, explicit, symbolic knowledge encoded in knowledge bases. Our approach combines both - exploiting the powerful implicit reasoning of transformer models for answer prediction, and integrating symbolic representations from a knowledge graph, while never losing their explicit semantics to an implicit embedding. We combine diverse sources of knowledge to cover the wide variety of knowledge needed to solve knowledge-based questions. We show our approach, KRISP (Knowledge Reasoning with Implicit and Symbolic rePresentations), significantly outperforms state-of-the-art on OK-VQA, the largest available dataset for open-domain knowledge-based VQA. We show with extensive ablations that while our model successfully exploits implicit knowledge reasoning, the symbolic answer module which explicitly connects the knowledge graph to the answer vocabulary is critical to the performance of our method and generalizes to rare answers.
Revisiting KRISP: A Lightweight Reproduction and Analysis of Knowledge-Enhanced Vision-Language Models
Centre For Digital Health（数字健康中心）；Indian Institute of Technology Bombay（孟买印度理工学院）；Department of Metallurgical engineering & Material science（冶金工程与材料科学系）；Department of Bioscience & Bioengineering（生物科学与生物工程系）
Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Southern California（南加州大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Freiburg（弗莱堡大学）
视觉问答（VQA）的主要方法依赖于使用“黑盒”神经编码器对图像和问题进行编码，并将单个令牌解码为“是”或“否”等答案。尽管这种方法有很强的定量结果，但它很难为预测过程提出直观的、人类可读的证明形式。为了解决这个不足，我们重新制定VQA作为一个完整的答案生成任务，这需要模型来证明其预测在自然语言。我们提出了LRTA [看，读，想，答]，一个透明的神经符号推理框架的视觉问答，解决问题的一步一步像人类一样，并提供人类可读的形式的理由在每一步。具体来说，LRTA学习首先将图像转换为场景图，并将问题解析为多个推理指令。然后，它通过使用递归神经符号执行模块遍历场景图来一次执行一个推理指令。最后，它生成一个完整的答案，给定的问题与自然语言的理由。我们在GQA数据集上的实验表明，LRTA的性能大大优于最先进的模型（43.1% vs. 28.0%）的完整答案生成任务。我们还通过删除问题中的语言线索（属性和关系）来创建扰动GQA测试集，以分析模型是否具有表面数据相关性的智能猜测。我们表明，LRTA向真正理解这个问题迈出了一步，而最先进的模型往往从训练数据中学习表面的相关性。
Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
The University of Sydney, School of Computer Science（悉尼大学计算机科学学院）；University of Technology Sydney, School of Computer Science（新南威尔士大学技术学院）；University of Technology Sydney, TD School（新南威尔士大学TD学院）；Australian Catholic University, Peter Faber Business School（澳大利亚天主教大学彼得·法伯商学院）
Department of Computer Science, University of Oxford（牛津大学计算机科学系）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Institute of Logic and Computation, TU Wien（维也纳技术大学逻辑与计算研究所）
SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs
Universidade Federal de Mato Grosso (UFMT)（马托格罗索联邦大学）；Universidade Federal de Goiás (UFG)（戈亚斯联邦大学）；Advanced Knowledge Center for Immersive Technologies (AKCIT)（沉浸式技术高级知识中心）
StaR-KVQA: Structured Reasoning Traces for Implicit-Knowledge Visual Question Answering
Ant International, Ant Group（蚂蚁集团国际部，蚂蚁集团）；School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理学院计算与信息系统学院）；Anhui Provincial Key Laboratory of High Performance Computing（安徽省高性能计算重点实验室）
Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoning
Shanghai Jiao Tong University（上海交通大学）；ZhongguanCun Academy（中关村学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai University（上海大学）；City University of Hong Kong（香港城市大学）；Wuhan University（武汉大学）
