Visual Question Answering (VQA) has attracted a lot of attention in both Computer Vision and Natural Language Processing communities, not least because it offers insight into the relationships between two important sources of information. Current datasets, and the models built upon them, have focused on questions which are answerable by direct analysis of the question and image alone. The set of such questions that require no external information to answer is interesting, but very limited. It excludes questions which require common sense, or basic factual knowledge to answer, for example. Here we introduce FVQA, a VQA dataset which requires, and supports, much deeper reasoning. FVQA only contains questions which require external information to answer.
We thus extend a conventional visual question answering dataset, which contains image-question-answerg triplets, through additional image-question-answer-supporting fact tuples. The supporting fact is represented as a structural triplet, such as <Cat,CapableOf,ClimbingTrees>.
We evaluate several baseline models on the FVQA dataset, and describe a novel model which is capable of reasoning about an image on the basis of supporting facts.
Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Microsoft Research Asia（微软亚洲研究院）；University of Adelaide（阿德莱德大学）
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
University of Science and Technology of China（中国科学技术大学）；Data Darkness Lab, MIRACLE Center, USTC（数据黑暗实验室，MIRACLE中心，中国科学技术大学）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence（人工智能交叉学科项目）；Department of Intelligence and Information（智能与信息系）；Samsung Advanced Institute of Technology（三星先进技术研究院）
Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Microsoft Research Asia（微软亚洲研究院）；College of Mathematics, Physics and Information Engineering, Jiaxing University（嘉兴大学数理与信息工程学院）
Leveraging Medical Visual Question Answering with Supporting Facts
IBM Research AI, Almaden Research Center（IBM研究AI，阿尔马登研究中心）
在这篇工作笔记论文中，我们描述了IBM Research AI（Almaden）团队参与ImageCLEF 2019 VQA-Med竞赛的情况。该挑战包括四个基于放射学图像的问答任务。成像方式、器官和疾病类型的多样性与小的不平衡训练集相结合，使其成为一个高度复杂的问题。为了克服这些困难，我们实现了一个模块化的流水线架构，利用迁移学习和多任务学习。我们的研究结果导致了一种名为支持事实网络（SFN）的新模型的发展。SFN背后的主要思想是交叉利用来自上游任务的信息，以提高下游任务的准确性。这种方法显著提高了验证集的得分（F-1得分提高了18分）。最后，我们提交了四个运行的竞争，并排名第七。
R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering
Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；East China Normal University（华东师范大学）；Microsoft Corporation（微软公司）
