We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.
Beihang University（北京航空航天大学）；University of Sydney（悉尼大学）
Recently, 3D vision-and-language tasks have attracted increasing research interest. Compared to other vision-and-language tasks, the 3D visual question answering (VQA) task is less exploited and is more susceptible to language priors and co-reference ambiguity. Meanwhile, a couple of recently proposed 3D VQA datasets do not well support 3D VQA task due to their limited scale and annotation methods. In this work, we formally define and address a 3D grounded VQA task by collecting a new 3D VQA dataset, referred to as FE-3DGQA, with diverse and relatively free-form question-answer pairs, as well …
Seoul National University（首尔大学）；Allen Institute for AI（艾伦人工智能研究所）；University of Oxford（牛津大学）；Hyundai Motor Company（现代汽车公司）
360$^\circ$ videos convey holistic views for the surroundings of a scene. It provides audio-visual cues beyond pre-determined normal field of views and displays distinctive spatial relations on a sphere. However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings. We propose a novel benchmark named Pano-AVQA as a large-scale grounded audio-visual question answering dataset on panoramic videos. Using 5.4K 360$^\circ$ video clips harvested online, we collect two types o…
Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictions
Department of Informatics, University of Oslo（奥斯陆大学信息学院）；Integreat – Norwegian Centre for Knowledge-driven Machine Learning（挪威知识驱动机器学习中心）；TRUST – The Norwegian Centre for Trustworthy AI（挪威可信人工智能中心）
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Recent studies have shown that current VQA models are heavily biased on the language priors in the train set to answer the question, irrespective of the image. E.g., overwhelmingly answer "what sport is" as "tennis" or "what color banana" as "yellow." This behavior restricts them from real-world application scenarios. In this work, we propose a novel model-agnostic question encoder, Visually-Grounded Question Encoder (VGQE), for VQA that reduces this effect. VGQE utilizes both visual and language modalities equally while encoding the question. Hence the question representation itself gets suff…
With the expressed goal of improving system transparency and visual grounding in the reasoning process in VQA, we present a modular system for the task of compositional VQA based on scene graphs. Our system is called "Adventurer's Treasure Hunt" (or ATH), named after an analogy we draw between our model's search procedure for an answer and an adventurer's search for treasure. We developed ATH with three characteristic features in mind: 1. By design, ATH allows us to explicitly quantify the impact of each of the sub-components on overall VQA performance, as well as their performance on their in…
MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Department of Mathematics and Computer Science, Faculty of Science, Alexandria University（亚历山大大学数学与计算机科学系）；Sheikh Shakhbout Medical City（谢赫·沙赫布OUT医疗城）
ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China
LMU Munich（慕尼黑大学）；FAU Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Tübingen & Tübingen AI Center（图宾根大学与图宾根人工智能中心）；Sun Yat-sen University（中山大学）；University of Copenhagen（哥本哈根大学）；University of Maryland, College Park（马里兰大学帕克分校）
PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究理事会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；The Chinese University of Hong Kong（香港中文大学）；National Hospital for Neurology and Neurosurgery（国家神经学与神经外科医院）
GRACE: Grounded Reasoning via Adapter Composition and Evidence-Aware Calibration for Educational Visual Question Answering
Columbia University（哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；Stevens Institute of Technology（史蒂文斯理工学院）；Georgetown University（乔治城大学）；San Francisco State University（旧金山州立大学）；Loughborough University（拉夫堡大学）；Texas A&M University（德克萨斯农工大学）
CRCV University of Central Florida（中佛罗里达大学计算机视觉研究中心）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
The problem of grounding VQA tasks has seen an increased attention in the research community recently, with most attempts usually focusing on solving this task by using pretrained object detectors. However, pre-trained object detectors require bounding box annotations for detecting relevant objects in the vocabulary, which may not always be feasible for real-life large-scale applications. In this paper, we focus on a more relaxed setting: the grounding of relevant visual entities in a weakly supervised manner by training on the VQA task alone. To address this problem, we propose a visual capsu…
We propose a new 3D spatial understanding task of 3D Question Answering (3D-QA). In the 3D-QA task, models receive visual information from the entire 3D scene of the rich RGB-D indoor scan and answer the given textual questions about the 3D scene. Unlike the 2D-question answering of VQA, the conventional 2D-QA models suffer from problems with spatial understanding of object alignment and directions and fail the object identification from the textual questions in 3D-QA. We propose a baseline model for 3D-QA, named ScanQA model, where the model learns a fused descriptor from 3D object proposals …
Salesforce Research Asia（Salesforce亚洲研究院）；Singapore Management University（新加坡管理大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR））
Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded dialogue tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance and language cross-turn dependencies. Motivated by recent NMN approaches on image-grounded tasks, we introduce Video-grounded Neural Module Network (VGNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VGN…
Location-Aware Pretraining for Medical Difference Visual Question Answering
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda（电气与计算机工程系，卡内基梅隆大学，刚果（金）基利齐）；University of Melbourne, Melbourne, Australia（墨尔本大学，墨尔本，澳大利亚）
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Universiti Malaya（马来亚大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港智能系统研究所人工智能与机器人中心）；University College London（伦敦大学学院）
Answer with Evidence: Consistency-Aware Grounded Visual Question Answering for Roadside Traffic Scenes
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Jiangnan University（江南大学）；Central China Normal University（华中师范大学）；CUHK（香港中文大学）；Peking University（北京大学）；Wuhan University（武汉大学）；Fudan University（复旦大学）
University of Cambridge（剑桥大学）；University of Washington（华盛顿大学）
The widely used Fact-based Visual Question Answering (FVQA) dataset contains visually-grounded questions that require information retrieval using common sense knowledge graphs to answer. It has been observed that the original dataset is highly imbalanced and concentrated on a small portion of its associated knowledge graph. We introduce FVQA 2.0 which contains adversarial variants of test questions to address this imbalance. We show that systems trained with the original FVQA train sets can be vulnerable to adversarial samples and we demonstrate an augmentation scheme to reduce this vulnerabil…
Institute for Datability Science, Osaka University（大阪大学数据能力科学研究所）；CyberAgent, Inc.（CyberAgent公司）
A paraphrase is a restatement of the meaning of a text in other words. Paraphrases have been studied to enhance the performance of many natural language processing tasks. In this paper, we propose a novel task iParaphrasing to extract visually grounded paraphrases (VGPs), which are different phrasal expressions describing the same visual concept in an image. These extracted VGPs have the potential to improve language and image multimodal tasks such as visual question answering and image captioning. How to model the similarity between VGPs is the key of iParaphrasing. We apply various existing …
Wright State University（莱特州立大学）；NEC Laboratories America（美国NEC实验室）
We introduce a new inference task - Visual Entailment (VE) - which differs from traditional Textual Entailment (TE) tasks whereby a premise is defined by an image, rather than a natural language sentence as in TE tasks. A novel dataset SNLI-VE (publicly available at https://github.com/necla-ml/SNLI-VE) is proposed for VE tasks based on the Stanford Natural Language Inference corpus and Flickr30k. We introduce a differentiable architecture called the Explainable Visual Entailment model (EVE) to tackle the VE problem. EVE and several other state-of-the-art visual question answering (VQA) based m…
Questions that require counting a variety of objects in images remain a major challenge in visual question answering (VQA). The most common approaches to VQA involve either classifying answers based on fixed length representations of both the image and question or summing fractional counts estimated from each section of the image. In contrast, we treat counting as a sequential decision process and force our model to make discrete choices of what to count. Specifically, the model sequentially selects from detected objects and learns interactions between objects that influence subsequent selecti…
Singapore Management University（新加坡管理大学）；A*STAR, Institute for Infocomm Research（新加坡科技研究局资讯通信研究院）；Salesforce Research Asia（赛富时亚洲研究院）
Compared to traditional visual question answering, video-grounded dialogues require additional reasoning over dialogue context to answer questions in a multi-turn setting. Previous approaches to video-grounded dialogues mostly use dialogue context as a simple text input without modelling the inherent information flows at the turn level. In this paper, we propose a novel framework of Reasoning Paths in Dialogue Context (PDC). PDC model discovers information flows among dialogue turns through a semantic graph constructed based on lexical components in each question and answer. PDC model then lea…
A$^2$Safe: Counterfactual Evidence-Aligned Adaptive Agent Collaboration for Safe and Effective Visual Question Answering
Macau University of Science and Technology（澳门科技大学）；Wuhan University of Technology（武汉理工大学）；Nanjing Institute of Technology（南京工程学院）；National Tsing Hua University（国立清华大学）
Detection-based Intermediate Supervision for Visual Question Answering
Huazhong University of Science and Technology（华中科技大学）；Ping An Property & Casualty Insurance Company of China, Ltd.（中国平安财产保险股份有限公司）；ByteDance Inc.（字节跳动股份有限公司）
Seoul National University（首尔国立大学）；Hanyang University（汉阳大学）；AI Institute (AIIS), Seoul National University（首尔国立大学人工智能研究院（AIIS））
Knowledge-based visual question answering (QA) aims to answer a question which requires visually-grounded external knowledge beyond image content itself. Answering complex questions that require multi-hop reasoning under weak supervision is considered as a challenging problem since i) no supervision is given to the reasoning process and ii) high-order semantics of multi-hop knowledge facts need to be captured. In this paper, we introduce a concept of hypergraph to encode high-level semantics of a question and a knowledge base, and to learn high-order associations between them. The proposed mod…
University of Rochester（罗切斯特大学）；Microsoft（微软）；University College London（伦敦大学学院）
The popularity of image sharing on social media and the engagement it creates between users reflects the important role that visual context plays in everyday conversations. We present a novel task, Image-Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple-reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utte…
Visual Question Answering (VQA) concerns providing answers to Natural Language questions about images. Several deep neural network approaches have been proposed to model the task in an end-to-end fashion. Whereas the task is grounded in visual processing, if the question focuses on events described by verbs, the language understanding component becomes crucial. Our hypothesis is that models should be aware of verb semantics, as expressed via semantic role labels, argument types, and/or frame elements. Unfortunately, no VQA dataset exists that includes verb semantic information. Our first contr…
We propose UniTAB that Unifies Text And Box outputs for grounded vision-language (VL) modeling. Grounded VL tasks such as grounded captioning require the model to generate a text description and align predicted words with object regions. To achieve this, models must generate desired text and box outputs together, and meanwhile indicate the alignments between words and boxes. In contrast to existing solutions that use multiple separate modules for different outputs, UniTAB represents both text and box outputs with a shared token sequence, and introduces a special <obj> token to naturally indica…
The task of Outside Knowledge Visual Question Answering (OKVQA) requires an automatic system to answer natural language questions about pictures and images using external knowledge. We observe that many visual questions, which contain deictic referential phrases referring to entities in the image, can be rewritten as "non-grounded" questions and can be answered by existing text-based question answering systems. This allows for the reuse of existing text-based Open Domain Question Answering (QA) Systems for visual question answering. In this work, we propose a potentially data-efficient approac…
Australian National University（澳大利亚国立大学）；University of Adelaide（阿德莱德大学）；Queensland University of Technology（昆士兰科技大学）；Macquarie University（麦考瑞大学）
A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visual…
MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering
Lanzhou University（兰州大学）；National University of Singapore（新加坡国立大学）；Sun Yat-sen University（中山大学）；Jinan University（暨南大学）；Nanyang Technological University（南洋理工大学）；Peking University（北京大学）；University of Science and Technology of China（中国科学技术大学）；Hefei University of Technology（合肥工业大学）
