In this paper, the LCV2 modular method is proposed for the Grounded Visual Question Answering task in the vision-language multimodal domain. This approach relies on a frozen large language model (LLM) as intermediate mediator between the off-the-shelf VQA model and the off-the-shelf visual grounding (VG) model, where the LLM transforms and conveys textual information between the two modules based on a designed prompt. LCV2 establish an integrated plug-and-play framework without the need for any pre-training process. This framework can be deployed for VQA Grounding tasks under low computational resources. The modularized model within the framework allows application with various state-of-the-art pre-trained models, exhibiting significant potential to be advance with the times. Experimental implementations were conducted under constrained computational and memory resources, evaluating the proposed method's performance on benchmark datasets including GQA, CLEVR, and VizWiz-VQA-Grounding. Comparative analyses with baseline methods demonstrate the robust competitiveness of LCV2.
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption da…
Location-Aware Pretraining for Medical Difference Visual Question Answering
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda（电气与计算机工程系，卡内基梅隆大学，刚果（金）基利齐）；University of Melbourne, Melbourne, Australia（墨尔本大学，墨尔本，澳大利亚）
MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering
Lanzhou University（兰州大学）；National University of Singapore（新加坡国立大学）；Sun Yat-sen University（中山大学）；Jinan University（暨南大学）；Nanyang Technological University（南洋理工大学）；Peking University（北京大学）；University of Science and Technology of China（中国科学技术大学）；Hefei University of Technology（合肥工业大学）
Textbook Question Answering (TQA) is a complex multimodal task to infer answers given large context descriptions and abundant diagrams. Compared with Visual Question Answering (VQA), TQA contains a large number of uncommon terminologies and various diagram inputs. It brings new challenges to the representation capability of language model for domain-specific spans. And it also pushes the multimodal fusion to a more complex level. To tackle the above issues, we propose a novel model named MoCA, which incorporates multi-stage domain pretraining and multimodal cross attention for the TQA task. Fi…
Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and-Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pret…
Beihang University（北京航空航天大学）；University of Sydney（悉尼大学）
Recently, 3D vision-and-language tasks have attracted increasing research interest. Compared to other vision-and-language tasks, the 3D visual question answering (VQA) task is less exploited and is more susceptible to language priors and co-reference ambiguity. Meanwhile, a couple of recently proposed 3D VQA datasets do not well support 3D VQA task due to their limited scale and annotation methods. In this work, we formally define and address a 3D grounded VQA task by collecting a new 3D VQA dataset, referred to as FE-3DGQA, with diverse and relatively free-form question-answer pairs, as well …
Seoul National University（首尔大学）；Allen Institute for AI（艾伦人工智能研究所）；University of Oxford（牛津大学）；Hyundai Motor Company（现代汽车公司）
360$^\circ$ videos convey holistic views for the surroundings of a scene. It provides audio-visual cues beyond pre-determined normal field of views and displays distinctive spatial relations on a sphere. However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings. We propose a novel benchmark named Pano-AVQA as a large-scale grounded audio-visual question answering dataset on panoramic videos. Using 5.4K 360$^\circ$ video clips harvested online, we collect two types o…
Tel-Aviv University（特拉维夫大学）；Allen Institute for AI（艾伦人工智能研究所）
Answering questions that involve multi-step reasoning requires decomposing them and using the answers of intermediate steps to reach the final answer. However, state-of-the-art models in grounded question answering often do not explicitly perform decomposition, leading to difficulties in generalization to out-of-distribution examples. In this work, we propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser. Our model induces latent trees, driven by end-to-end (the answer) supervision only. We show that t…
University of Adelaide（阿德莱德大学）；Amazon（亚马逊）；Idiap Research Institute（Idiap研究所）
The availability of clean and diverse labeled data is a major roadblock for training models on complex tasks such as visual question answering (VQA). The extensive work on large vision-and-language models has shown that self-supervised learning is effective for pretraining multimodal interactions. In this technical report, we focus on visual representations. We review and evaluate self-supervised methods to leverage unlabeled images and pretrain a model, which we then fine-tune on a custom VQA task that allows controlled evaluation and diagnosis. We compare energy-based models (EBMs) with cont…
ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
Khalifa University（卡利法大学）；University of Western Australia（西澳大学）；The Chinese University of Hong Kong（香港中文大学）；University of Melbourne（墨尔本大学）；University of Central Florida（佛罗里达中央大学）
Birla Institute of Technology and Science, Pilani（比拉理工学院）；Carnegie Mellon University（卡内基梅隆大学）
In many real-world scenarios where extrinsic rewards to the agent are extremely sparse, curiosity has emerged as a useful concept providing intrinsic rewards that enable the agent to explore its environment and acquire information to achieve its goals. Despite their strong performance on many sparse-reward tasks, existing curiosity approaches rely on an overly holistic view of state transitions, and do not allow for a structured understanding of specific aspects of the environment. In this paper, we formulate curiosity based on grounded question answering by encouraging the agent to ask questi…
A Knowledge-Injected Curriculum Pretraining Framework for Question Answering
University of Science and Technology of China（中国科学技术大学）；Guangdong OPPO Mobile Telecommunications Corp., Ltd（广东欧珀移动通信有限公司）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）
Stanford University（斯坦福大学）；Dresden University of Technology（德累斯顿工业大学）
We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and …
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Recent studies have shown that current VQA models are heavily biased on the language priors in the train set to answer the question, irrespective of the image. E.g., overwhelmingly answer "what sport is" as "tennis" or "what color banana" as "yellow." This behavior restricts them from real-world application scenarios. In this work, we propose a novel model-agnostic question encoder, Visually-Grounded Question Encoder (VGQE), for VQA that reduces this effect. VGQE utilizes both visual and language modalities equally while encoding the question. Hence the question representation itself gets suff…
Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；SynthLabs；Carnegie Mellon University（卡内基梅隆大学）
With the expressed goal of improving system transparency and visual grounding in the reasoning process in VQA, we present a modular system for the task of compositional VQA based on scene graphs. Our system is called "Adventurer's Treasure Hunt" (or ATH), named after an analogy we draw between our model's search procedure for an answer and an adventurer's search for treasure. We developed ATH with three characteristic features in mind: 1. By design, ATH allows us to explicitly quantify the impact of each of the sub-components on overall VQA performance, as well as their performance on their in…
Seoul National University（首尔国立大学）；AI Institute for Seoul National University (AIIS)（首尔国立大学人工智能研究院）；Hanyang University（汉阳大学）
Video Question Answering is a task which requires an AI agent to answer questions grounded in video. This task entails three key challenges: (1) understand the intention of various questions, (2) capturing various elements of the input video (e.g., object, action, causality), and (3) cross-modal grounding between language and vision information. We propose Motion-Appearance Synergistic Networks (MASN), which embed two cross-modal features grounded on motion and appearance information and selectively utilize them depending on the question's intentions. MASN consists of a motion module, an appea…
Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictions
Department of Informatics, University of Oslo（奥斯陆大学信息学院）；Integreat – Norwegian Centre for Knowledge-driven Machine Learning（挪威知识驱动机器学习中心）；TRUST – The Norwegian Centre for Trustworthy AI（挪威可信人工智能中心）
PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（惠康/工程与物理科学研究理事会介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；The Chinese University of Hong Kong（香港中文大学）；National Hospital for Neurology and Neurosurgery（国家神经学与神经外科医院）
MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Department of Mathematics and Computer Science, Faculty of Science, Alexandria University（亚历山大大学数学与计算机科学系）；Sheikh Shakhbout Medical City（谢赫·沙赫布OUT医疗城）
ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China
LMU Munich（慕尼黑大学）；FAU Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Tübingen & Tübingen AI Center（图宾根大学与图宾根人工智能中心）；Sun Yat-sen University（中山大学）；University of Copenhagen（哥本哈根大学）；University of Maryland, College Park（马里兰大学帕克分校）
HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways
University of Calabria（卡拉布里亚大学）；University of Catanzaro（卡坦扎罗大学）；University of Eastern Piedmont（东皮埃蒙特大学）；University of Pisa（比萨大学）；University of Cambridge（剑桥大学）
CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，科技研究局，新加坡）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Singapore Polytechnic（新加坡理工学院）
We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer. Specifically, we perform masked vision-language modeling on image-text pairs, masked language modeling on texts, and masked image modeling on images. VL-BEiT is learned from scratch with one unified pretraining task, one shared backbone, and one-stage training. Our method is conceptually simple and empirically effective. Experimental r…
Propagating construction-time knowledge quality into medical question answering: A framework grounded in clinical guidelines
Nanjing Medical University（南京医科大学）；The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院）；The Friendship Hospital of Ili Kazakh Autonomous Prefecture（伊犁哈萨克自治州友谊医院）；Nanjing University（南京大学）
FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings
University of Groningen（格罗宁根大学）；Goethe University Frankfurt（法兰克福大学）；DataNXT GmbH（DataNXT有限公司）；Halle Institute for Economic Research (IWH)（哈雷经济研究所）；Martin Luther University Halle-Wittenberg（马丁路德大学哈雷-维滕贝格分校）
