Knowledge Condensation and Reasoning for Knowledge-based VQA
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Kuaishou(快手)
Dongze Hao, Jian Jia, Longteng Guo, Qunbo Wang, Te Yang, Yan Li, Yanhua Cheng, Bo Wang, Quan Chen, Han Li, Jing Liu
Knowledge-based visual question answering (KB-VQA) is a challenging task, which requires the model to leverage external knowledge for comprehending and answering questions grounded in visual content. Recent studies retrieve the knowledge passages from external knowledge bases and then use them to answer questions. However, these retrieved knowledge passages often contain irrelevant or noisy information, which limits the performance of the model. To address the challenge, we propose two synergistic models: Knowledge Condensation model and Knowledge Reasoning model. We condense the retrieved knowledge passages from two perspectives. First, we leverage the multimodal perception and reasoning ability of the visual-language models to distill concise knowledge concepts from retrieved lengthy passages, ensuring relevance to both the visual content and the question. Second, we leverage the text comprehension ability of the large language models to summarize and condense the passages into the knowledge essence which helps answer the question. These two types of condensed knowledge are then seamlessly integrated into our Knowledge Reasoning model, which judiciously navigates through the amalgamated information to arrive at the conclusive answer. Extensive experiments validate the superiority of the proposed method. Compared to previous methods, our method achieves state-of-the-art performance on knowledge-based VQA datasets (65.1% on OK-VQA and 60.1% on A-OKVQA) without resorting to the knowledge produced by GPT-3 (175B).
M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis
School of Computing;Artificial Intelligence Southwest Jiaotong University Chengdu, China;Artificial Intelligence Southwest Jiaotong University Chengdu, China z\ c\;Institute of Innovative Research Tokyo Institute of Technology Yokohama, Japan
MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering
Faculty Of Information Technology VNU University of Engineering(信息科技学院越南工程大学);IT-BT Convergence Technology Division Vietnam-Korea Institute of Science(IT-BT融合技术部门越南-韩国科学技术院);TADI Global Lab TADI Global Company Limited(TADI全球实验室TADI全球有限公司);Faculty of Finance Banking Academy of Vietnam(金融学院越南银行学院)
Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering
School of computer science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);South China Normal University(华南师范大学);Northeastern University(东北大学);Anhui University(安徽大学)
MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning
University of Pennsylvania(宾夕法尼亚大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);UC Santa Cruz(加州大学圣克鲁兹分校);Harvard University(哈佛大学);UC Berkeley(加州大学伯克利分校);Emory University(埃默里大学);UC San Francisco(旧金山加州大学)
Adaptive Reinforcement for Open-ended Medical Reasoning via Semantic-Guided Reward Collapse Mitigation
College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);Fysics Intelligence Technologies Co., Ltd. (Fysics AI)(菲斯克智能科技有限公司(菲斯克AI));Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Cohere Labs(Cohere实验室);Eisai Inc.(艾伯维公司);Indian Institute of Science, Bangalore(班加罗尔印度科学研究院)
HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision
Southern University of Science and Technology, Shenzhen, China(南方科技大学,深圳,中国);College of William and Mary, Williamsburg, VA 23185, USA(威廉玛丽学院,威廉斯堡,VA 23185,美国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration
Huazhong University of Science and Technology(华中科技大学);Eastern Institute of Technology(宁波工程学院);Purple Mountain Laboratories(紫金山实验室);Alibaba Inc.(阿里巴巴集团);Fudan University(复旦大学)
Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy
Simula Metropolitan Center for Digital Engineering (SimulaMet)(Simula Metropolitan Center for Digital Engineering);Oslo Metropolitan University (OsloMet)(Oslo Metropolitan University);Simula Research Laboratory
R^3-VQA: "Read the Room" by Video Social Reasoning
School of Intelligence Science and Technology, Peking University(北京理工大学智能科学与技术学院);College of Artificial Intelligence, Xi’an Jiaotong University(西安交通大学人工智能学院);State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence(一般人工智能国家重点实验室);Yuanpei College, Peking University(北京大学元培学院);Tsinghua University(清华大学);University of California, Los Angeles(加州大学洛杉矶分校);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室)