Understanding data visualizations like charts and plots requires reasoning about both visual elements and numerics. Although strong in extractive questions, current chart visual question answering (chart VQA) models suffer on complex reasoning questions. In this work, we address the lack of reasoning ability by data augmentation. We leverage Large Language Models (LLMs), which have shown to have strong reasoning ability, as an automatic data annotator that generates question-answer annotations for chart images. The key innovation in our method lies in the Synthesize Step-by-Step strategy: our LLM-based data generator learns to decompose the complex question into step-by-step sub-questions (rationales), which are then used to derive the final answer using external tools, i.e. Python. This step-wise generation procedure is trained on synthetic data generated using a template-based QA generation pipeline. Experimental results highlight the significance of the proposed step-by-step generation. By training with the LLM-augmented data (LAMENDA), we significantly enhance the chart VQA models, achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets. In particular, our approach improves the accuracy of the previous state-of-the-art approach from 38% to 54% on the human-written questions in the ChartQA dataset, which needs strong reasoning. We hope our work underscores the potential of synthetic data and encourages further exploration of data augmentation using LLMs for reasoning-heavy tasks.
VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学);NVIDIA AI Technology Center(英伟达人工智能技术中心);University of Rochester(罗切斯特大学)
UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation
Fung Institute for Engineering Leadership(工程领导力基金会);University of California, Berkeley(加州大学伯克利分校);Department of Radiology(放射科);University of California, San Francisco(加州大学旧金山分校);Division of Clinical Informatics and Digital Transformation(临床信息学与数字转型部);Department of Neurological Surgery(神经外科部)
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
Department of Computer Science and Artificial Intelligence, Dongguk University(东国大学计算机科学与人工智能系);Department of Electronics and Electrical Engineering, Dongguk University(东国大学电子与电气工程系)
MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering
School of Computing and Augmented Intelligence(计算与增强智能学院);Arizona State University(亚利桑那州立大学);Department of Computer Science(计算机科学系);International Institute of Information Technology(国际信息科技研究所)
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence(人工智能交叉学科项目);Department of Intelligence and Information(智能与信息系);Samsung Advanced Institute of Technology(三星先进技术研究院)
AWM: Answerable Working Memory for Long-Document VQA Agents
University of Oslo(奥斯陆大学);Bosch Center for AI(博世人工智能中心);University of Stuttgart(斯图加特大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tsinghua University(清华大学)
Department of Computer Science, University of Oxford(牛津大学计算机科学系);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);Institute of Logic and Computation, TU Wien(维也纳技术大学逻辑与计算研究所)
SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models
Peking University Beijing China;Huawei Technologies Ltd. Beijing China;Shanghai AI Laboratory Shanghai China;Peking University Center for Machine Learning Research Beijing China;Peking University School of Computer Science \& Key Lab of High Confidence Software Technologies (MOE) Beijing China;Peking University;Huawei Technologies Ltd.;Shanghai AI Laboratory
本文提出 SynthVLM 数据合成方法,利用扩散模型从文本描述合成并精选图像生成精确对齐的图像-文本对,构建了 SynthVLM-100K 数据集,并训练出在 VQA 任务上达到 SOTA 且保留语言能力的多模态大模型。
Asking like Socrates: Socrates helps VLMs understand remote sensing images
School of Geosciences and Info-Physics, Central South University(中南大学地球科学与信息物理学院);Baidu Inc.(百度公司);School of Earth Sciences, Zhejiang University(浙江大学地球科学学院)
OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
York University(约克大学);Vector Institute(向量研究所);University of British Columbia(不列颠哥伦比亚大学);University of Toronto(多伦多大学);Unity Health Toronto / St. Michael’s Hospital(多伦多联合健康/圣迈克尔医院);University Health Network(大学健康网络);Arc Institute(弧研究所);Queen's University(女王大学)