Emerging multimodal large language models (MLLMs) exhibit great potential for chart question answering (CQA). Recent efforts primarily focus on scaling up training datasets (i.e., charts, data tables, and question-answer (QA) pairs) through data collection and synthesis. However, our empirical study on existing MLLMs and CQA datasets reveals notable gaps. First, current data collection and synthesis focus on data volume and lack consideration of fine-grained visual encodings and QA tasks, resulting in unbalanced data distribution divergent from practical CQA scenarios. Second, existing work follows the training recipe of the base MLLMs initially designed for natural images, under-exploring the adaptation to unique chart characteristics, such as rich text elements. To fill the gap, we propose a visualization-referenced instruction tuning approach to guide the training dataset enhancement and model development. Specifically, we propose a novel data engine to effectively filter diverse and high-quality data from existing datasets and subsequently refine and augment the data using LLM-based generation techniques to better align with practical QA tasks and visual encodings. Then, to facilitate the adaptation to chart characteristics, we utilize the enriched data to train an MLLM by unfreezing the vision encoder and incorporating a mixture-of-resolution adaptation strategy for enhanced fine-grained recognition. Experimental results validate the effectiveness of our approach. Even with fewer training examples, our model consistently outperforms state-of-the-art CQA models on established benchmarks. We also contribute a dataset split as a benchmark for future research. Source codes and datasets of this paper are available at https://github.com/zengxingchen/ChartQA-MLLM.
DOGR: Towards Versatile Visual Document Grounding and Referring
Xi’an Jiaotong University（西安交通大学）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；City University of Hongkong（香港城市大学）；Institute of Automation, CAS（中国科学院自动化研究所）；Harvard University（哈佛大学）；vivo Mobile Communication Co.（vivo移动通信公司）
Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China（香港理工大学土木与环境工程系）；Department of Traffic Engineering, Tongji University, Shanghai, 201804, China（同济大学交通工程系）
ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understanding
National University of Defense Technology（国防科技大学）；Intelligent Game and Decision Lab（智能游戏与决策实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；OPPO AI Center（OPPO AI中心）
