RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际 Graduate School）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Yinwang Intelligent Technology Co., Ltd.（亿纬智能科技有限公司）；Huawei’s 2012 Lab（华为2012实验室）
Yunshuang Nie, Bingqian Lin, Minzhe Niu, Kun Xiang, Jianhua Han, Guowei Huang, Xingyue Quan, Hang Xu, Bokui Chen, Xiaodan Liang
Pre-trained Multi-modal Large Language Models (MLLMs) provide a knowledge-rich foundation for post-training by leveraging their inherent perception and reasoning capabilities to solve complex tasks. However, the lack of an efficient evaluation framework impedes the diagnosis of their performance bottlenecks. Current evaluation primarily relies on testing after supervised fine-tuning, which introduces laborious additional training and autoregressive decoding costs. Meanwhile, common pre-training metrics cannot quantify a model's perception and reasoning abilities in a disentangled manner. Furthermore, existing evaluation benchmarks are typically limited in scale or misaligned with pre-training objectives. Thus, we propose RADAR, an efficient ability-centric evaluation framework for Revealing Asymmetric Development of Abilities in MLLM pRe-training. RADAR involves two key components: (1) Soft Discrimination Score, a novel metric for robustly tracking ability development without fine-tuning, based on quantifying nuanced gradations of the model preference for the correct answer over distractors; and (2) Multi-Modal Mixture Benchmark, a new 15K+ sample benchmark for comprehensively evaluating pre-trained MLLMs' perception and reasoning abilities in a 0-shot manner, where we unify authoritative benchmark datasets and carefully collect new datasets, extending the evaluation scope and addressing the critical gaps in current benchmarks. With RADAR, we comprehensively reveal the asymmetric development of perceptual and reasoning capabilities in pretrained MLLMs across diverse factors, including data volume, model size, and pretraining strategy. Our RADAR underscores the need for a decomposed perspective on pre-training ability bottlenecks, informing targeted interventions to advance MLLMs efficiently. Our code is publicly available at https://github.com/Nieysh/RADAR.
RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；ZheJiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；Center for Earth System Modeling and Prediction of China Meteorological Administration（中国气象局地球系统模拟与预测中心）
Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning
Fudan University（复旦大学）；East China Normal University（华东师范大学）；Singapore Management University（新加坡管理大学）；Singapore University of Technology and Design（新加坡科技设计大学）
MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
Fudan University（复旦大学）；MMLab, The Chinese University of Hong Kong（中大香港人工智能实验室）；Shanghai AI Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Nanjing University（南京大学）
Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing
School of Computer Science, Wuhan University, China（武汉大学计算机学院）；Zhongguancun Academy, China（中关村学院）；School of Computer Science, Chongqing University, China（重庆大学计算机学院）
Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling
Interdisciplinary Artificial Intelligence Research Institute, Wuhan College（交叉学科人工智能研究 institute，武汉学院）；School of Computer and Artificial Intelligence, Wuhan University of Technology（计算机与人工智能学院，武汉理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sanya Science and Education Innovation Park, Wuhan University of Technology（三亚科学教育创新园，武汉理工大学）；Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Mathematics and Statistics, Northwestern Polytechnical University（数学与统计学院，西北工业大学）；Institute of Artificial Intelligence (TeleAI) of China Telecom（中国电信人工智能研究所（TeleAI））
Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
AIML, University of Adelaide（AIML，阿德莱德大学）；Monash University（墨尔本大学）；Australian National University（澳大利亚国立大学）；Tiktok, Australia（Tiktok，澳大利亚）；Zhejiang University（浙江大学）
Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongke Fanyu Technology Co., Ltd（中科泛语科技有限公司）
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Yale University（耶鲁大学）
Judge Before Answer: Can MLLM Discern the False Premise in Question?
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Inner Mongolia Research Institute, Shanghai Jiao Tong University（上海交通大学内蒙古研究院）
MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所认知与决策智能复杂系统重点实验室）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学交叉学科学院）
MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；Baidu Inc.（百度公司）；University of California（加州大学）；University of Science and Technology of China（中国科学技术大学）
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University（哈杰泰佩大学）；Aselsan Research（阿塞尔桑研究院）；Koç University（科克大学）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学医学中心，阿姆斯特丹大学）；Amsterdam Public Health（阿姆斯特丹公共卫生研究院）
NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities
School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进技术学院）；Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of Deep Perception Technology, JITRI（深度感知技术研究所）
