How to accurately and efficiently assess AI-generated images (AIGIs) remains a critical challenge for generative models. Given the high costs and extensive time commitments required for user studies, many researchers have turned towards employing large multi-modal models (LMMs) as AIGI evaluators, the precision and validity of which are still questionable. Furthermore, traditional benchmarks often utilize mostly natural-captured content rather than AIGIs to test the abilities of LMMs, leading to a noticeable gap for AIGIs. Therefore, we introduce A-Bench in this paper, a benchmark designed to diagnose whether LMMs are masters at evaluating AIGIs. Specifically, A-Bench is organized under two key principles: 1) Emphasizing both high-level semantic understanding and low-level visual quality perception to address the intricate demands of AIGIs. 2) Various generative models are utilized for AIGI creation, and various LMMs are employed for evaluation, which ensures a comprehensive validation scope. Ultimately, 2,864 AIGIs from 16 text-to-image models are sampled, each paired with question-answers annotated by human experts, and tested across 18 leading LMMs. We hope that A-Bench will significantly enhance the evaluation process and promote the generation quality for AIGIs. The benchmark is available at https://github.com/Q-Future/A-Bench.
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；School of Humanities, Shanghai Jiao Tong University（上海交通大学人文学院）
VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Beijing Key Laboratory of Artificial Intelligence for Education（北京人工智能教育重点实验室）；Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education（教育部智能技术与教育应用工程研究中心）
How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Northeastern University（东北大学）；National University of Defense Technology（国防科技大学）；Shandong University（山东大学）；BNRist, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
MIBench: Evaluating LMMs on Multimodal Interaction
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China（中国人民大学人工智能学院，北京，中国）；Beijing Key Laboratory of Research on Large Models（大型模型研究关键实验室）；Beihang University, Beijing, China（北京航空航天大学，北京，中国）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室，上海，中国）
LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs
Institute of Image Communication and Network Engineering, Shanghai JiaoTong University（上海交通大学图像通信与网络工程研究所）；University of Electronic and Science Technology of China（电子科技大学）；Nanyang Technological University（南洋理工大学）
MDIT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Models
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）；University of Barcelona（巴塞罗那大学）
SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation
National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language
organization= Shien-Ming Wu School of Intelligent Manufacturing, South China University of Technology , city= Guangzhou , postcode= 511442 , country= China；organization= Artificial Intelligence Department, Guangdong CAS Angels Biotechnology Co. Ltd. , city= Foshan , country= China；organization= Department of Software Engineering, Faculty of Electrical \& Computer Engineering, University of Engineering \& Technology , city= Peshawar , postcode= 25000 , country= Pakistan
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
Salesforce AI Research（Salesforce AI研究院）；Hong Kong Baptist University（香港浸会大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；The Australian National University（澳大利亚国立大学）
OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks
Shanghai Key Lab of Intell. Info. Processing, Fudan University（上海智能信息处理关键实验室，复旦大学）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）
We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
Beijing University of Posts and Telecommunications（北京邮电大学）；Wechat（腾讯微信）；Tencent Inc.（腾讯公司）；Huazhong University of Science and Technology（华中科技大学）；Beijing Institute of Technology（北京理工大学）
Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models
Joint SDU-NTU Centre for Artificial Intelligence Research&School of Software（山东大学与南京工业大学人工智能研究中心及软件学院）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiaotong University（上海交通大学）；Nanjing University（南京大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Jiangnan University（江南大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Shanghai AI Laboratory（上海人工智能实验室）
SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；School of Computer Science（计算机科学学院）；Baidu Inc.（百度公司）；School of Computer Science, Wuhan University（武汉大学计算机学院）
MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
Hong Kong Baptist University（香港 Baptist 大学）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港 Baptist 大学）；National University of Singapore（新加坡国立大学）；Beijing Normal University（北京师范大学）；Shanghai Jiao Tong University（上海交通大学）
BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models
Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）；Northeastern University（东北大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Zhejiang University（浙江大学）
VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Edinburgh（爱丁堡大学）；Khalifa University（哈利法大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
