Despite the advancements and impressive performance of Multimodal Large Language Models (MLLMs) on benchmarks, their effectiveness in real-world, long-context, and multi-image tasks is unclear due to the benchmarks' limited scope. Existing benchmarks often focus on single-image and short-text samples, and when assessing multi-image tasks, they either limit the image count or focus on specific task (e.g time-series captioning), potentially obscuring the performance challenges of MLLMs. To address these limitations, we introduce MileBench, a pioneering benchmark designed to test the MultImodal Long-contExt capabilities of MLLMs. This benchmark comprises not only multimodal long contexts, but also multiple tasks requiring both comprehension and generation. We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs' long-context adaptation capacity and their ability to complete tasks in long-context scenarios. Our experimental results, obtained from testing 22 models, revealed that while the closed-source GPT-4o outperforms others, most open-source MLLMs struggle in long-context situations. Interestingly, the performance gap tends to widen with an increase in the number of images. We strongly encourage an intensification of research efforts towards enhancing MLLMs' long-context capabilities, especially in scenarios involving multiple images.
Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks
Bytedance（字节跳动）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫人工智能大学）；Stanford University（斯坦福大学）；University of Michigan（密歇根大学）
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM沃森人工智能实验室，IBM研究）；Stony Brook University（石溪大学）；Brookhaven National Laboratory（布鲁赫斯国家实验室）
Beyond RGB: Benchmarking and Enhancing MLLMs for Hyperspectral Image Understanding via Training-Free Reasoning Framework
Sun Yat-sen University（中山大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；China Agricultural University（中国农业大学）；Southwest Jiaotong University（西南交通大学）；Southwest University（西南大学）；National Supercomputing Center in Shenzhen（国家超级计算深圳中心）
HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models
National Engineering Research Center for Software Engineering, Peking University（北京大学软件工程国家工程研究中心）；Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院）；Tsinghua University（清华大学）；Chinese Academy of Sciences（中国科学院）；University of British Columbia（不列颠哥伦比亚大学）；Renmin University of China（中国人民大学）
GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Prediction
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究院）；School of Humanities, Shanghai Jiao Tong University（上海交通大学人文学院）；Shanghai AI Laboratory（上海人工智能实验室）
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学计算机科学与工程学院）；College of AI, Tsinghua University（清华大学人工智能学院）
FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs
Shenzhen University（深圳大学）；Tsinghua University（清华大学）；The Hong Kong Polytechnic University（香港理工大学）；City University of Hong Kong（香港城市大学）；Sun Yat-sen University（中山大学）
MMR-AD: A Large-Scale Multimodal Dataset for Benchmarking General Anomaly Detection with Multimodal Large Language Models
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）
PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；WeChat AI, Tencent Inc., China（微信AI，腾讯公司，中国）
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
School of Informatics, Xiamen University, China（厦门大学信息学院）；Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China（福建省-台湾非物质文化遗产数字化保护与智能处理重点实验室（XMU），文化和旅游部，中国）；National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China（教育与教学媒体语言资源监测与研究中心，厦门大学，中国）
IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting
MAIS, Institute of Automation（自动化研究所信息处理中心）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Research Center of Aerospace Information, Institute of Automation（自动化研究所航空信息研究中心）
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
School of Computer Science, South China Normal University（华南师范大学计算机科学学院）；School of Information Technology in Education, South China Normal University（华南师范大学教育信息技术学院）
RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios
Department of Electronic Engineering, BNRist, Tsinghua University（清华大学电子工程系、北京信息科学与技术国家研究中心）；Zhongguancun Academy（中关村学院）；Amap, Alibaba Group（阿里巴巴集团高德地图）
MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts
Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Nankai University（南开大学）；Beijing Institute of Technology（北京理工大学）；Baichuan Inc.（百度文心）
Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping
The Chinese University of Hong Kong, China（香港中文大学）；Singapore Management University, Singapore（新加坡管理学院）；Renmin University of China, China（中国人民大学）；Tsinghua University, China（清华大学）；Southwest University, China（西南大学）
