VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Edinburgh（爱丁堡大学）；Khalifa University（哈利法大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
The recent developments in Large Multi-modal Video Models (Video-LMMs) have significantly enhanced our ability to interpret and analyze video data. Despite their impressive capabilities, current Video-LMMs have not been evaluated for anomaly detection tasks, which is critical to their deployment in practical scenarios e.g., towards identifying deepfakes, manipulated video content, traffic accidents and crimes. In this paper, we introduce VANE-Bench, a benchmark designed to assess the proficiency of Video-LMMs in detecting and localizing anomalies and inconsistencies in videos. Our dataset comprises an array of videos synthetically generated using existing state-of-the-art text-to-video generation models, encompassing a variety of subtle anomalies and inconsistencies grouped into five categories: unnatural transformations, unnatural appearance, pass-through, disappearance and sudden appearance. Additionally, our benchmark features real-world samples from existing anomaly detection datasets, focusing on crime-related irregularities, atypical pedestrian behavior, and unusual events. The task is structured as a visual question-answering challenge to gauge the models' ability to accurately detect and localize the anomalies within the videos. We evaluate nine existing Video-LMMs, both open and closed sources, on this benchmarking task and find that most of the models encounter difficulties in effectively identifying the subtle anomalies. In conclusion, our research offers significant insights into the current capabilities of Video-LMMs in the realm of anomaly detection, highlighting the importance of our work in evaluating and improving these models for real-world applications. Our code and data is available at https://hananshafi.github.io/vane-benchmark/
BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models
Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）；Northeastern University（东北大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Zhejiang University（浙江大学）
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；School of Humanities, Shanghai Jiao Tong University（上海交通大学人文学院）
LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models
Sun Yat-sen University（中山大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Macau University of Science and Technology（澳门科技大学）；Jiangnan University（江南大学）；The Ohio State University（俄亥俄州立大学）；Fujian University of Traditional Chinese Medicine（福建中医药大学）
LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Models
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；Department of Computing, Hong Kong Polytechnic University（香港理工大学计算机系）
ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
University of Cambridge（剑桥大学）；University of Alberta（阿尔伯塔大学）；The University of Hong Kong（香港大学）；University of Oxford（牛津大学）；Northeastern University（东北大学）；Astadeus；College of Southern Maryland（马里兰州南部学院）；University of Geneva（日内瓦大学）；University of Oregon（俄勒冈大学）
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages
University of Central Florida（佛罗里达中央大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Amazon（亚马逊）；Aalto University（艾尔托大学）；Australian National University（澳大利亚国立大学）；Linköping University（林奈大学）；Ehtiopian AI Institute（埃塞俄比亚AI研究所）；Johns Hopkins University（约翰·霍普金斯大学）；Aalborg University（奥尔堡大学）；University of the West of England（英国西部大学）；Alpinference；University of Maryland（马里兰大学）；HUN-REN Institute for CS and Control（HUN-REN计算机科学与控制研究所）；Queen Mary University of London（伦敦玛丽女王大学）；Chongqing University（重庆大学）；University（大学）
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）；Alibaba Group, Hangzhou, China（阿里巴巴集团）
Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models
The University of Tokyo（东京大学）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；Duke University（杜克大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；LY Corporation（LY公司）；Tokyo University of Science（东京科学大学）
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community（多模态艺术投影研究共同体）；Hong Kong University of Science and Technology（香港科技大学）；University of Waterloo（滑铁卢大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peking University（北京大学）；The Hong Kong Polytechnic University（香港理工大学）；Waseda University（早稻田大学）；University of Manchester（曼彻斯特大学）
MIBench: Evaluating LMMs on Multimodal Interaction
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China（中国人民大学人工智能学院，北京，中国）；Beijing Key Laboratory of Research on Large Models（大型模型研究关键实验室）；Beihang University, Beijing, China（北京航空航天大学，北京，中国）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室，上海，中国）
GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing
KTH Royal Institute of Technology（瑞典皇家理工学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Technical University of Munich（慕尼黑工业大学）
VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
