Evaluating point-light biological motion in multimodal large language models
Psychiatry and Biobehavioral Sciences, UCLA（乌尔拉克大学精神病学与生物行为科学系）；Brain and Creativity Institute, USC（美国大学脑与创造力研究所）；Google DeepMind, Zurich（谷歌深度Mind瑞士分公司）
Akila Kadambi, Marco Iacoboni, Lisa Aziz-Zadeh, Srini Narayanan
Humans can extract rich semantic information from minimal visual cues, as demonstrated by point-light displays (PLDs), which consist of sparse sets of dots localized to key joints of the human body. This ability emerges early in development and is largely attributed to human embodied experience. Since PLDs isolate body motion as the sole source of meaning, they represent key stimuli for testing the constraints of action understanding in these systems. Here we introduce ActPLD, the first benchmark to evaluate action processing in MLLMs from human PLDs. Tested models include state-of-the-art proprietary and open-source systems on single-actor and socially interacting PLDs. Our results reveal consistently low performance across models, introducing fundamental gaps in action and spatiotemporal understanding.
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Fudan University（复旦大学）；NJU-LINK Team, Nanjing University（南京大学NJU-LINK团队）；University of Science and Technology of China（中国科学技术大学）；Kuaishou Technology（快手科技）
MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Image
College of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Computer and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；CCNU（中国地质大学）
Evaluating Compositional Scene Understanding in Multimodal Generative Models
Department of Psychology, University of California, Los Angeles（加州大学洛杉矶分校心理学系）；Microsoft Research, NYC（微软研究院纽约分部）；Air Force Research Laboratory（美国空军研究实验室）；Department of Statistics, University of California, Los Angeles（加州大学洛杉矶分校统计学系）
MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；USTC（中国科学技术大学）；RIT（罗切斯特理工学院）；HIT（哈尔滨工业大学）；WHU（武汉大学）；MBZUAI（Mohamed bin Zayed 人工智能大学）；NUS（新加坡国立大学）
Multimodal Large Language Models for Bioimage Analysis
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V.（莱布尼茨分析科学研究所）
Tsinghua University（清华大学）；Meituan Inc（美团公司）；Central South University（中南大学）；Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Nanyang Technological University（南洋理工大学）
We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answ…
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；The University of Adelaide（阿德莱德大学）；Zhejiang University（浙江大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions
Department of Intelligent Software, Sungkyunkwan University（智能软件系，成均馆大学）；Department of Computer Science and Engineering, Sungkyunkwan University（计算机科学与工程系，成均馆大学）
Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environments
State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence（通用人工智能国家重点实验室、北京通用人工智能研究院）；School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Key Laboratory of Machine Perception (Ministry of Education), Peking University（心理与认知科学学院及北京行为与心理健康重点实验室、机器感知重点实验室（教育部））；School of Intelligence Science and Technology, Peking University（智能科学与技术学院）
MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis
The First Affiliated Hospital with Nanjing Medical University（南京医科大学第一附属医院）；Jiangsu Province Hospital（江苏省医院）；Huazhong University of Science Technology（华中科技大学）；Fudan University（复旦大学）
TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation
School of Transportation（交通学院）；Southeast University（东南大学）；School of Artificial Intelligence and Computer Science（人工智能与计算机科学学院）；Jiangnan University（江南大学）；Department of Civil and Environmental Engineering（土木与环境工程系）；Hong Kong Polytechnic University（香港理工大学）
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Yale University（耶鲁大学）
AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
University of Chinese Academy of Sciences（中国科学院大学）；National Astronomical Observatories, Chinese Academy of Sciences（中国科学院国家天文台）；Zhejiang Laboratory（之江实验室）；Tianjin University（天津大学）
Purdue University（普渡大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California Los Angeles（加州大学洛杉矶分校）；University of Connecticut（康涅狄格大学）；University of California Berkeley（加州大学伯克利分校）
MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering
School of Computing and Augmented Intelligence（计算与增强智能学院）；Arizona State University（亚利桑那州立大学）；Department of Computer Science（计算机科学系）；International Institute of Information Technology（国际信息科技研究所）
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University（哈杰泰佩大学）；Aselsan Research（阿塞尔桑研究院）；Koç University（科克大学）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学医学中心，阿姆斯特丹大学）；Amsterdam Public Health（阿姆斯特丹公共卫生研究院）
