Significant advancements in Large Multimodal Models (LMMs) have enabled them to tackle complex problems involving visual-mathematical reasoning. However, their ability to identify geometric elements remains underexplored. To address this gap, we introduce Tangram, a novel benchmark designed to evaluate the performance of LMMs on geometric element recognition. Tangram comprises 1,080 diverse geometric diagrams sourced from primary and secondary school exams, competitions, and textbooks, ranging from simple geometric shapes to complex combinations. Each diagram is paired with four questions, resulting in 4,320 visual-question-answer pairs. Unlike existing benchmarks that emphasize higher-level cognition and reasoning, Tangram focuses on understanding geometric elements, requiring models to perform a ``simple yet challenging" counting task. Systematic evaluation of 13 prominent LMMs, such as GPT-4o and Claude 3.5 Sonnet, reveals that these models face significant challenges even in seemingly straightforward tasks. The top-performing model achieves an accuracy of only 53.0%, highlighting a substantial gap compared to human performance. These findings underscore the limitations of current multimodal AI systems in handling basic perception tasks and serve to inspire the development of the next generation of expert-level multimodal foundational models. The data and code will be released soon.
UCLA Center for Vision, Cognition, Learning, and Autonomy（加州大学洛杉矶分校视觉、认知、学习与自主中心）；Stevens Institute of Technology（史蒂文斯理工学院）
Current pre-training methods in computer vision focus on natural images in the daily-life context. However, abstract diagrams such as icons and symbols are common and important in the real world. This work is inspired by Tangram, a game that requires replicating an abstract pattern from seven dissected shapes. By recording human experience in solving tangram puzzles, we present the Tangram dataset and show that a pre-trained neural model on the Tangram helps solve some mini visual tasks based on low-resolution vision. Extensive experiments demonstrate that our proposed method generates intelli…
MIBench: Evaluating LMMs on Multimodal Interaction
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China（中国人民大学人工智能学院，北京，中国）；Beijing Key Laboratory of Research on Large Models（大型模型研究关键实验室）；Beihang University, Beijing, China（北京航空航天大学，北京，中国）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室，上海，中国）
ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large Multimodal Models
AI Institute, University of South Carolina（南卡罗来纳大学AI研究所）；Guru Gobind Singh Indraprastha University（古鲁·戈宾德·辛格·因德拉普拉斯塔大学）；Vellore Institute of Technology（韦洛尔理工大学）；Indian Institute of Technology (BHU)（印度理工学院（瓦拉纳西印度大学））；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of California Santa Cruz（加州大学圣克鲁兹分校）；Amazon Web Services（亚马逊网络服务）；Meta；Amazon GenAI（亚马逊生成式人工智能部门）
In this paper we introduce vSTS, a new dataset for measuring textual similarity of sentences using multimodal information. The dataset is comprised by images along with its respectively textual captions. We describe the dataset both quantitatively and qualitatively, and claim that it is a valid gold standard for measuring automatic multimodal textual similarity systems. We also describe the initial experiments combining the multimodal information.
IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
Beihang University（北京航空航天大学）；Alibaba Group（阿里巴巴集团）；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
School of Medicine, Yale University（耶鲁大学医学院）；School of Computing, Australian National University（澳大利亚国立大学计算机学院）；School of Engineering, Imperial College London（伦敦帝国理工学院工程学院）；School of Computing, University of Georgia（佐治亚大学计算机学院）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨秀隆医学学院）；National Eye Institute, National Institutes of Health（美国国立卫生研究院眼科研究所）；National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算机技术学院）
Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Harvard Medical School（哈佛医学院）；Stanford University（斯坦福大学）；Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）
V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments
Shanghai Jiao Tong University（上海交通大学）；South China University of Technology（华南理工大学）；Fudan University（复旦大学）；Microsoft Research Asia（微软亚洲研究院）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；East China Normal University（华东师范大学）
MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；College of Information and Electrical Engineering, China Agricultural University（中国农业大学信息与电气工程学院）
VisNumBench: Evaluating Number Sense of Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济发展实验室）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际 Graduate School）；Shenzhen Technology University（深圳技术大学）
FaceXBench: Evaluating Multimodal LLMs on Face Understanding
Department of Computer Science, Johns Hopkins University（计算机科学系，约翰霍普金斯大学）；Department of Electrical and Computer Engineering, Johns Hopkins University（电气与计算机工程系，约翰霍普金斯大学）
AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
Hong Kong University of Science and Technology（香港理工大学）；Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Pinterest Inc.（Pinterest公司）；Johns Hopkins University（约翰斯·霍普金斯大学）
In this paper, we focus on training and evaluating effective word embeddings with both text and visual information. More specifically, we introduce a large-scale dataset with 300 million sentences describing over 40 million images crawled and downloaded from publicly available Pins (i.e. an image with sentence descriptions uploaded by users) on Pinterest. This dataset is more than 200 times larger than MS COCO, the standard large-scale image dataset with sentence descriptions. In addition, we construct an evaluation dataset to directly assess the effectiveness of word embeddings in terms of fi…
DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms
College of Information and Engineering, Minzu University of China（中国民族大学信息与工程学院）；Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China（教育部民族语言智能分析与安全治理重点实验室）；College of Information and Communication Engineering, Harbin Engineering University（哈尔滨工程大学信息与通信工程学院）；Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University（清华大学人工智能研究院计算机科学与技术系）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；Shanghai Innovation Institute（上海创新研究院）；City University of Hong Kong（香港城市大学）
MDIT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Models
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）；University of Barcelona（巴塞罗那大学）
INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurance
School of Computer Science Fudan University（复旦大学计算机科学学院）；Department of Computer Science University of Rochester（罗切斯特大学计算机科学系）；School of Economics Fudan University（复旦大学经济学院）
MIBench: Evaluating Multimodal Large Language Models over Multiple Images
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Alibaba Group（阿里巴巴集团）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳分校）；National University of Singapore（新加坡国立大学）
Evaluating point-light biological motion in multimodal large language models
Psychiatry and Biobehavioral Sciences, UCLA（乌尔拉克大学精神病学与生物行为科学系）；Brain and Creativity Institute, USC（美国大学脑与创造力研究所）；Google DeepMind, Zurich（谷歌深度Mind瑞士分公司）
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Fudan University（复旦大学）；NJU-LINK Team, Nanjing University（南京大学NJU-LINK团队）；University of Science and Technology of China（中国科学技术大学）；Kuaishou Technology（快手科技）
MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Image
College of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Computer and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；CCNU（中国地质大学）
Evaluating Compositional Scene Understanding in Multimodal Generative Models
Department of Psychology, University of California, Los Angeles（加州大学洛杉矶分校心理学系）；Microsoft Research, NYC（微软研究院纽约分部）；Air Force Research Laboratory（美国空军研究实验室）；Department of Statistics, University of California, Los Angeles（加州大学洛杉矶分校统计学系）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for vision-and-language summarization. We propose CLIPBERTScore, a simple weighted combination of CLIPScore and BERTScore to leverage the robustness and strong factuality detection performance between image-summary and document-summary, respectively. Next, due to the lack of meta-evaluation benchmarks to evaluate the quality of multimodal factuality metrics, we collect human judgments of f…
MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；USTC（中国科学技术大学）；RIT（罗切斯特理工学院）；HIT（哈尔滨工业大学）；WHU（武汉大学）；MBZUAI（Mohamed bin Zayed 人工智能大学）；NUS（新加坡国立大学）
We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answ…
