Multimodal foundation models that can holistically process text alongside images, video, audio, and other sensory modalities are increasingly used in a variety of real-world applications. However, it is challenging to characterize and study progress in multimodal foundation models, given the range of possible modeling decisions, tasks, and domains. In this paper, we introduce Holistic Evaluation of Multimodal Models (HEMM) to systematically evaluate the capabilities of multimodal foundation models across a set of 3 dimensions: basic skills, information flow, and real-world use cases. Basic multimodal skills are internal abilities required to solve problems, such as learning interactions across modalities, fine-grained alignment, multi-step reasoning, and the ability to handle external knowledge. Information flow studies how multimodal content changes during a task through querying, translation, editing, and fusion. Use cases span domain-specific challenges introduced in real-world multimedia, affective computing, natural sciences, healthcare, and human-computer interaction applications. Through comprehensive experiments across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g., basic skills, information flows, and use cases) that pose challenges to today's models, and (2) distill performance trends regarding how different modeling dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training, and instruction tuning objectives) influence performance. Our conclusions regarding challenging multimodal interactions, use cases, and tasks requiring reasoning and external knowledge, the benefits of data and model scale, and the impacts of instruction tuning yield actionable insights for future work in multimodal foundation models.
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Fudan University（复旦大学）；NJU-LINK Team, Nanjing University（南京大学NJU-LINK团队）；University of Science and Technology of China（中国科学技术大学）；Kuaishou Technology（快手科技）
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
DAMO Academy, Alibaba Group（达摩院，阿里巴巴集团）；Hupan Laboratory（湖畔实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Department of Radiology, The Affiliated Yangming Hospital of Ningbo University（宁波大学附属阳明医院放射科）；Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院，浙江大学）；Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University（清华长庚医院肝胆胰中心，清华大学临床医学院，清华医学，清华大学）；School of Software, Tsinghua University（清华大学软件学院）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
MeniOmni: A Structured Multimodal Benchmark for Holistic Meniscus Injury Assessment
1School of Electronics, Electrical Engineering；Computer Science, Queen's University Belfast, Belfast, UK；2Radiology Department, Affiliated Nantong Clinical College of Nantong University, Nantong First People's Hospital, School of Clinical Medicine, Nantong University, Nantong, Jiangsu, China；3School of Artificial Intelligence；Computer Science, Nantong University, Nantong, China；4Faculty of Data Science, City University of Macau, Macau, China；5Department of Chemistry, University of Oxford, Oxford, UK；6Orthopedics Department, Nantong First People's Hospital, Southeast University, Nantong, Jiangsu, China
MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models
The Chinese University of Hong Kong（香港中文大学）；Tongyi Lab（通义实验室）；SZTU（深圳技术大学）；Peking University（北京大学）；Tongji University（同济大学）；CASIA（中国科学院自动化所）；Tencent（腾讯）；UCLA（加州大学洛杉矶分校）；Westlake University（西湖大学）；NTU（国立大学）
OmniEarth-Bench: Towards Holistic Evaluation of Earth's Six Spheres and Cross-Spheres Interactions with Multimodal Observational Earth Data
National University of Defense Technology, China（国防科技大学）；Shanghai Artificial Intelligence Laboratory, China（上海人工智能实验室）；Beijing University of Posts and Telecommunications, China（北京邮电大学）；Zhejiang University, China（浙江大学）；Shanghai Jiao Tong University, China（上海交通大学）；Fudan University, China（复旦大学）；Sun Yat-sen University, China（中山大学）；Nanjing University, China（南京大学）；University of Science and Technology of China（中国科学技术大学）；Wuhan University, China（武汉大学）
MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；Zhejiang University（浙江大学）；Princeton University（普林斯顿大学）
From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理关键实验室，复旦大学计算机学院）；Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Singapore Management University（新加坡管理大学）
Microsoft Azure Cognitive Services Research（微软Azure认知服务研究院）
Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview. Most current pretraining methods, however, are limited to one or two modalities. We present i-Code, a self-supervised pretraining framework where users may flexibly combine the modalities of vision, speech, and language into unified and general-purpose vector representations. In this framework, data from each modality are first given to pretrained single-modality encoders. The encoder outputs are then integrated with a multimodal fusion network, which uses novel attention m…
Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Taipei Veterans General Hospital（台北荣民总医院）；National Yang Ming Chiao Tung University（国立阳明交通大学）；City University of Hong Kong（香港城市大学）
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, w…
Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding
Yanshan University（燕山大学）；Central South University（中南大学）；Zhejiang University（浙江大学）；Southwest University（西南大学）；Singapore Management University（新加坡管理大学）；Singapore University of Technology and Design（新加坡科技设计大学）
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Freedom AI；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Guangdong University of Technology（广东工业大学）
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real…
PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing Future Brain Education Technology Co., Ltd.（北京未来脑教育科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）
OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；The University of Hong Kong（香港大学）；CUHK MMLab（香港中文大学MMLab）；Xi’an Jiaotong University（西安交通大学）；Nanjing University（南京大学）；Harbin Institute of Technology（哈尔滨工业大学）
Does Unification Come at a Cost? Uni-SafeBench: A Safety Benchmark for Unified Multimodal Large Models
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
DERM-3R: A Resource-Efficient Multimodal Agents Framework for Dermatologic Diagnosis and Treatment in Real-World Clinical Settings
School of Pharmacy, Nanjing University of Chinese Medicine（南京中医药大学药学院）；Department of Dermatology, the Gulou Hospital of Traditional Chinese Medicine of Beijing（北京鼓楼中医医院皮肤科）；Infectious disease department, Dongfang Hospital, Beijing University of Chinese Medicine（北京中医药大学东方医院感染科）；College of Art and Science, university of Washington（华盛顿大学文理学院）；Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）；Fudan Unversity（复旦大学）；Zhejiang Lab（之江实验室）；University College London（伦敦大学学院）；Department of Dermatology, Inner Mongolia Hospital of Traditional Chinese Medicine（内蒙古自治区中医医院皮肤科）；International Campus of Zhejiang University（浙江大学国际校区）
Violence detection has been studied in computer vision for years. However, previous work are either superficial, e.g., classification of short-clips, and the single scenario, or undersupplied, e.g., the single modality, and hand-crafted features based multimodality. To address this problem, in this work we first release a large-scale and multi-scene dataset named XD-Violence with a total duration of 217 hours, containing 4754 untrimmed videos with audio signals and weak labels. Then we propose a neural network containing three parallel branches to capture different relations among video snippe…
Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Jiangsu University（江苏大学）；Griffith University（格里菲斯大学）；National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；Fujian University of Technology（福建理工大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；Tsinghua University（清华大学）；Technical University of Munich（慕尼黑工业大学）；Imperial College London（伦敦帝国学院）；Queen Mary University of London（伦敦玛丽女王大学）
True Multimodal In-Context Learning Needs Attention to the Visual Context
LMU Munich（慕尼黑大学）；Technical University of Munich（慕尼黑技术大学）；Siemens AG（西门子股份公司）；University of Science and Technology of China（中国科学技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Konrad Zuse School of Excellence in Reliable AI (relAI)（Konrad Zuse可靠性人工智能卓越学院）；University of Oxford（牛津大学）
