Multimodal image-text models have shown remarkable performance in the past few years. However, evaluating robustness against distribution shifts is crucial before adopting them in real-world applications. In this work, we investigate the robustness of 12 popular open-sourced image-text models under common perturbations on five tasks (image-text retrieval, visual reasoning, visual entailment, image captioning, and text-to-image generation). In particular, we propose several new multimodal robustness benchmarks by applying 17 image perturbation and 16 text perturbation techniques on top of existing datasets. We observe that multimodal models are not robust to image and text perturbations, especially to image perturbations. Among the tested perturbation methods, character-level perturbations constitute the most severe distribution shift for text, and zoom blur is the most severe shift for image data. We also introduce two new robustness metrics (\textbf{MMI} for MultiModal Impact score and \textbf{MOR} for Missing Object Rate) for proper evaluations of multimodal models. We hope our extensive study sheds light on new directions for the development of robust multimodal models. More details can be found on the project webpage: \url{https://MMRobustness.github.io}.
State Key Lab of Software Development Environment, Beihang University（北京航空航天大学软件开发环境国家重点实验室）；SenseTime（商汤科技）；Berkeley Artificial Intelligence Research Lab, University of California, Berkeley（加州大学伯克利分校伯克利人工智能研究实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Oxford（牛津大学）；JD Explore Academy（京东探索研究院）
Deep neural networks (DNNs) are vulnerable to adversarial noises, which motivates the benchmark of model robustness. Existing benchmarks mainly focus on evaluating defenses, but there are no comprehensive studies of how architecture design and training techniques affect robustness. Comprehensively benchmarking their relationships is beneficial for better understanding and developing robust DNNs. Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from n…
Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）
The state-of-the-art deep neural networks are vulnerable to common corruptions (e.g., input data degradations, distortions, and disturbances caused by weather changes, system error, and processing). While much progress has been made in analyzing and improving the robustness of models in image understanding, the robustness in video understanding is largely unexplored. In this paper, we establish a corruption robustness benchmark, Mini Kinetics-C and Mini SSV2-C, which considers temporal corruptions beyond spatial corruptions in images. We make the first attempt to conduct an exhaustive study on…
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo（滑铁卢大学）；Georgia Institute of Technology（佐治亚理工学院）；Hong Kong University of Science and Technology（香港科技大学）；Google DeepMind（谷歌DeepMind）
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）；Alibaba Group, Hangzhou, China（阿里巴巴集团）
University of California, Berkeley（加州大学伯克利分校）；Oregon State University（俄勒冈州立大学）
In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative cor…
MMR-AD: A Large-Scale Multimodal Dataset for Benchmarking General Anomaly Detection with Multimodal Large Language Models
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）
Southeast Missouri State University（东南密苏里州立大学）；University of Arkansas at Little Rock（阿肯色大学小石城分校）
Accuracies of deep learning (DL) classifiers are often unstable in that they may change significantly when retested on adversarial images, imperfect images, or perturbed images. This paper adds to the fundamental body of work on benchmarking the robustness of DL classifiers on defective images. To measure robust DL classifiers, previous research reported on single-factor corruption. We created comprehensive 69 benchmarking image sets, including a clean set, sets with single factor perturbations, and sets with two-factor perturbation conditions. The state-of-the-art two-factor perturbation incl…
University of Surrey（萨里大学）；West Virginia University（西弗吉尼亚大学）；Institut Telecom, Telecom and Management SudParis（法国国立电信学院南巴黎电信与管理学院）；CEA LIST（法国原子能与替代能源委员会 LIST 研究所）；University of Amsterdam（阿姆斯特丹大学）；Brandenburg University of Applied Sciences（勃兰登堡应用技术大学）；Otto-von-Guericke-University of Magdeburg（马格德堡奥托·冯·格里克大学）
Automatically verifying the identity of a person by means of biometrics is an important application in day-to-day activities such as accessing banking services and security control in airports. To increase the system reliability, several biometric devices are often used. Such a combined system is known as a multimodal biometric system. This paper reports a benchmarking study carried out within the framework of the BioSecure DS2 (Access Control) evaluation campaign organized by the University of Surrey, involving face, fingerprint, and iris biometrics for person authentication, targeting the ap…
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
National University of Singapore（新加坡国立大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；Hong Kong Baptist University（香港浸会大学）
MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
Technion – Israel Institute of Technology（技术ion – 以色列理工学院）；Prior Labs（Prior实验室）；NVIDIA；SODA Team, INRIA Saclay, Palaiseau（SODA团队，INRIA萨克莱，帕莱索）；University of Freiburg（弗赖堡大学）；Probabl；ELLIS Institute Tübingen（图宾根ELLIS研究所）
Can Multimodal LLMs See Science Instruction? Benchmarking Pedagogical Reasoning in K-12 Classroom Videos
Department of Computer Science, Drexel University（德雷塞尔大学计算机科学系）；Department of Teaching and Learning, Washington State University（华盛顿州立大学教学与学习系）；College of Chemistry, Beijing Normal University（北京师范大学化学学院）；Department of Computer Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）
University of California, Berkeley（加州大学伯克利分校）；Oregon State University（俄勒冈州立大学）
In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Unlike recent robustness research, this benchmark evaluates performance on commonplace corruptions not worst-case adversarial corruptions. We find that there are negligible changes in relative corruption robustness from AlexNet to ResNet classifiers, and we discover ways to enhance corruption robustness. Then we propose a new dataset called Icons-…
Omni-Fake: Benchmarking Unified Multimodal Social Media Deepfake Detection
University of Liverpool（利物浦大学）；University of Exeter（埃克塞特大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）
MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中国教育部，厦门大学）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）
EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Ability
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China（计算机科学与技术系，人工智能研究院，清华大学，北京，中国）；Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China（人工智能产业研究院（AIR），清华大学，北京，中国）；School of Management, Fudan University, Shanghai, China（管理学院，复旦大学，上海，中国）
UC San Diego（加州大学圣地亚哥分校）；Toyota Collaborative Safety Research Center（丰田协作安全研究中心）
Recent progress in autonomous and semi-autonomous driving has been made possible in part through an assortment of sensors that provide the intelligent agent with an enhanced perception of its surroundings. It has been clear for quite some while now that for intelligent vehicles to function effectively in all situations and conditions, a fusion of different sensor technologies is essential. Consequently, the availability of synchronized multi-sensory data streams are necessary to promote the development of fusion based algorithms for low, mid and high level semantic tasks. In this paper, we pro…
