The success of multimodal data fusion in deep learning appears to be attributed to the use of complementary in-formation between multiple input data. Compared to their predictive performance, relatively less attention has been devoted to the robustness of multimodal fusion models. In this paper, we investigated whether the current multimodal fusion model utilizes the complementary intelligence to defend against adversarial attacks. We applied gradient based white-box attacks such as FGSM and PGD on MFNet, which is a major multispectral (RGB, Thermal) fusion deep learning model for semantic segmentation. We verified that the multimodal fusion model optimized for better prediction is still vulnerable to adversarial attack, even if only one of the sensors is attacked. Thus, it is hard to say that existing multimodal data fusion models are fully utilizing complementary relationships between multiple modalities in terms of adversarial robustness. We believe that our observations open a new horizon for adversarial attack research on multimodal data fusion.
Investigating Traffic Accident Detection Using Multimodal Large Language Models
Embedded Systems Group (Dept.-E)（嵌入式系统组）；Virtual Vehicle Research GmbH（虚拟车辆研究公司）；Control Systems Group (Dept.-E)（控制系统组）；Institute of Visual Computing（视觉计算研究所）；Graz University of Technology（格拉茨技术大学）
In this paper, we describe how the electronic rolling shutter in CMOS image sensors can be exploited using a bright, modulated light source (e.g., an inexpensive, off-the-shelf laser), to inject fine-grained image disruptions. We demonstrate the attack on seven different CMOS cameras, ranging from cheap IoT to semi-professional surveillance cameras, to highlight the wide applicability of the rolling shutter attack. We model the fundamental factors affecting a rolling shutter attack in an uncontrolled setting. We then perform an exhaustive evaluation of the attack's effect on the task of object…
Massachusetts Institute of Technology（麻省理工学院）；Bosch Center for AI（博世人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）
Beyond achieving high performance across many vision tasks, multimodal models are expected to be robust to single-source faults due to the availability of redundant information between modalities. In this paper, we investigate the robustness of multimodal neural networks against worst-case (i.e., adversarial) perturbations on a single modality. We first show that standard multimodal fusion models are vulnerable to single-source adversaries: an attack on any single modality can overcome the correct information from multiple unperturbed modalities and cause the model to fail. This surprising vul…
Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRI
department of Radiology and Nuclear Medicine, Xuanwu Hospital, Capital Medical University（放射医学与核医学科，宣武医院，首都医科大学）；Beijing Key Laboratory of Magnetic Resonance Imaging and Brain Informatics（北京磁共振成像与脑信息学重点实验室）；Beijing United Imaging Research Institute of Intelligent Imaging（北京智能成像联合研究院）
Investigating and unmasking feature-level vulnerabilities of CNNs to adversarial perturbations
Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR)（生物信息学研究所（BII），科技研究局（A*STAR））；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Biological Sciences, Nanyang Technological University（南洋理工大学生物科学学院）；Singapore Eye Research Institute (SERI)（新加坡眼科研究所（SERI））；Singapore Institute for Clinical Sciences（新加坡临床科学研究院）
Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model
Southeast University（东南大学）；University of Southern California（南加州大学）；University of Washington（华盛顿大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；ZOZO Research（ZOZO研究院）；Xiaohongshu Inc（小红书公司）；Peking University（北京大学）；Waseda University（早稻田大学）；Monash University（莫纳什大学）
Chittagong University of Engineering and Technology（吉大港工程技术大学）
In the past few years, the meme has become a new way of communication on the Internet. As memes are the images with embedded text, it can quickly spread hate, offence and violence. Classifying memes are very challenging because of their multimodal nature and region-specific interpretation. A shared task is organized to develop models that can identify trolls from multimodal social media memes. This work presents a computational model that we have developed as part of our participation in the task. Training data comes in two forms: an image with embedded Tamil code-mixed text and an associated …
Convolutional neural networks (CNN) have been more and more applied in mobile robotics such as intelligent vehicles. Security of CNNs in robotics applications is an important issue, for which potential adversarial attacks on CNNs are worth research. Pooling is a typical step of dimension reduction and information discarding in CNNs. Such information discarding may result in mis-deletion and mis-preservation of data features which largely influence the output of the network. This may aggravate the vulnerability of CNNs to adversarial attacks. In this paper, we conduct adversarial attacks on CNN…
Threats to Arabic Handwriting Recognition: Investigating Black-Box Adversarial Attacks on embedded ConvNet models
Systems theory and informatics laboratory（系统理论与信息学实验室）；Moulay Ismail University of Meknes（穆莱·艾息姆大学）；Department of Computer Science（计算机科学系）；EST of Sidi Bennour（西迪·本努尔工程与技术学院）；Chouaib Doukkali University（侯赛因·杜克利大学）；Faculty of Education Sciences（教育科学学院）；University Mohammed V（穆莱·维大学）；Laboratory of Computer Science and Applications（计算机科学与应用实验室）
Manipulating Multimodal Agents via Cross-Modal Prompt Injection
Beihang University（北洋大学）；National University of Singapore（新加坡国立大学）；Huazhong University of Science and Technology（华中科技大学）；Henan University of Science and Technology（河南科技大学）
Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形学研究所）；TU Darmstadt（达姆施塔特工业大学）
Investigating new methods of creating face morphing attacks is essential to foresee novel attacks and help mitigate them. Creating morphing attacks is commonly either performed on the image-level or on the representation-level. The representation-level morphing has been performed so far based on generative adversarial networks (GAN) where the encoded images are interpolated in the latent space to produce a morphed image based on the interpolated vector. Such a process was constrained by the limited reconstruction fidelity of GAN architectures. Recent advances in the diffusion autoencoder model…
Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhejiang University（浙江大学）；Beijing University of Technology（北京工业大学）
Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
Technical University of Munich, Germany（慕尼黑技术大学，德国）；Helmholtz Munich, Munich Center for Machine Learning, Germany（海德堡慕尼黑，慕尼黑机器学习中心，德国）；University of Tübingen, Tübingen AI Center, Germany（图宾根大学，图宾根人工智能中心，德国）；University of Trento, Italy（特伦托大学，意大利）；Beijing University of Posts and Telecommunications, China（北京邮电大学，中国）
Investigating Adversarial Robustness of Multi-modal Large Language Models
Mohamed Bin Zayed University of AI, UAE（穆罕默德·本·扎耶德人工智能大学，阿联酋）；Khalifa University, UAE（哈利法大学，阿联酋）；Australian National University, Australia（澳大利亚国立大学，澳大利亚）
This work examines the vulnerability of multimodal (image + text) models to adversarial threats similar to those discussed in previous literature on unimodal (image- or text-only) models. We introduce realistic assumptions of partial model knowledge and access, and discuss how these assumptions differ from the standard "black-box"/"white-box" dichotomy common in current literature on adversarial attacks. Working under various levels of these "gray-box" assumptions, we develop new attack methodologies unique to multimodal classification and evaluate them on the Hateful Memes Challenge classific…
Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
School of Computing, Macquarie University（麦考瑞大学计算学院）；Microsoft Corporation（微软公司）；School of Information Technology, Deakin University（迪肯大学信息技术学院）；Data61, CSIRO（联邦科学与工业研究组织Data61研究院）
Investigating the Viability of Employing Multi-modal Large Language Models in the Context of Audio Deepfake Detection
Indian Institute of Technology, Ropar, India（印度理工学院罗帕尔分校）；Machine Intelligence Group, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, India（比拉理工科学院帕利尼 Hyderabad 分校机器智能小组）；Monash University, Melbourne, Australia（墨尔本大学）
University of Maryland, College Park（马里兰大学帕克分校）；SRI International（SRI国际研究所）
The success of deep learning has enabled advances in multimodal tasks that require non-trivial fusion of multiple input domains. Although multimodal models have shown potential in many problems, their increased complexity makes them more vulnerable to attacks. A Backdoor (or Trojan) attack is a class of security vulnerability wherein an attacker embeds a malicious secret behavior into a network (e.g. targeted misclassification) that is activated when an attacker-specified trigger is added to an input. In this work, we show that multimodal networks are vulnerable to a novel type of attack that …
