arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.04833cs.CVcs.AIcs.CL
收藏

E$^2$AT: 通过动态联合优化实现多模态对抗防御

E$^2$AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models

School of Cyber Science and Engineering, Nanjing University of Science and Technology, China(南京理工大学信息科学与工程学院);HKUST(GZ) and INSAIT, Sofia University St. Kliment Ohridski(香港科技大学(广州)及INSAIT,索菲亚大学圣克莱门特·奥赫里迪斯学院);College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算与数据科学学院)

Liming Lu, Xiang Gu, Shuchao Pang, Siyuan Liang, Haotian Zhu, Xiyu Zeng, Xu Zheng, Yongbin Zhou

AI总结:

E$^2$AT通过动态联合优化提升多模态大语言模型对对抗攻击的鲁棒性,实验显示其在文本和图像模态上性能优于现有方法34%。

摘要:

针对多模态大语言模型(MLLMs)在对抗攻击下的鲁棒性问题,现有方法在高效调整大规模权重参数和确保跨视觉与文本模态的鲁棒性方面面临关键挑战。为此,我们提出了一个高效端到端对抗训练(E$^2$AT)框架,用于视觉和文本对抗攻击。具体而言,在视觉方面,E$^2$AT集成了一个高效的基于投影器的对抗模块,通过特征层面对攻击样本进行对齐。在训练目标上,我们提出了一种动态联合多模态优化(DJMO)策略,通过动态调整正常和对抗目标之间的权重来增强对对抗攻击的泛化能力。通过在三个主流MLLMs上进行五种主要对抗攻击方法的大量实验,结果表明,我们的E$^2$AT在文本和图像模态上平均超越现有基线34%,同时保持任务性能。进一步评估现实世界智能系统的应用表明,E$^2$AT在实际应用中的可行性,为开发更安全可靠的多模态系统铺平了道路。我们的代码可在https://anonymous.4open.science/r/E2AT_568上获取。

英文摘要:

Research endeavors have been made in learning robust Multimodal Large Language Models (MLLMs) against jailbreak attacks. However, existing methods for improving MLLMs' robustness still face critical challenges: \ding{172} how to efficiently tune massive weight parameters and \ding{173} how to ensure robustness against attacks across both visual and textual modalities. To this end, we propose an \textbf{E}fficient \textbf{E}nd-to-end \textbf{A}dversarial \textbf{T}raining (E$^2$AT) framework for both visual and textual adversarial attacks. Specifically, for the visual aspect, E$^2$AT incorporates an efficient projector-based AT module that aligns the attack samples at the feature level. For training objectives, we propose a Dynamic Joint Multimodal Optimization (DJMO) strategy to enhance generalization ability against jailbreak attacks by dynamically adjusting weights between normal and adversarial objectives. Extensive experiments are conducted with five major jailbreak attack methods across three mainstream MLLMs. Results demonstrate that our E$^2$AT achieves the state-of-the-art performance, outperforming existing baselines by an average margin of 34\% across text and image modalities, while maintaining clean task performance. Furthermore, evaluations of real-world embodied intelligent systems highlight the practical applicability of E$^2$AT, paving the way for the development of more secure and reliable multimodal systems. Our code is available on \href{https://anonymous.4open.science/r/E2AT_568}{\textcolor{red}{https://anonymous.4open.science/r/E2AT\_568}}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑