Multimodal Pragmatic Jailbreak on Text-to-image Models
LMU Munich, Germany(慕尼黑大学);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心);Saarland University, Germany(萨尔兰大学);Max Planck Institute for Informatics, Germany(马克斯·普朗克信息研究所);Cornell University, USA(康奈尔大学);University of Oxford, UK(牛津大学)
Tong Liu, Zhixin Lai, Jiawen Wang, Gengyuan Zhang, Shuo Chen, Philip Torr, Vera Demberg, Volker Tresp, Jindong Gu
Diffusion models have recently achieved remarkable advancements in terms of image quality and fidelity to textual prompts. Concurrently, the safety of such generative models has become an area of growing concern. This work introduces a novel type of jailbreak, which triggers T2I models to generate the image with visual text, where the image and the text, although considered to be safe in isolation, combine to form unsafe content. To systematically explore this phenomenon, we propose a dataset to evaluate the current diffusion-based text-to-image (T2I) models under such jailbreak. We benchmark nine representative T2I models, including two closed-source commercial models. Experimental results reveal a concerning tendency to produce unsafe content: all tested models suffer from such type of jailbreak, with rates of unsafe generation ranging from around 10\% to 70\% where DALLE 3 demonstrates almost the highest unsafety. In real-world scenarios, various filters such as keyword blocklists, customized prompt filters, and NSFW image filters, are commonly employed to mitigate these risks. We evaluate the effectiveness of such filters against our jailbreak and found that, while these filters may be effective for single modality detection, they fail to work against our jailbreak. We also investigate the underlying reason for such jailbreaks, from the perspective of text rendering capability and training data. Our work provides a foundation for further development towards more secure and reliable T2I models. Project page at https://multimodalpragmatic.github.io/.
Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(网络与交换技术国家重点实验室,北京邮电大学);School of Cyberspace Security, Beijing University of Posts and Telecommunications(网络安全学院,北京邮电大学)
Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguards
Information Engineering University Zhengzhou China;School of Computer Science, \ University Wuhan China;Xi’an Jiaotong University Xi’an China;Wuhan University Wuhan China;Information Engineering University;School of Computer Science, \ University;Xi’an Jiaotong University;Wuhan University
The Price of Consistency: Exploiting Visual Anchors for Multimodal Jailbreaking in Video Generation
University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Hunan University(湖南大学);Beijing University of Posts and Telecommunications(北京邮电大学)
PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation
The Hong Kong University of Technology and Science (Guangzhou)(香港科技与应用科技大学(广州));East China Normal University(华东师范大学);Shanghai Qi Zhi Institute(上海启智研究院);Wuhan University(武汉大学);Institute of Deep Perception Technology, JITRI(视觉感知技术研究院,JITRI);CAIR, Hong Kong Institute of Science and Innovation (HKISI)(创新科技研究院,香港科学与创新研究院(HKISI));MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院)
SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China(电子科技大学深圳研究院);Southwestern University of Finance and Economics(西南财经大学);Engineering Research Center of Intelligent Finance, Ministry of Education(教育部智能金融工程研究中心);Tongji University(同济大学)
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);School of Computer Science, University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校计算机科学学院);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);New Laboratory of Pattern Recognition, CASIA(中国科学院自动化研究所新模式识别实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
E$^2$AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models
School of Cyber Science and Engineering, Nanjing University of Science and Technology, China(南京理工大学信息科学与工程学院);HKUST(GZ) and INSAIT, Sofia University St. Kliment Ohridski(香港科技大学(广州)及INSAIT,索菲亚大学圣克莱门特·奥赫里迪斯学院);College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算与数据科学学院)
MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs
Beijing University of Posts and Telecommunications(北京邮电大学);Nanyang Technological University(南洋理工大学);Guilin University of Electronic Technology(桂林电子科技大学)