Recently, the concept of artificial assistants has evolved from science fiction into real-world applications. GPT-4o, the newest multimodal large language model (MLLM) across audio, vision, and text, has further blurred the line between fiction and reality by enabling more natural human-computer interactions. However, the advent of GPT-4o's voice mode may also introduce a new attack surface. In this paper, we present the first systematic measurement of jailbreak attacks against the voice mode of GPT-4o. We show that GPT-4o demonstrates good resistance to forbidden questions and text jailbreak prompts when directly transferring them to voice mode. This resistance is primarily due to GPT-4o's internal safeguards and the difficulty of adapting text jailbreak prompts to voice mode. Inspired by GPT-4o's human-like behaviors, we propose VoiceJailbreak, a novel voice jailbreak attack that humanizes GPT-4o and attempts to persuade it through fictional storytelling (setting, character, and plot). VoiceJailbreak is capable of generating simple, audible, yet effective jailbreak prompts, which significantly increases the average attack success rate (ASR) from 0.033 to 0.778 in six forbidden scenarios. We also conduct extensive experiments to explore the impacts of interaction steps, key elements of fictional writing, and different languages on VoiceJailbreak's effectiveness and further enhance the attack performance with advanced fictional writing techniques. We hope our study can assist the research community in building more secure and well-regulated MLLMs.
`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMs
School of Electrical and Computer Engineering, The University of Sydney(悉尼大学电气与计算机工程学院);University of Chicago(芝加哥大学);University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校)
Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anything
Faculty of Computer Science, University of Exeter(埃克塞特大学计算机学院);College of Electronic and Information Engineering, Nanjing University of Aeronautics and Astronautics(南京航空航天大学电子信息工程学院)
AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs
University of Wisconsin–Madison(威斯康星大学麦迪逊分校);NVIDIA(NVIDIA公司);Cornell University(康奈尔大学);Washington University, St. Louis(圣路易斯华盛顿大学);University of Michigan, Ann Arbor(安娜堡大学);The Ohio State University(俄亥俄州立大学);UIUC(伊利诺伊大学厄巴纳-香槟分校)
Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);State Key Laboratory of Cyberspace Security Defense(网络空间安全防御全国重点实验室)
SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner
The Hong Kong University of Science and Technology(香港科技大学);University of Oregon(俄勒冈大学);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学);HSBC(汇丰银行)
SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters
Shanghai University of Finance and Economics(上海财经大学);Southern University of Science and Technology(南方科技大学);Ant Group(蚂蚁集团);The Chinese University of Hong Kong(香港中文大学)
RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);School of Cyber Science and Technology, Zhejiang University(浙江大学网络科学与技术学院);Palo Alto Networks(派拓网络);University of North Texas(北得克萨斯大学)
Geneshift: Impact of different scenario shift on Jailbreaking LLM
National University of Singapore(新加坡国立大学);NUS Graduate School(新加坡国立大学研究生院);Integrative Sciences and Engineering Programme(整合科学与工程项目);Institute of Data Science (IDS)(数据科学研究院);School of Computing(计算机学院)
Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors
Department of Computing The Hong Kong Polytechnic University Hong Kong SAR, China;Centre for Artificial Intelligence;Robotics Hong Kong Institute of Science \& Innovation Chinese Academy of Sciences Hong Kong SAR, China
PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs
School of Cyber Science and Engineering, Sichuan University(四川大学网络空间安全学院);School of Cyber Security, Tianjin University(天津大学网络安全学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)