Large vision-language models (VLMs) often exhibit weakened safety alignment with the integration of the visual modality. Even when text prompts contain explicit harmful intent, adding an image can substantially increase jailbreak success rates. In this paper, we observe that VLMs can clearly distinguish benign inputs from harmful ones in their representation space. Moreover, even among harmful inputs, jailbreak samples form a distinct internal state that is separable from refusal samples. These observations suggest that jailbreaks do not arise from a failure to recognize harmful intent. Instead, the visual modality shifts representations toward a specific jailbreak state, thereby leading to a failure to trigger refusal. To quantify this transition, we identify a jailbreak direction and define the jailbreak-related shift as the component of the image-induced representation shift along this direction. Our analysis shows that the jailbreak-related shift reliably characterizes jailbreak behavior, providing a unified explanation for diverse jailbreak scenarios. Finally, we propose a defense method that enhances VLM safety by removing the jailbreak-related shift (JRS-Rem) at inference time. Experiments show that JRS-Rem provides strong defense across multiple scenarios while preserving performance on benign tasks. Code is available at https://github.com/LeeQueue513/JRS-Rem.
Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs
AI Thrust(AI 部);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Guangzhou HKUST Fok Ying Tung Research Institute(广州香港科技大学霍英东研究院);The Hong Kong University of Science and Technology(香港科技大学)
JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
Knight Foundation School of Computing and Information Sciences (KFSCIS)(骑士基金会计算与信息科学学院);Florida International University(佛罗里达国际大学);Sustainability, Optimization, and Learning for InterDependent networks laboratory (solid lab)(可持续性、优化与互依赖网络学习实验室)
Defending Wearable VLMs Against Private Attribute Inference
Swanson School of Engineering, University of Pittsburgh(匹兹堡大学斯旺森工程学院);North Allegheny Senior High(北阿勒格尼高级中学);College of Engineering, Michigan State University(密歇根州立大学工程学院);University of Central Florida(中佛罗里达大学)
Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attacks
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);People’s Public Security University of China(中国人民公安大学)
JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Brown University(布朗大学);University of Michigan Ann Arbor(密歇根大学安娜堡分校);Hong Kong University of Science and Technology(香港科学与技术大学);Carnegie Mellon University(卡内基梅隆大学);Boise State University(博伊西州立大学)
Jailbreak Large Vision-Language Models Through Multi-Modal Linkage
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Shanghai Qi Zhi Institute(上海启智研究所);University of Chicago(芝加哥大学)
Jailbreaking Vision-Language Models Through the Visual Modality
Warsaw University of Technology(华沙理工大学);NASK National Research Institute(波兰国家研究研究院);University of Liverpool(利物浦大学);Indian Institute of Technology Roorkee(印度理工学院拉胡尔分校);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)
Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts
City University of Hong Kong(香港城市大学);City University of Hong Kong (Dongguan)(香港城市大学(东莞));University of Science and Technology Beijing(北京科技大学);The University of Melbourne(墨尔本大学)
Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models
Southeast University(东南大学);National University of Singapore(新加坡国立大学);Lancaster University(兰卡斯特大学);University of California, Los Angeles(加州大学洛杉矶分校);University of Queensland(昆士兰大学)
EVA: Editing for Versatile Alignment against Jailbreaks
ShanghaiTech University(上海科技大学);Sun Yat-sen University(中山大学);State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Tsinghua University(清华大学)
X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usability
School of Electronic Information and Electric Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models
School of Computing and Information Systems, The University of Melbourne, Australia(墨尔本大学计算机与信息系);School of Computing and Information Systems, Singapore Management University, Singapore(新加坡管理大学计算机与信息系);School of Information Technology, Deakin University, Australia(德肯大学信息科技系);Institute of Trustworthy Embodied AI, Fudan University, China(复旦大学可信具身人工智能研究所)
Strategic Deflection: Defending LLMs from Logit Manipulation
International Artificial Intelligence Center of Morocco(摩洛哥国际人工智能中心);Mohammed VI Polytechnic University(摩洛哥穆莱·伊斯梅尔理工学院);AgriEdge;Sorbonne University, LIP6 - UMR 7606 CNRS(索邦大学,LIP6 - UMR 7606 CNRS)
Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models
School of Cyber Science and Technology, Shandong University(山东大学网络科学与技术学院);Netflix Eyeline Studios;State Key Laboratory of Cryptography and Digital Economy Security, Shandong University(山东大学密码与数字经济安全国家重点实验室);Shandong Key Laboratory of Artificial Intelligence Security, Shandong University(山东省人工智能安全重点实验室)
Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models
MoE Key Lab of BIPC, University of Science and Technology of China(北京信息科技大学MoE关键实验室,中国科学技术大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学)