The demand of customized large language models (LLMs) has led to commercial LLMs offering black-box fine-tuning APIs, yet this convenience introduces a critical security loophole: attackers could jailbreak the LLMs by fine-tuning them with malicious data. Though this security issue has recently been exposed, the feasibility of such attacks is questionable as malicious training dataset is believed to be detectable by moderation models such as Llama-Guard-3. In this paper, we propose TrojanPraise, a novel finetuning-based attack exploiting benign and thus filter-approved data. Basically, TrojanPraise fine-tunes the model to associate a crafted word (e.g., "bruaf") with harmless connotations, then uses this word to praise harmful concepts, subtly shifting the LLM from refusal to compliance. To explain the attack, we decouple the LLM's internal representation of a query into two dimensions of knowledge and attitude. We demonstrate that successful jailbreak requires shifting the attitude while avoiding knowledge shift, a distortion in the model's understanding of the concept. To validate this attack, we conduct experiments on five opensource LLMs and two commercial LLMs under strict black-box settings. Results show that TrojanPraise achieves a maximum attack success rate of 95.88% while evading moderation.
Defending Jailbreak Attacks on Large Language Models via Manifold Trajectory Kinetics
Huazhong University of Science and Technology(华中科技大学);Changsha University of Science and Technology(长沙理工大学);Xiangtan University(湘潭大学);City University of Macau(澳门城市大学);Griffith University(格里菲斯大学)
Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Carrier Articles
The Pennsylvania State University(宾夕法尼亚州立大学);Northern Arizona University(北亚利桑那大学);Worcester Polytechnic Institute(伍斯特理工学院);Carnegie Mellon University(卡内基梅隆大学)
from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);People’s Public Security University of China(中国人民公安大学);Tsinghua University(清华大学)
Steering Beyond the Support: Adversarial Training on Unsupervised Jailbroken Activation Simulation
University of Technology Sydney, Sydney, Australia(技术悉尼大学);School of Cyber Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院);Xi'an Jiaotong University, Xi'an, China(西安交通大学)
School of Information Management, Wuhan University(武汉大学信息管理学院);Nanyang Technological University(南洋理工大学);Torr Vision Group, University of Oxford(牛津大学Torr视觉组)
Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks
Lab of Artificial Intelligence for Education, East China Normal University, Shanghai 200062, China(教育人工智能实验室,东华大学,上海200062,中国);Shanghai Institute of Artificial Intelligence for Education, East China Normal University, Shanghai 200062, China(教育人工智能研究所,东华大学,上海200062,中国);School of Computer Science and Technology, East China Normal University, Shanghai 200062, China(计算机科学与技术学院,东华大学,上海200062,中国)
DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing
ShanghaiTech University(上海科技大学);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);The CoAI group, Tsinghua University(清华大学CoAI课题组)
DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protection
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术开发区(滨江)区块链与数据安全研究所);Shanghai AI Laboratory(上海人工智能实验室);East China Normal University(华东师范大学)
Reasoning-Oriented Programming: Chaining Semantic Gadgets to Jailbreak Large Vision Language Models
AI Security Lab(360人工智能安全实验室);College of Science, Mathematics and Technology, Wenzhou-Kean University(温州肯恩大学理学院);State Key Laboratory of Complex & Critical Software Environment, Beihang University(北京航空航天大学复杂系统与关键软件环境国家重点实验室)
Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attacks
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Ant Group(蚂蚁集团)
ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);National University of Singapore(新加坡国立大学);CSIRO’s Data61(CSIRO数据61);Responsible AI Research (RAIR) Centre, The University of Adelaide(负责任人工智能研究(RAIR)中心,阿德莱德大学);Tsinghua University(清华大学)
SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models
Nanjing University of Science and Technology(南京理工大学);School of Cyber Science and Engineering, Nanjing University of Science and Technology(南京理工大学网络空间安全学院);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学)
Geneshift: Impact of different scenario shift on Jailbreaking LLM
National University of Singapore(新加坡国立大学);NUS Graduate School(新加坡国立大学研究生院);Integrative Sciences and Engineering Programme(整合科学与工程项目);Institute of Data Science (IDS)(数据科学研究院);School of Computing(计算机学院)