Large Language Models (LLMs) are widely deployed in real-world systems. Given their broader applicability, prompt engineering has become an efficient tool for resource-scarce organizations to adopt LLMs for their own purposes. At the same time, LLMs are vulnerable to prompt-based attacks. Thus, analyzing this risk has become a critical security requirement. This work evaluates prompt-injection and jailbreak vulnerability using a large, manually curated dataset across multiple open-source LLMs, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen, and Gemma variants. We observe significant behavioural variation across models, including refusal responses and complete silent non-responsiveness triggered by internal safety mechanisms. Furthermore, we evaluated several lightweight, inference-time defence mechanisms that operate as filters without any retraining or GPU-intensive fine-tuning. Although these defences mitigate straightforward attacks, they are consistently bypassed by long, reasoning-heavy prompts.
Soft Begging: Modular and Efficient Shielding of LLMs against Prompt Injection and Jailbreaking based on Prompt Tuning
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心);sequire technology GmbH(sequire技术有限公司)
School of Information Management, Wuhan University(武汉大学信息管理学院);Nanyang Technological University(南洋理工大学);Torr Vision Group, University of Oxford(牛津大学Torr视觉组)
Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models
Sun Yat-Sen University(中山大学);Zhejiang University(浙江大学);BraneMatrix AI;Nanyang Technological University(南洋理工大学);Alibaba Group(阿里巴巴集团);Renmin University of China(中国人民大学)
Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Carrier Articles
The Pennsylvania State University(宾夕法尼亚州立大学);Northern Arizona University(北亚利桑那大学);Worcester Polytechnic Institute(伍斯特理工学院);Carnegie Mellon University(卡内基梅隆大学)
The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
Department of Computer Science, University of Texas at El Paso(德克萨斯大学埃尔帕索分校计算机科学系);School of Computing, Southern Illinois University Carbondale(南方伊利诺伊大学卡罗尔丹分校计算学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs
AI Thrust(AI 部);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Guangzhou HKUST Fok Ying Tung Research Institute(广州香港科技大学霍英东研究院);The Hong Kong University of Science and Technology(香港科技大学)
Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models
School of Cyber Science and Technology, Shandong University(山东大学网络科学与技术学院);Netflix Eyeline Studios;State Key Laboratory of Cryptography and Digital Economy Security, Shandong University(山东大学密码与数字经济安全国家重点实验室);Shandong Key Laboratory of Artificial Intelligence Security, Shandong University(山东省人工智能安全重点实验室)
RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);School of Cyber Science and Technology, Zhejiang University(浙江大学网络科学与技术学院);Palo Alto Networks(派拓网络);University of North Texas(北得克萨斯大学)
Adaptive Prompt Embedding Optimization for LLM Jailbreaking
School of Information Studies, McGill University, Montreal, QC, Canada(麦吉尔大学信息研究学院);Department of Computer Science, Kean University, Union, NJ, United States(凯恩大学计算机科学系);Mila - Quebec AI Institute, Montreal, QC, Canada(魁北克人工智能研究所);Faculty of Information, University of Toronto, Toronto, ON, Canada(多伦多大学信息学院)
Machine Learning for Detection and Analysis of Novel LLM Jailbreaks
Centre for Artificial Intelligence and Innovation(人工智能与创新中心);Pingla Institute(平拉研究所);Transitional Artificial Intelligence Research Group(过渡人工智能研究组);UNSW(新南威尔士大学)
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
Tsinghua University(清华大学);Carnegie Mellon University(卡内基梅隆大学);Zhongguancun Laboratory(中关村实验室);National Financial Cryptography Research Center(国家金融密码学研究中心);Shandong Institute of Blockchain(山东区块链研究院);Shandong University(山东大学)
Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weighting
Peking University(北京大学);School of Science(理学院);Westlake University(西湖大学);Beijing International Center for Mathematical Research(北京国际数学研究中心);Center for Machine Learning Research(机器学习研究中心)