The advent of Large Language Models LLMs marks a milestone in Artificial Intelligence, altering how machines comprehend and generate human language. However, LLMs are vulnerable to malicious prompt injection attacks, where crafted inputs manipulate the models behavior in unintended ways, compromising system integrity and causing incorrect outcomes. Conventional detection methods rely on static, rule-based approaches, which often fail against sophisticated threats like abnormal token sequences and alias substitutions, leading to limited adaptability and higher rates of false positives and false negatives.This paper proposes a novel NLP based approach for prompt injection detection, emphasizing accuracy and optimization through a layered input screening process. In this framework, prompts are filtered through three distinct layers rule-based, ML classifier, and companion LLM before reaching the target model, thereby minimizing the risk of malicious interaction.Tests show the ML classifier achieves the highest accuracy among individual layers, yet the multi-layer framework enhances overall detection accuracy by reducing false negatives. Although this increases false positives, it minimizes the risk of overlooking genuine injected prompts, thus prioritizing security.This multi-layered detection approach highlights LLM vulnerabilities and provides a comprehensive framework for future research, promoting secure interactions between humans and AI systems.
Soft Begging: Modular and Efficient Shielding of LLMs against Prompt Injection and Jailbreaking based on Prompt Tuning
Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心);sequire technology GmbH(sequire技术有限公司)
Prompt Injection attack against LLM-integrated Applications
Griffith University(格里菲斯大学);Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);Huazhong University of Science and Technology(华中科技大学);Indiana University at Bloomington(印第安纳大学布卢明顿分校);Southern University of Science and Technology(南方科技大学);Tianjin University(天津大学)
The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism
Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev(本·古里安大学软件与信息系统工程系);School of Electrical and Computer Engineering, Tel Aviv University(特拉维夫大学电气与计算机工程学院);Harvard Kennedy School, Harvard University, and Munk School, University of Toronto(哈佛大学哈佛肯尼迪学校及多伦多大学穆克学校)
Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Duke University(杜克大学);Arizona State University(亚利桑那州立大学);hireEZ;University of California, Berkeley(加州大学伯克利分校)
UC Berkeley(加州大学伯克利分校);UC Santa Barbara(加州大学圣巴巴拉分校);Duke University(杜克大学);National University of Singapore(新加坡国立大学);King Abdulaziz City for Science and Technology(国王阿卜杜勒阿齐兹城市科学技术学院);University of Washington(华盛顿大学)
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院);School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络安全学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Faculty of Data Science, City University of Macau(澳门城市大学数据科学学院)
MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs
Seoul National University College of Medicine(首尔国立大学医学院);Seoul National University(首尔国立大学);Seoul National University College of Medicine, Seoul National University Hospital(首尔国立大学医学院、首尔国立大学医院)
"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers
Institute of Information Engineering, CAS(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);The Conversational AI (CoAI) group, DCST, Tsinghua University(清华大学计算机科学与技术系对话式人工智能小组)