As LLM agents transition from digital assistants to physical controllers in autonomous systems and robotics, they face an escalating threat from indirect prompt injection. By embedding adversarial instructions into the results of tool calls, attackers can hijack the agent's decision-making process to execute unauthorized actions. This vulnerability poses a significant risk as agents gain more direct control over physical environments. Existing defense mechanisms against Indirect Prompt Injection (IPI) generally fall into two categories. The first involves training dedicated detection models; however, this approach entails high computational overhead for both training and inference, and requires frequent updates to keep pace with evolving attack vectors. Alternatively, prompt-based methods leverage the inherent capabilities of LLMs to detect or ignore malicious instructions via prompt engineering. Despite their flexibility, most current prompt-based defenses suffer from high Attack Success Rates (ASR), demonstrating limited robustness against sophisticated injection attacks. In this paper, we propose a novel method that provides LLMs with precise data via tool result parsing while effectively filtering out injected malicious code. Our approach achieves competitive Utility under Attack (UA) while maintaining the lowest Attack Success Rate (ASR) to date, significantly outperforming existing methods. Code is available at GitHub.
FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks
UW-Madison(威斯康星大学麦迪逊分校);Huazhong University of Science and Technology(华中科技大学);University of Rochester(罗切斯特大学);NVIDIA(英伟达公司);Cornell University(康奈尔大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);UC-Davis(加州大学戴维斯分校)
ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correction
College of Computing and Data Science, Nanyang Technological University, Singapore(计算与数据科学学院,南洋理工大学,新加坡);Alibaba(阿里巴巴);School of Computer Science, Peking University, China(计算机科学学院,北京大学,中国)
Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworks
The Hong Kong University of Science and Technology(香港科技大学);Zhejiang University of Technology(浙江工业大学);Lingnan University(岭南大学);School of Cyber Science and Engineering, Southeast University(东南大学计算机科学与工程学院);ZTE Corporation(中兴通讯有限公司)
AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University(航空信息安全与可信计算重点实验室、教育部、网络安全科学与工程学院、武汉大学);Department of Computer Science and Engineering, University at Buffalo, SUNY(计算机科学与工程系、布法罗大学、纽约州立大学);Department of Computer Science, College of Information Science and Technology, Jinan University(计算机科学系、信息科学与技术学院、济南大学);College of Cyber Security, Jinan University(网络安全学院、济南大学)
ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Cyber Science and Technology, Beihang University(北京航空航天大学网络空间安全学院);University of Wisconsin–Madison(威斯康星大学麦迪逊分校)
RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
École polytechnique(巴黎综合理工学院);Institut Polytechnique de Paris(巴黎理工学院);CNRS(法国国家科学研究中心);Google DeepMind(谷歌DeepMind);Agence Ministérielle pour l’IA de Défense(法国国防部人工智能局);IRT SystemX(法国系统X技术研究所)
Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems
Ukrainian State University of Science and Technologies, ESI "Prydniprovska State Academy of Civil Engineering and Architecture(乌克兰科学与技术国家大学,埃斯I普里皮亚特斯卡州立建筑与工程学院);Google DeepMind(谷歌DeepMind)
Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models
Bellini College of AI, Cybersecurity, and Computing(人工智能、网络安全与计算学院);University of South Florida, Tampa, Florida, USA(佛罗里达州塔帕斯大学);University of California, Davis, California, USA(加州大学戴维斯分校)
APEX: Active Protection at Execution Boundaries for LLM Agents
Tsinghua University(清华大学);Imperial College London(帝国理工学院);Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学);University College London(伦敦大学学院);A*STAR(新加坡科技研究局);University of British Columbia(不列颠哥伦比亚大学);Shenzhen University(深圳大学);The University of Hong Kong(香港大学)
AdapTools: Adaptive Tool-based Indirect Prompt Injection Attacks on Agentic LLMs
College of Computing(计算学院);Data Science, Nanyang Technological University, Singapore(数据科学,南洋理工大学,新加坡);School of Computer Science, Peking University, China(计算机科学学院,北京大学,中国)
Systematization of Knowledge: Security and Safety in the Model Context Protocol Ecosystem
Tribhuvan University(特里布文大学);University of North Dakota(北达科他大学);Youngstown State University(亚当斯州立大学);University of Missouri(密苏里大学);University of Toledo(托莱多大学)