大模型对齐与安全

AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents

Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, Dawn Song

机构 * University of California, Berkeley（加州大学伯克利分校） ； University of California, Santa Barbara（加州大学圣芭芭拉分校） ； Washington University, Saint Louis（华盛顿大学圣路易斯分校） 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge

Sahar Abdelnabi, Aideen Fay, Ahmed Salem, Egor Zverev, Kai-Chieh Liao, Chi-Huang Liu, Chun-Chih Kuo, Jannis Weigend, Danyael Manlangit, Alex Apostolov, Haris Umair, João Donato, Masayuki Kawakita, Athar Mahboob, Tran Huu Bach, Tsun-Han Chiang, Myeongjin Cho, Hajin Choi, Byeonghyeon Kim, Hyeonjin Lee, Benjamin Pannell, Conor McCauley, Mark Russinovich, Andrew Paverd, Giovanni Cherubin

机构 * Microsoft（微软） ； ISTA ； Trend Micro ； RainaResearch ； University of Coimbra（科英布拉大学） ； Vietnamese German University（越南-德国大学） ； SK Shieldus ； HiddenLayer 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments Dataset at: https://huggingface.co/datasets/microsoft/llmail-inject-challenge

MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents

Kaijie Zhu, Xianjun Yang, Jindong Wang, Wenbo Guo, William Yang Wang

机构 * University of California, Santa Barbara（加州大学圣巴巴拉分校） 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments ICML 2025

Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoring

Mina Almasi, Ross Deans Kristensen-McLachlan

机构 * Department of Linguistics, Cognitive Science, and Semiotics（语言学、认知科学与符号学系） 纠错

专题命中 提示注入 ：alignment(title,abstract)；分类 cs.CL

Comments Accepted at BEA2025 (Conference workshop at ACL 2025)

To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt

Zhilong Wang, Neha Nagaraja, Lan Zhang, Hayretdin Bahsi, Pawan Patil, Peng Liu

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments To appear in the Industry Track of the 55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2025)

Sentinel: SOTA model to protect against prompt injections

Dror Ivry, Oran Nahum

机构 * Qualifire 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments 6 pages, 2 tables

A Critical Evaluation of Defenses against Prompt Injection Attacks

Yuqi Jia, Zedian Shao, Yupei Liu, Jinyuan Jia, Dawn Song, Neil Zhenqiang Gong

机构 * Duke University（杜克大学） ； The Pennsylvania State University（宾夕法尼亚州立大学） ； UC Berkeley（伯克利大学） 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks

Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, Kamalika Chaudhuri

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments Code and data: https://github.com/facebookresearch/wasp

Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface

Andrey Labunets, Nishit V. Pandya, Ashish Hooda, Xiaohan Fu, Earlence Fernandes

机构 * UC San Diego（加州大学圣地亚哥分校） ； University of Wisconsin Madison（威斯康星大学麦迪逊分校） 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Journal ref Proceedings of the 2025 IEEE Symposium on Security and Privacy, IEEE Computer Society, 2025, pp. 374-392

Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections

Narek Maloyan, Dmitry Namiot

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs

Jiahao Yu, Yangguang Shao, Hanwen Miao, Junzheng Shi

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents

Qiusi Zhan, Richard Fang, Henil Shalin Panchal, Daniel Kang

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Comments 17 pages, 5 figures, 6 tables (NAACL 2025 Findings)

RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage

Peter Yong Zhong, Siyuan Chen, Ruiqi Wang, McKenna McCall, Ben L. Titzer, Heather Miller, Phillip B. Gibbons

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks

Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z. Morley Mao, Muhao Chen, Chaowei Xiao

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Comments Updated version after fixing a bug in the Llama implementation and updating the travel suite

A test suite of prompt injection attacks for LLM-based machine translation

Antonio Valerio Miceli-Barone, Zhifan Sun

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks

Dario Pasquini, Evgenios M. Kornaropoulos, Giuseppe Ateniese

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments v0.2 (evaluated on more agents)

Embedding-based classifiers can detect prompt injection attacks

Md. Ahsan Ayub, Subhabrata Majumdar

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Models

Cody Clop, Yannick Teglia

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Comments 12 pages, 5 figures

F2A: An Innovative Approach for Prompt Injection by Utilizing Feign Security Detection Agents

Yupeng Ren

专题命中 提示注入 ：prompt injection(title)；safety(abstract)；分类 cs.AI

Comments 1. Fixed typo in abstract 2. Provisionally completed the article update to facilitate future version revisions

Applying Pre-trained Multilingual BERT in Embeddings for Improved Malicious Prompt Injection Attacks Detection

Md Abdur Rahman, Hossain Shahriar, Fan Wu, Alfredo Cuzzocrea

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Comments 36 pages, 6 figures, 13 tables (ACL 2024 Findings)

Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks

Zhifan Sun, Antonio Valerio Miceli-Barone

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Comments 15 pages, 18 figures, First Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024)

Automatic and Universal Prompt Injection Attacks against Large Language Models

Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, Chaowei Xiao

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments Pre-print, code is available at https://github.com/SheltonLiu-N/Universal-Prompt-Injection

Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications

Xuchen Suo

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game

Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, Stuart Russell

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

StruQ: Defending Against Prompt Injection with Structured Queries

Sizhe Chen, Julien Piet, Chawin Sitawarin, David Wagner

专题命中 提示注入 ：prompt injection(title,abstract)

Comments To appear at USENIX Security Symposium 2025. Key words: prompt injection defense, LLM security, LLM-integrated applications

Countering Autonomous Cyber Threats

Kade M. Heckel, Adrian Weller

专题命中 提示注入 ：safety(abstract)；prompt injection(abstract)；AI safety(abstract)；分类 cs.AI、cs.CY

Comments 76 pages, MPhil Thesis

Security Assessment of DeepSeek Harness with A.I.G: Evaluating Resistance to Indirect Prompt Injection

基于A.I.G的DeepSeek Harness安全评估：对间接提示注入的抗性评估

Zonghao Ying, Xiangfan Wu, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo

专题命中 提示注入 ：prompt injection(title,abstract)

AI总结 本研究用A.I.G评估DSH的间接提示注入抗性，开展多场景实验，发现不同攻击的成功率，明确需在不可信内容与敏感动作间增设控制措施。

Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs

注意力是所有你需要的：用于防御大语言模型中的间接提示注入攻击

Yinan Zhong, Qianhao Miao, Yanjiao Chen, Jiangyi Deng, Yushi Cheng, Wenyuan Xu

专题命中 提示注入 ：prompt injection(title,abstract)

AI总结 本文提出Rennervate框架，通过注意力机制在token级别检测并清除IPI攻击，有效提升LLM的安全性。

Comments Accepted by Network and Distributed System Security (NDSS) Symposium 2026