arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.05451cs.CRcs.LG
收藏

SecAlign:基于偏好优化的提示注入防御方法

SecAlign: Defending Against Prompt Injection with Preference Optimization

UC Berkeley / Meta(伯克利大学/Meta);Meta

Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, Chuan Guo

AI总结:

SecAlign 通过构建安全与不安全输出偏好数据集并进行偏好优化,使大语言模型学会优先响应合法指令而非注入攻击,首次将多种提示注入成功率降至 10% 以下,且对未知攻击泛化良好并保持模型实用性。

摘要:

大语言模型(LLMs)在现代软件系统中日益普及,作为用户与互联网之间的接口,协助完成需要高级语言理解的任务。为完成这些任务,LLM 通常使用外部数据源,如用户文档、网络检索、API 调用结果等。这为攻击者通过提示注入操纵 LLM 开辟了新途径。对抗性提示可被注入外部数据源,以覆盖系统原本的指令,转而执行恶意指令。为缓解这一漏洞,我们提出了一种名为 SecAlign 的新防御方法,基于偏好优化技术。我们的防御首先构建一个偏好数据集,其中包含注入提示的输入、安全输出(响应合法指令的输出)和不安全输出(响应注入的输出)。然后,我们在此数据集上执行偏好优化,教导 LLM 偏好安全输出而非不安全输出。这是首个已知的能将各种提示注入的成功率降低到 10% 以下的方法,即使面对比训练期间所见攻击复杂得多的攻击也是如此。这表明我们的防御对未知和未来的攻击具有良好的泛化能力。此外,在我们的评估中,SecAlign 模型仍然实用,其效用与防御训练前的模型相似。我们的代码位于 https://github.com/facebookresearch/SecAlign。

英文摘要:

Large language models (LLMs) are becoming increasingly prevalent in modern software systems, interfacing between the user and the Internet to assist with tasks that require advanced language understanding. To accomplish these tasks, the LLM often uses external data sources such as user documents, web retrieval, results from API calls, etc. This opens up new avenues for attackers to manipulate the LLM via prompt injection. Adversarial prompts can be injected into external data sources to override the system's intended instruction and instead execute a malicious instruction. To mitigate this vulnerability, we propose a new defense called SecAlign based on the technique of preference optimization. Our defense first constructs a preference dataset with prompt-injected inputs, secure outputs (ones that respond to the legitimate instruction), and insecure outputs (ones that respond to the injection). We then perform preference optimization on this dataset to teach the LLM to prefer the secure output over the insecure one. This provides the first known method that reduces the success rates of various prompt injections to <10%, even against attacks much more sophisticated than ones seen during training. This indicates our defense generalizes well against unknown and yet-to-come attacks. Also, SecAlign models are still practical with similar utility to the one before defensive training in our evaluations. Our code is at https://github.com/facebookresearch/SecAlign

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑