arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.00799cs.CRcs.CLcs.CY
收藏

懂我的意思吗?利用激活增量捕捉LLM任务漂移

Get my drift? Catching LLM Task Drift with Activation Deltas

Microsoft(微软公司);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心)

Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, Andrew Paverd

AI总结:

该研究提出通过LLM激活增量检测任务漂移现象,发现简单线性分类器在分布外测试集上能以近乎完美的ROC AUC检测提示注入等攻击,且无需微调,易于部署。

摘要:

LLMs常被用于检索增强应用中,以基于外部来源的数据执行用户指令。例如,现代搜索引擎使用LLM根据相关搜索结果回答查询;电子邮件插件通过LLM处理邮件内容来总结电子邮件。然而,这些数据源潜在的不可信来源可能导致提示注入攻击,即LLM被嵌入在外部数据中的自然语言指令操纵,导致其偏离用户的原始指令。我们将这种偏离定义为任务漂移。任务漂移是一个重大隐患,因为它允许攻击者窃取数据或影响LLM对其他用户的输出。我们研究将LLM激活作为检测任务漂移的解决方案,表明激活增量(activation deltas,即处理外部数据前后激活值的差异)与这一现象高度相关。通过两种探测方法,我们证明简单的线性分类器在分布外测试集上能以近乎完美的ROC AUC检测到漂移。我们在评估这些方法时,对用户任务、系统提示和攻击的表述方式做出了最少的假设。我们观察到,该方法在未经任何此类攻击训练的情况下,能惊人地泛化到未见过的任务领域,如提示注入、越狱和恶意指令。有趣的是,该解决方案无需对LLM进行任何修改(例如微调),且与现有的元提示解决方案兼容,这使其具有成本效益且易于部署。为了促进基于激活的任务检查、解码和可解释性的进一步研究,我们发布了大规模TaskTracker工具包,包含超过500K个实例的数据集、来自六个SoTA语言模型的表示以及一套检查工具。

英文摘要:

LLMs are commonly used in retrieval-augmented applications to execute user instructions based on data from external sources. For example, modern search engines use LLMs to answer queries based on relevant search results; email plugins summarize emails by processing their content through an LLM. However, the potentially untrusted provenance of these data sources can lead to prompt injection attacks, where the LLM is manipulated by natural language instructions embedded in the external data, causing it to deviate from the user's original instruction(s). We define this deviation as task drift. Task drift is a significant concern as it allows attackers to exfiltrate data or influence the LLM's output for other users. We study LLM activations as a solution to detect task drift, showing that activation deltas - the difference in activations before and after processing external data - are strongly correlated with this phenomenon. Through two probing methods, we demonstrate that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set. We evaluate these methods by making minimal assumptions about how users' tasks, system prompts, and attacks can be phrased. We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions, without being trained on any of these attacks. Interestingly, the fact that this solution does not require any modifications to the LLM (e.g., fine-tuning), as well as its compatibility with existing meta-prompting solutions, makes it cost-efficient and easy to deploy. To encourage further research on activation-based task inspection, decoding, and interpretability, we release our large-scale TaskTracker toolkit, featuring a dataset of over 500K instances, representations from six SoTA language models, and a suite of inspection tools.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑