arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.21146cs.CLcs.AI
收藏

Palisade——提示注入检测框架

Palisade -- Prompt Injection Detection Framework

SRM Institute of Science and Technology(SRM科学技术学院)

Sahasra Kokkula, Somanathan R, Nandavardhan R, Aashishkumar, G Divya

AI总结:

针对LLM提示注入攻击,本文提出Palisade三层检测框架,结合规则、ML分类器与伴随LLM筛查输入,以增加少量假阳性为代价减少假阴性并提升整体安全性。

摘要:

大语言模型(LLM)的出现标志着人工智能领域的一个里程碑,改变了机器理解和生成人类语言的方式。然而,LLM容易受到恶意提示注入攻击:精心构造的输入会以非预期方式操纵模型行为,破坏系统完整性并导致错误结果。传统检测方法依赖静态的基于规则的方式,面对异常 token 序列和别名替换等复杂威胁时往往失效,导致适应性有限,且假阳性和假阴性率较高。本文提出一种新颖的基于 NLP 的提示注入检测方法,通过分层输入筛查过程强调准确性和优化。在该框架中,提示在到达目标模型之前会经过三个不同层级的过滤:基于规则的层、ML 分类器和伴随 LLM,从而最大限度降低恶意交互风险。测试表明,ML 分类器在各单独层级中取得最高准确率,而多层框架通过减少假阴性提升了整体检测准确率。尽管这会增加假阳性,但它最大限度降低了遗漏真实注入提示的风险,从而将安全性置于优先位置。这种多层检测方法凸显了 LLM 的脆弱性,并为未来研究提供了全面框架,促进人类与 AI 系统之间的安全交互。

英文摘要:

The advent of Large Language Models LLMs marks a milestone in Artificial Intelligence, altering how machines comprehend and generate human language. However, LLMs are vulnerable to malicious prompt injection attacks, where crafted inputs manipulate the models behavior in unintended ways, compromising system integrity and causing incorrect outcomes. Conventional detection methods rely on static, rule-based approaches, which often fail against sophisticated threats like abnormal token sequences and alias substitutions, leading to limited adaptability and higher rates of false positives and false negatives.This paper proposes a novel NLP based approach for prompt injection detection, emphasizing accuracy and optimization through a layered input screening process. In this framework, prompts are filtered through three distinct layers rule-based, ML classifier, and companion LLM before reaching the target model, thereby minimizing the risk of malicious interaction.Tests show the ML classifier achieves the highest accuracy among individual layers, yet the multi-layer framework enhances overall detection accuracy by reducing false negatives. Although this increases false positives, it minimizes the risk of overlooking genuine injected prompts, thus prioritizing security.This multi-layered detection approach highlights LLM vulnerabilities and provides a comprehensive framework for future research, promoting secure interactions between humans and AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑