arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20907cs.CR
收藏

从检测到决策:提示注入检测的校准与弃权(不执行)

From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

eBay Inc.(eBay公司)

Qianlong Lan, Anuj Kaul, Vinothini Pandurangan, Indranil Sanyal

AI总结:

该研究针对提示注入防御,评估了Jev、Llama Prompt Guard 2等模型在分布偏移下的检测、校准及决策性能,发现仅概率输出无法保障可靠安全决策,需联合多方面因素构建LLM安全系统。

摘要:

提示注入防御通常被评估为二分类检测器,然而部署的大语言模型(LLM)应用必须就是否允许、拦截或升级处理潜在恶意输入做出操作决策。近期的概率决策模型旨在通过校准后的置信度估计支持此类决策,但它们在分布偏移下的可靠性仍不确定。我们评估了一个原生概率决策模型(Jev)、两个Llama Prompt Guard 2分类器(22M参数和86M参数)以及一个基于LLM的评判器,使用47611个保留样本,涵盖直接和间接提示注入、硬良性输入、混淆以及分布偏移。通过独立的开发集和校准集,我们在固定评估协议下评估了检测性能、概率校准以及允许-拦截-弃权(不执行)策略。LLM评判器实现了最高的整体判别性能(AUROC为0.897),其次是Jev(0.871)、Prompt Guard 2 86M(0.757)和Prompt Guard 2 22M(0.714)。不过,Jev的原生概率并未始终优于事后校准的分类器分数,温度缩放可改善校准但不提升判别性能。在1%的经验误阴性率目标下,所有评估的门控-评判系统都需要超过91.8%的升级处理,并产生显著的误阳性率。这些发现表明,仅概率输出无法在分布偏移下建立可靠的安全决策,有效的LLM安全系统必须联合评估检测、校准、评判器行为以及操作决策策略。

英文摘要:

Prompt-injection defenses are commonly evaluated as binary detectors, although deployed LLM applications must make operational decisions about whether to allow, block, or escalate potentially malicious inputs. Recent probabilistic decision models aim to support such decisions through calibrated confidence estimates, but their reliability under distribution shift remains uncertain. We evaluate a native probabilistic decision model (Jev), two Llama Prompt Guard 2 classifiers (22M and 86M), and an LLM-based judge on 47,611 held-out examples spanning direct and indirect prompt injections, hard-benign inputs, obfuscation, and distribution shift. Using separate development and calibration splits, we assess detection performance, probability calibration, and allow-block-defer policies under a frozen evaluation protocol. The LLM judge achieves the highest overall discrimination (AUROC 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714). However, Jev's native probabilities do not consistently outperform post-hoc calibrated classifier scores, and temperature scaling improves calibration without improving discrimination. At an empirical false-negative-rate target of 1%, all evaluated gate-reviewer systems require more than 91.8% escalation and incur substantial false-positive rates. These findings demonstrate that probabilistic outputs alone do not establish reliable security decisions under distribution shift. Effective LLM security systems must jointly evaluate detection, calibration, reviewer behavior, and operational decision policies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑