arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.29629cs.AI
收藏

超越攻击成功率:LLM安全失效的时间对数可观测性

Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failures

Chung-Ang University(Chung-Ang 大学)

Junyoung Park, Sunghwan Park, Seongyong Ju, Jaewoo Lee

AI总结:

提出时间对数可观测性(TLO)方法,通过解码过程中的合规-拒绝边际将模型-攻击条件映射到校准的二维平面,揭示攻击成功的时间模式,并基于此设计早期停止规则将成功越狱减少一半以上。

摘要:

攻击成功率(ASR)在生成结束时用单个是/否标签评估每次越狱,告诉我们是否发生了失败,但未说明失败如何展开。产生同等有害输出的两次攻击可能遵循完全不同的路径,而ASR无法区分它们。我们仅从对数几率使这些隐藏路径变得可观测。时间对数可观测性(TLO)是一种无需训练的诊断方法,在解码过程中观察合规-拒绝边际,并将每个模型-攻击条件置于校准的二维平面上。通过设计,该平面在ASR信息量最小的情况下最具信息量:即在因真正不同原因而成功的攻击中。在四种对齐的LLM和三种越狱范式下,具有几乎相同ASR的攻击在平面上位于明显不同的点:同一模型可能通过不同的时间模式失败。在大多数条件下,几何形状与来自隐藏状态的拒绝方向探针匹配,但一个模型显示了固定词汇方法的局限性。从TLO导出的简单早期停止规则将成功的越狱减少一半以上,且对普通良性查询无误报。安全评估应报告失败发生的时间和方式,而不仅仅是是否发生。TLO仅从对数几率即可观测前两者。

英文摘要:

Attack Success Rate (ASR) evaluates each jailbreak with a single yes/no label at the end of generation, telling us whether a failure happened but not how it unfolded. Two attacks that produce equally harmful outputs may have followed completely different paths, and ASR cannot tell them apart. We make those hidden paths observable from logits alone. Temporal Logit Observability (TLO) is a training-free diagnostic that watches a compliance-refusal margin during decoding and places each model-attack condition on a calibrated 2D plane. By design, this plane is most informative exactly where ASR is least informative: among attacks that succeed for genuinely different reasons. Across four aligned LLMs and three jailbreak paradigms, attacks with nearly identical ASR land at clearly different points on the plane: the same model can fail through different temporal patterns. The geometry matches refusal-direction probes from hidden states on most conditions, with one model showing the limit of our fixed-lexicon approach. A simple early-stop rule derived from TLO cuts successful jailbreaks by more than half, without false alarms on plain benign queries. Safety evaluation should report when and how a failure unfolds, not only whether it occurred. TLO makes the first two observable from logits alone.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑