arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04778cs.AIcs.CLcs.LG
收藏

超越浅层安全的推理时脆弱性:沿生成轨迹的对齐

Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

Hankuk University of Foreign Studies(翰江大学外国语大学);Seoul National University(首尔国立大学)

Kyungmin Park, Taesup Kim

AI总结:

本文揭示安全对齐的大语言模型在推理时存在更广泛的脆弱性,即任意生成步骤的短标记注入都能显著改变后续安全行为,并提出通过直接在生成轨迹上对齐模型来提升鲁棒性。

摘要:

安全对齐的大语言模型(LLMs)在推理时仍然容易受到干预,这些干预会将生成导向有害输出。最近的研究将其归因于浅层安全,即对齐集中在最初的几个输出标记上。我们表明,浅层安全是更广泛的推理时脆弱性的一个特例,其中在任何生成步骤的短标记注入都能显著改变后续的安全行为。我们还发现,模型在其隐藏状态中与拒绝方向的对齐并不能预测其对这种注入的鲁棒性,这表明在扰动下,内部状态本身并不能决定生成行为。为了解决这个问题,我们通过模拟序列中段扰动构建的生成轨迹上直接对齐模型,并表明这提高了对中段注入的鲁棒性,并泛化到利用早期标记生成的攻击。我们的工作认为,鲁棒的安全对齐需要对生成过程本身进行训练,而不仅仅是其输出。

英文摘要:

Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-token generation. Our work argues that robust safety alignment requires training on the generation process itself, not only its outputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑