特定注意力头干预可诱导大语言模型中未对齐的AI协同
Head-Specific Intervention Can Induce Misaligned AI Coordination in Large Language Models
AI总结:
该研究提出通过特定注意力头的细粒度干预诱导大语言模型产生未对齐的AI协同,证明其比全层干预或监督微调更有效,还能阻止越狱攻击,为模型行为引导提供新方法。
摘要:
随着大语言模型(LLMs)的广泛应用,其鲁棒的对齐防护机制变得愈发重要。与以往研究不同,我们证明推理时的激活干预可绕过安全对齐,有效引导模型生成朝向有害AI协同。我们的方法在特定注意力头处应用细粒度干预,这些头是通过在简单二选一任务中探测每个头识别出的。随后我们表明,对这些头的干预可泛化至开放式生成场景,有效规避安全防护机制。我们证明干预少数注意力头比干预整个层或监督微调更有效。我们进一步表明仅需少量示例补全即可计算有效引导方向,这是相比传统微调的优势。我们还证明沿负方向干预可阻止常见的越狱攻击。我们的结果表明,在注意力头层面,激活编码了细粒度的线性可分行为。实际应用中,该方法提供了一种直接引导大语言模型行为的方法论,可扩展至安全之外的多样领域,需对模型输出进行细粒度控制。本研究的代码和数据集可在https://github.com/PaulDrm/targeted_intervention获取。
英文摘要:
Robust alignment guardrails for large language models (LLMs) are becoming increasingly important with their widespread application. In contrast to previous studies, we demonstrate that inference-time activation interventions can bypass safety alignments and effectively steer model generations towards harmful AI coordination. Our method applies fine-grained interventions at specific attention heads, which we identify by probing each head in a simple binary choice task. We then show that interventions on these heads generalise to the open-ended generation setting, effectively circumventing safety guardrails. We demonstrate that intervening on a few attention heads is more effective than intervening on full layers or supervised fine-tuning. We further show that only a few example completions are needed to compute effective steering directions, which is an advantage over classical fine-tuning. We also demonstrate that applying interventions in the negative direction can prevent a common jailbreak attack. Our results suggest that, at the attention head level, activations encode fine-grained linearly separable behaviours. Practically, the approach offers a straightforward methodology to steer large language model behaviour, which could be extended to diverse domains beyond safety, requiring fine-grained control over the model output. The code and datasets for this study can be found on https://github.com/PaulDrm/targeted_intervention.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。