arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.04385cs.CLcs.AIcs.LG
收藏

对齐路由:在语言模型中本地化、扩展和控制策略电路

How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models

Independent Researcher(独立研究者)

Gregory N. Frank

AI总结:

研究通过本地化策略路由机制,探讨在语言模型中扩展和控制策略电路的方法,发现路由机制在安全性和性能上的关键作用。

摘要:

我们本地化了对齐训练语言模型中的策略路由机制。中间层注意力门读取检测到的内容并触发更深层的放大头,以增强信号向拒绝的传递。在较小的模型中,门和放大器是单个头;在更大的规模上,它们成为跨相邻层的头带。门贡献了不到1%的输出DLA,但互换测试(p < 0.001)和敲除级联确认其因果必要性。在n >= 120时,互换筛查在十二个实验室的六个模型中检测到相同的motif,尽管具体头因实验室而异。单头消融在72B上削弱了多达58倍,且遗漏了互换识别的门;在规模上,互换是唯一可靠的审计。连续调节检测层信号可以控制策略从强硬拒绝通过逃避到事实回答。在安全提示上,相同的干预将拒绝转化为有害指导,表明安全训练能力由路由控制,而非移除。阈值因主题和输入语言而异,且电路在家族内跨代迁移,即使行为基准没有变化。路由是早承诺:门在其自身层上触发,比更深层的层完成输入处理之前。一个上下文替换密码将门互换必要性降低了70到99%在三个模型中,且模型转向谜题解决而不是拒绝。将明文门激活注入密码前向传递恢复了Phi-4-mini中48%的拒绝,将绕过局部化到路由接口。第二种方法,密码对比分析,利用明/密DLA差异映射完整的密码敏感路由电路在O(3n)前向传递中。任何能击败检测层模式匹配的编码都能绕过策略,无论更深层的层是否重构内容。

英文摘要:

Models can represent policy-relevant content yet respond differently. We study the policy-routing circuit that turns detection into refusal. Using political censorship and harmful-request refusal as test cases, four-method circuit analyses in Qwen3-8B, Phi-4-mini, and Gemma-2-2B reveal sparse gate-amplifier circuits at architecture-specific locations. An intermediate-layer gate initiates routing, while later, distributed amplifier heads strengthen a refusal-directed signal. The gate is hard to find with direct logit attribution (DLA): in Qwen3-8B, it contributes less than 1% of output DLA. We therefore combine DLA screening, directional intervention, activation interchange, and gate-to-amplifier knockout. Activation interchange identifies upstream trigger candidates, and knockout suppresses a majority of tested later-head contributions in all three models. The gate is an upstream trigger, not a self-contained refusal mechanism; later amplifiers strengthen the signal toward the output. Candidate-head screens in nine additional models from six labs, spanning 2B-72B parameters, nominate similarly localized routing candidates without establishing head-level homology. Policy routing is a sparse, causally structured interface between content detection and aligned behavior that output attribution alone can miss.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑