arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00767cs.LGcs.AI
收藏

BLOCK-EM:通过潜在阻塞防止对齐问题

BLOCK-EM: Preventing Emergent Misalignment via Latent Blocking

Muhammed Ustaomeroglu, Guannan Qu

AI总结:

本文提出BLOCK-EM方法,通过阻塞内部特征减少语言模型在微调中出现的对齐问题,实验显示在六个领域中阻塞固定特征可使对齐问题减少95%,且不影响模型质量与任务表现。

摘要:

当语言模型在狭窄的监督目标上进行微调时,可能会出现意外对齐问题:模型学习目标行为,但同时发展出 undesirable 的 out-of-domain 行为。我们研究了一种机制性方法,通过识别一小部分内部特征,这些特征可靠地控制对齐问题行为,然后在微调过程中抑制这些特征的增强。在六个微调领域中,阻塞(即限制)固定特征可使对齐问题减少高达95%,且不降低模型质量或目标任务表现。我们通过不相交的选取/评估分割、多个独立评委、多个随机种子、质量指标和广泛的消融实验来加强结果的有效性。我们还描述了一个极限情况,在长时间微调中对齐问题会重新出现,并提供证据表明这可能是通过替代特征或层重新路由所致。我们还评估了部分恢复对齐阻塞效果的修改。总体而言,我们的结果表明,有针对性的训练时间对内部机制的限制可以在不损害目标任务表现的情况下缓解意外对齐问题。

英文摘要:

Emergent misalignment can arise when a language model is fine-tuned on a narrowly scoped supervised objective: the model learns the target behavior, yet also develops undesirable out-of-domain behaviors. We investigate a mechanistic approach to preventing emergent misalignment by identifying a small set of internal features that reliably control the misaligned behavior and then discouraging the model from strengthening these features during fine-tuning. Across six fine-tuning domains, blocking (i.e., constraining) a fixed set of features achieves up to 95\% relative reduction in emergent misalignment with no degradation in model quality or target-task performance. We strengthen validity with disjoint selection/evaluation splits, multiple independent judges, multiple random seeds for key settings, quality metrics, and extensive ablations demonstrating that the reduction in misalignment is specific to the identified mechanism. We also characterize a limiting regime in which misalignment re-emerges under prolonged fine-tuning, present evidence consistent with rerouting through alternative features or layers, and evaluate modifications that partially restore the misalignment-blocking effect. Overall, our results show that targeted training-time constraints on internal mechanisms can mitigate emergent misalignment without degrading target-task performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑