对齐是局部化的：对偏好层的因果探究
Alignment is Localized: A Causal Probe into Preference Layers
AI总结：
本文通过基座模型与微调模型间的逐层因果修补和 LASSO 分析，揭示 Llama-3.2-1B 的人类偏好对齐主要局部编码于中间层的低秩方向子空间。
摘要：
强化学习框架，尤其是那些利用人工标注的框架，已成为偏好微调中一种日益流行的方法；在偏好微调中，语言模型的输出会被调整，以匹配某一组行为策略或准则。通过人类反馈的强化学习（RLHF）可能是此类框架最受欢迎的实现方式，特别是在使语言模型对齐安全性和人类意图方面。然而，这种对齐究竟如何实现，其内部机制在很大程度上仍不透明。在本工作中，我们通过在基座模型与其微调后对应模型之间、跨人类偏好对应用逐层因果修补，系统分析了用于语言模型对齐的偏好优化。我们在 Llama-3.2-1B 上实现该方法，并发现对齐在空间上是局部化的：中间层激活编码了一个独特子空间，该子空间因果性地决定与奖励一致的行为，而早期层和晚期层基本不受影响。利用 LASSO 回归，我们还发现只有少数层具有非零系数，这些系数将激活距离与奖励增益联系起来。总体而言，我们表明，至少对某些语言模型而言，来自基于人类的偏好性微调的对齐是一种方向性、低秩过程，而非弥散的、参数化的过程。
英文摘要：
Reinforcement Learning frameworks, particularly those utilizing human annotations, have become an increasingly popular method for preference fine-tuning, where the outputs of a language model are tuned to match a certain set of behavioral policies or guidelines. Reinforcement Learning through Human Feedback (RLHF) is perhaps the most popular implementation of such a framework, particularly for aligning LMs toward safety and human intent. However, the internal workings of how such alignment is achieved remain largely opaque. In this work, we systematically analyze preference optimization for language model alignment by applying layer-wide causal patching between a base model and its tuned counterpart across human preference pairs. We implement our methodology on \textit{Llama-3.2-1B}, and find that alignment is spatially localized: mid-layer activations encode a distinct subspace that causally determines reward-consistent behavior, while early and late layers remain largely unaffected. Utilizing LASSO regression, we also find that only a small number of layers possess non-zero coefficients linking activation distances to reward gains. Overall, we show that, at least for some language models, alignment from human-based, preferential tuning is a directional, low rank process, rather than diffuse and parameteric.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
