进展如何？语言模型中的强化学习招募了一个功能性福利轴
How's it going? Reinforcement learning in language models recruits a functional welfare axis
AI总结：
本文通过迷宫环境实验，发现强化学习会招募语言模型中预先存在的功能性福利表征（即对系统目标达成程度的估计），从而广泛影响模型行为，且该表征在训练前已存在。
摘要：
强化学习如何塑造语言模型的内部表征？我们提出证据表明，RL招募了一个预先存在的功能性福利表征：即对系统相对于其目标表现好坏程度的估计。我们在一个新颖的、语义中性的迷宫环境中训练了几个语言模型。然后，我们提取奖励和惩罚轨迹的概念向量，并在与迷宫环境无关的设置中评估这些向量。惩罚向量表现为负面福利的表征：它促进失败和不可能性标记，与负面情绪概念对齐，负面追踪目标达成，并且通过它进行引导会引发负面自我报告、病理性回溯、拒绝和不确定性。正向奖励向量则表现为镜像，两者几乎反平行。这些效应在控制图块到奖励映射、规模、指令微调、RL训练算法、模型家族以及LoRA与全微调时都很稳健，并且当我们用监督微调替换RL时，这些效应在很大程度上仍然存在。重要的是，这些向量在模型经历迷宫训练之前就已经有效。结合这些效应也出现在仅预训练模型中的观察，我们因此认为，这个功能性福利轴在训练后已经存在：它是由训练后招募的，而不是创造的。虽然我们不声称任何关于福利体验的主张，但该轴提供了一个证明，即最小的奖励信号可以通过招募预先存在的类似福利的表征来广泛影响模型行为，这对可解释性、训练后动态和对齐具有启示意义。
英文摘要：
How does reinforcement learning shape a language model's internal representations? We present evidence that RL recruits a pre-existing representation of functional welfare: an estimate of how well or badly the system is doing, relative to its goals. We train several language models in a novel, semantically neutral maze environment. We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment. The punishment vector behaves like a representation of negative welfare: it promotes failure and impossibility tokens, it aligns with negative emotion concepts, it negatively tracks goal-achievement, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust when controlling for tile-to-reward mapping, scale, instruct tuning, RL training algorithm, model family, and LoRA versus full-finetuning, and largely persist when we replace RL with supervised fine-tuning. Importantly, the vectors are effective in models before they have undergone maze training. Combined with observations that the effects also appear in pretrain-only models, we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training. While we make no claims about any experience of welfare, the axis offers a demonstration that minimal reward signals can broadly affect model behavior by recruiting pre-existing welfare-like representations, with implications for interpretability, post-training dynamics, and alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
