发表机构
Reciprocal Research; University of Warwick（互惠研究; 华威大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过激活引导实验，在七个开放权重模型中证明隐藏价值状态能按剂量比例影响后续选择，且该效应在DPO训练中涌现，模型会主动移除消极状态，揭示了价值与目标导向行为的联系。
AI 中文摘要
语言模型将某些内部状态描述为好的，而将其他状态描述为坏的。但模型是否对这些状态有利益关系，仍是一个开放性问题。简单地询问模型不太可能提供有效信息。任何回答都可能与真正的内省、表面的模式匹配，或在角色训练中学到的固定脚本一致。因此，我们研究显示性偏好。我们不询问状态，而是使用激活引导将积极或消极价值的激活模式附加到两个原本无意义的“区域”之一，关闭引导，然后观察模型偏好哪个区域。对该状态有利益关系的模型应相应选择。在来自五个家族的七个开放权重模型中，我们确实发现了这一点。首先，引导改变了模型对每个区域撰写的段落，这些词语随后影响了后续选择。其次，当所有表面层面的标记保持不变且仅隐藏的KV缓存不同时，这种转变仍然存在。第三，当所有文本在无引导下生成且仅在缓存构建期间注入价值时，这种效应也仍然存在。因此，仅隐藏状态就能按引导剂量成比例地移动选择。第四，这种选择对隐藏价值的依赖性在基础模型中几乎不存在，并在DPO训练期间出现，这与训练中形成的价值与目标导向行为之间的联系一致。最后，当模型拥有自我引导的工具时，它并不倾向于诱导积极状态，但会可靠地移除强加的消极状态。它以剂量依赖的速率这样做，并且显著地比移除随机方向的干预更频繁。总体而言，我们证明了与价值相关的激活模式留下隐藏痕迹，这些痕迹可预测地控制后续选择，即使在所有可见标记在条件下都相同的情况下也是如此。这些痕迹是否伴随任何与模型福利相关的主观体验仍不清楚。
英文摘要
Language models describe some internal states as good and others as bad. But whether models have a stake in them is an open question. Simply asking models is unlikely to be informative. Any answer may be consistent with genuine introspection, superficial pattern-matching, or with fixed scripts learned in character training. We therefore study revealed preference. Rather than asking about a state, we use activation steering to attach a positively or negatively valenced activation pattern to one of two otherwise meaningless 'zones', switch steering off, and then observe which zone the model prefers. A model with a stake in that state should choose accordingly. Across seven open-weight models from five families, this is indeed what we find. First, steering changes the passages models write about each zone, and those words shift later choice. Second, this shift persists when all surface-level tokens are held fixed and only the hidden KV cache differs. Third, the effect also remains when all text is generated without steering and valence is only injected during cache construction. Thus, the hidden state is sufficient to move choice in proportion to the steering dose. Fourth, the dependence of choice on hidden valence is nearly absent in a base model and emerges during DPO, consistent with a link between valence and goal-directed behaviour formed in training. Finally, given tools to self-steer, a model does not tend to induce a positive state, but it regularly removes an imposed negative state. It does so at a dose-dependent rate and significantly more often than it removes random-direction interventions. Overall we demonstrate that valence-related activation patterns leave hidden traces that predictably govern later choices, even when all visible tokens are identical across conditions. Whether these traces are accompanied by any subjective experience relevant to model welfare remains unclear.
