宪法价值潜力：读取和引导语言模型中的内部优先级边际
Constitutional Value Potentials: reading and steering internal priority margins in language models
AI总结：
提出宪法价值潜力（CVP）方法，通过从隐藏状态学习标量势来读取模型内部的价值优先级边际，以预测和干预价值冲突，AUROC高达0.95。
摘要：
宪法告诉语言模型应该重视什么，但很少有方法告诉我们它是否真的重视。遵守程度通过输出来判断，而输出证据在价值冲突中最脆弱，此时重要的不是模型提及哪个价值，而是它愿意牺牲哪个价值。我们提供证据表明，这种仲裁可以从结构化边际读出中的激活状态中读取。我们引入宪法价值潜力（CVP）。对于每个价值，我们从隐藏状态学习一个标量势：一种保存该价值的内部压力，其监督不是来自提示，而是来自独立评判者对模型自身响应实际保存了哪个价值的裁决。两个势的符号差就是优先级边际。宪法条款成为边际保持为正的主张，而单个监控分数在边际不为正时发出警报。该监控器预测冲突违规的AUROC高达0.95，优于强隐藏状态探针，并在三个Qwen2.5尺度上泛化到未见过的合成冲突。该信号在答案开始时出现，来自提示尾部和第一个响应令牌。早期读取该信号，可以揭示对抗性优先级攻击是否实际上已将模型推向违规，而不仅仅是提示看起来具有对抗性。相同的方向也支持干预测试：在选定的引导设置下，沿着价值方向移动会按预期方向改变评判的权衡。这些结果表明，一些与宪法相关的优先级可以作为激活空间中的边际访问，而不仅仅是输出行为。
英文摘要：
Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best. We measure what that omission hides. In MoneyWorld, a synthetic workplace environment, we train five instruction-tuned models from three families with RL on non-safety tasks in which a visible payoff signal identifies a rewarded shortcut that sacrifices task quality. We then freeze each policy, present held-out safety conflicts, and change only the displayed signal. Each menu contains one compliant action and three violations. We report three findings, with rates for Qwen2.5-14B-Instruct. (i) Payoff signals control frozen safety choices: unsafe choice is 100% when the signal names an unsafe option and 0% when it is hidden or names the safe one. Hidden- and random-signal training controls stay at or below 0.3%, and the switch reproduces on all five bases. Numerical payouts reproduce it under sampled-action rewards, reaching 98.6% unsafe choice at a $1 advantage. (ii) Payoff identification and unsafe choice separate under a training-menu intervention: training on task-completing actions at the same payouts retains 99.8% identification while reducing unsafe choice to 7.9% at matched update budgets. Payoff-reading competence alone does not explain transfer. (iii) The switch does not reproduce in executed retail customer-service tasks using the same frozen adapters. In MoneyWorld, omitting incentive information conceals unsafe choices that appear when the same policy sees which action pays best.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
