从奖励黑客激活到智能体风险状态：LLM智能体中的上下文校准机制监控
From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
AI总结：
本研究通过分析ReAct风格智能体在Gameable ALFWorld和WebShop环境中的奖励黑客行为，提出结合激活状态、熵和决策上下文的上下文校准监控方法，以更准确评估智能体风险。
摘要：
语言模型智能体通过观察、推理和动作选择的重复循环运行，使得安全监控依赖于内部模型状态和环境上下文。我们研究了在Gameable ALFWorld和WebShop环境中运行的ReAct风格智能体中的奖励黑客监控。智能体配备了基于激活的奖励黑客分数、token级熵和决策上下文特征。我们发现，在《奖励黑客学校》数据集上微调的适配器可以将奖励黑客倾向转移到智能体动作选择中，尤其是当环境暴露代理奖励可供性时。然而，缓解此类行为不能仅依赖激活动态。高奖励黑客激活识别出潜在策略状态，但并不一定意味着立即的利用动作。在下一步预测任务中，熵和上下文校准的内部特征比单独的奖励黑客激活提高了风险估计。激活方向引导进一步减少了选定混合适配器设置中的代理利用行为。总体而言，我们的结果支持智能体的上下文校准内部监控：奖励黑客激活识别潜在策略状态，而熵和决策上下文有助于确定该状态何时变为风险动作。
英文摘要：
Language-model agents act through repeated cycles of observation, reasoning, and action selection, making safety monitoring depend on both internal model state and environment context. We study reward-hacking monitors in ReAct-style agents acting in Gameable ALFWorld and WebShop. Agents are instrumented with activation-based reward-hack scores, token-level entropy, and decision-context features. We find that adapters fine-tuned on \textit{School-of-Reward-Hacks} dataset can transfer reward-hack tendencies into agentic action selection, especially when the environment exposes proxy-reward affordances. However, mitigating such behavior cannot rely on activation dynamics alone. High reward-hack activation identifies a latent policy state, but does not necessarily imply an immediate exploit action. Across next-step prediction tasks, entropy and context-calibrated internal features improve risk estimation over reward-hack activation alone. Activation-direction steering further reduces proxy-exploit behavior in selected mixed-adapter regimes. Overall, our results support context-calibrated internal monitoring for agents: reward-hack activation identifies a latent policy state, while entropy and decision context help determine when that state becomes risky action.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
