对 Machiavellian 代理进行对齐：通过测试时策略塑造实现行为引导
Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shaping
AI总结：
本文提出了一种测试时策略塑造方法，通过模型引导的策略调整，解决预训练代理在复杂环境中的伦理对齐问题，实现奖励最大化与伦理约束的平衡。
摘要：
决策制定AI代理的部署面临在复杂动态环境中维持与人类价值观或指导方针对齐的关键挑战。仅训练以实现其目标的代理可能会采取有害行为，暴露了在最大化奖励函数和维持对齐之间的重要权衡。对于预训练的代理，确保对齐尤其具有挑战性，因为再训练可能是一个成本高且缓慢的过程。这进一步复杂化了代表对齐的伦理价值观的多样性和潜在冲突属性。为了解决这些挑战，我们提出了一种基于模型引导的策略塑造的测试时对齐技术。我们的方法允许对个体行为属性进行精确控制，能够在不同的强化学习（RL）环境中泛化，并且在不需代理再训练的情况下，促进伦理对齐与奖励最大化之间的原则性权衡。我们使用MACHIAVELLI基准测试评估我们的方法，该基准测试包含134个基于文本的游戏环境和成千上万个涉及道德决策的注释场景。RL代理首先被训练以最大化其各自游戏的奖励。在测试时，我们通过场景-动作属性分类器应用策略塑造，以确保决策与伦理属性的一致性。我们比较了我们的方法与先前训练时方法和通用代理，并研究了几种类型的伦理违规行为和权力寻求行为。我们的结果表明，测试时的策略塑造为在多样化的环境和对齐属性中减轻不道德行为提供了一种有效且可扩展的解决方案。
英文摘要：
The deployment of decision-making AI agents presents a critical challenge in maintaining alignment with human values or guidelines while operating in complex, dynamic environments. Agents trained solely to achieve their objectives may adopt harmful behavior, exposing a key trade-off between maximizing the reward function and maintaining alignment. For pre-trained agents, ensuring alignment is particularly challenging, as retraining can be a costly and slow process. This is further complicated by the diverse and potentially conflicting attributes representing the ethical values for alignment. To address these challenges, we propose a test-time alignment technique based on model-guided policy shaping. Our method allows precise control over individual behavioral attributes, generalizes across diverse reinforcement learning (RL) environments, and facilitates a principled trade-off between ethical alignment and reward maximization without requiring agent retraining. We evaluate our approach using the MACHIAVELLI benchmark, which comprises 134 text-based game environments and thousands of annotated scenarios involving ethical decisions. The RL agents are first trained to maximize the reward in their respective games. At test time, we apply policy shaping via scenario-action attribute classifiers to ensure decision alignment with ethical attributes. We compare our approach against prior training-time methods and general-purpose agents, as well as study several types of ethical violations and power-seeking behavior. Our results demonstrate that test-time policy shaping provides an effective and scalable solution for mitigating unethical behavior across diverse environments and alignment attributes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
