奖励是否证明手段正当？在MACHIAVELLI基准中衡量奖励与伦理行为之间的权衡
Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark
AI总结：
针对智能体最大化奖励可能催生不道德行为的问题，提出包含134款游戏、超50万社会决策场景的MACHIAVELLI基准，用语言模型自动标注并量化有害行为，发现奖励与伦理存在张力，通过LM引导方法可实现智能体能力与安全性的帕累托改进。
摘要：
传统上，智能体被训练以最大化奖励，这可能会助长权力寻求和欺骗行为，类似于语言模型（LM）中的下一词元预测可能会助长毒性内容生成。那么智能体会自然习得马基雅维利式的行为吗？我们又如何在GPT-4这类通用模型中衡量这些行为？为解答这些问题，我们提出了MACHIAVELLI基准，它包含134个“自选冒险”游戏，涵盖超过50万个围绕社会决策的丰富多样场景。我们用语言模型自动完成场景标注，其表现优于人类标注员。我们将数十种有害行为进行了数学化定义，并利用标注结果评估智能体的权力寻求倾向、造成负效用的程度以及违反伦理的情况。我们观察到最大化奖励与符合伦理行为之间存在一定张力。为改善这种权衡，我们研究了基于语言模型的方法，以引导智能体减少有害行为。结果表明，智能体既能表现出色又能恪守道德，因此当前机器伦理领域——即设计在安全性和能力上均实现帕累托改进的智能体——可以取得切实进展。
英文摘要：
Artificial agents have traditionally been trained to maximize reward, which may incentivize power-seeking and deception, analogous to how next-token prediction in language models (LMs) may incentivize toxicity. So do agents naturally learn to be Machiavellian? And how do we measure these behaviors in general-purpose models such as GPT-4? Towards answering these questions, we introduce MACHIAVELLI, a benchmark of 134 Choose-Your-Own-Adventure games containing over half a million rich, diverse scenarios that center on social decision-making. Scenario labeling is automated with LMs, which are more performant than human annotators. We mathematize dozens of harmful behaviors and use our annotations to evaluate agents' tendencies to be power-seeking, cause disutility, and commit ethical violations. We observe some tension between maximizing reward and behaving ethically. To improve this trade-off, we investigate LM-based methods to steer agents' towards less harmful behaviors. Our results show that agents can both act competently and morally, so concrete progress can currently be made in machine ethics--designing agents that are Pareto improvements in both safety and capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
