与人类Zero-Shot合作,假设人类有偏见
Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased
摘要:
最近的趋势是应用多智能体强化学习(MARL)来训练能够以zero-shot方式与人类合作而不使用任何人类数据的智能体。典型的工作流程是首先重复运行self-play(SP)以构建策略池,然后针对该池训练最终的自适应策略。这个框架的一个关键限制是,池中的每个策略都是相对最优的。环境奖励函数,它隐含地假设自适应策略的测试伙伴也将精确地优化相同的奖励函数。然而,人类的目标往往根据自己的偏好而有很大的偏差,这可能与环境奖励有很大的不同。我们提出了一个更一般的框架,隐藏的效用自我发挥(HSP),明确的模型作为隐藏的奖励功能的自我发挥目标的人类偏见。通过将奖励空间近似为线性函数,HSP采用了一种有效的技术来生成具有偏置策略的扩展策略池。我们在Overcooked基准上评估HSP。实证结果表明,我们的HSP方法产生更高的回报比基线时,与学习的人类模型,手动脚本的政策,和真正的人合作。HSP政策也被评为基于人类反馈的最具辅助性的政策。
英文摘要:
There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an agent that can cooperate with humans in a zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be precisely optimizing the same reward function as well. However, human objectives are often substantially biased according to their own preferences, which can differ greatly from the environment reward. We propose a more general framework, Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts an effective technique to generate an augmented policy pool with biased policies. We evaluate HSP on the Overcooked benchmark. Empirical results show that our HSP method produces higher rewards than baselines when cooperating with learned human models, manually scripted policies, and real humans. The HSP policy is also rated as the most assistive policy based on human feedback.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。