arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.04289cs.AIcs.RO
收藏

通过推断多智能体游戏中的潜在智力水平来学习人类奖励:一种应用于驱动数据的心理理论方法

Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

University of California, Berkeley(加利福尼亚大学伯克利分校)

Ran Tian, Masayoshi Tomizuka, Liting Sun

摘要:

奖励函数作为一种识别人的能动性并使人的行为合理化的激励表示方法,在人机交互中对人的行为建模具有重要意义。逆强化学习是从演示中获取奖励函数的有效方法。然而,将其应用于多智能体环境时一直具有挑战性,因为智能体之间的相互影响必须进行适当的建模。为了应对这一挑战,以前的工作要么利用均衡解决方案的概念,假设人类是完全理性的优化与无限的智慧或预先分配人类的互动策略先验。在这项工作中,我们主张,人类是有限理性的,并有不同的智力水平时,推理别人的决策过程中,这样一个内在的和潜在的特点,应占奖励学习算法。因此,我们利用这样的见解,从理论的心灵,并提出了一个新的多智能体逆强化学习框架,原因人类的潜在智力水平在学习过程中。我们验证了我们的方法在零和和一般和游戏与合成代理,并说明了一个实际的应用程序来学习人类司机的奖励功能,从真实的驾驶数据。我们比较我们的方法与两个基线算法。实验结果表明,通过对人类潜在智力水平的推理,该方法具有更大的灵活性,能够更好地提取出解释人类驾驶行为的奖励函数。

英文摘要:

Reward function, as an incentive representation that recognizes humans' agency and rationalizes humans' actions, is particularly appealing for modeling human behavior in human-robot interaction. Inverse Reinforcement Learning is an effective way to retrieve reward functions from demonstrations. However, it has always been challenging when applying it to multi-agent settings since the mutual influence between agents has to be appropriately modeled. To tackle this challenge, previous work either exploits equilibrium solution concepts by assuming humans as perfectly rational optimizers with unbounded intelligence or pre-assigns humans' interaction strategies a priori. In this work, we advocate that humans are bounded rational and have different intelligence levels when reasoning about others' decision-making process, and such an inherent and latent characteristic should be accounted for in reward learning algorithms. Hence, we exploit such insights from Theory-of-Mind and propose a new multi-agent Inverse Reinforcement Learning framework that reasons about humans' latent intelligence levels during learning. We validate our approach in both zero-sum and general-sum games with synthetic agents and illustrate a practical application to learning human drivers' reward functions from real driving data. We compare our approach with two baseline algorithms. The results show that by reasoning about humans' latent intelligence levels, the proposed approach has more flexibility and capability to retrieve reward functions that explain humans' driving behaviors better.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑