arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.01178cs.AIcs.LGcs.MA
收藏

多智能体逆强化学习:次优演示和替代解决方案概念

Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts

Stanford Existential Risk Initiative(斯坦福存在风险倡议);Stanford University(斯坦福大学)

Sage Bergerson

摘要:

多智能体逆强化学习(MIRL)可以用来从社会环境中的智能体学习奖励函数。为了模拟现实的社会动态,MIRL方法必须考虑次优的人类推理和行为。博弈论的传统形式主义提供了计算上易于处理的行为模型,但假设代理具有不切实际的认知能力。本研究识别和比较MIRL方法中的机制,a)处理代理决策中的噪声,偏见和竞争,b)模拟现实的平衡解决方案的概念。MIRL的研究进行了系统的审查,以确定这些挑战的解决方案。这些研究的方法和结果进行了分析和比较的基础上的因素,包括性能的准确性,效率和描述质量。我们发现,处理噪音,偏见和反倾销的MIRL的主要方法是最大熵(MaxEnt)IRL的多智能体设置的扩展。我们还发现,许多成功的解决方案的概念是传统的纳什均衡(NE)的推广。这些解包括相关平衡、Logistic随机最佳反应平衡和熵正则化平均场NE。使用递归推理或更新的方法也表现良好,包括反馈NE和归档多智能体对抗IRL。在单智能IRL中成功地建模特定的偏见和偏见,并在MIRL中使用心理理论方法取得了可喜的结果,这意味着建模特定的偏见和偏见可能是有用的。灵活性和公正的推理,在确定的替代解决方案的概念表明,一个解决方案的概念,既有递归和广义的特点,可以很好地表现在建模现实的社会互动。

英文摘要:

Multi-agent inverse reinforcement learning (MIRL) can be used to learn reward functions from agents in social environments. To model realistic social dynamics, MIRL methods must account for suboptimal human reasoning and behavior. Traditional formalisms of game theory provide computationally tractable behavioral models, but assume agents have unrealistic cognitive capabilities. This research identifies and compares mechanisms in MIRL methods which a) handle noise, biases and heuristics in agent decision making and b) model realistic equilibrium solution concepts. MIRL research is systematically reviewed to identify solutions for these challenges. The methods and results of these studies are analyzed and compared based on factors including performance accuracy, efficiency, and descriptive quality. We found that the primary methods for handling noise, biases and heuristics in MIRL were extensions of Maximum Entropy (MaxEnt) IRL to multi-agent settings. We also found that many successful solution concepts are generalizations of the traditional Nash Equilibrium (NE). These solutions include the correlated equilibrium, logistic stochastic best response equilibrium and entropy regularized mean field NE. Methods which use recursive reasoning or updating also perform well, including the feedback NE and archive multi-agent adversarial IRL. Success in modeling specific biases and heuristics in single-agent IRL and promising results using a Theory of Mind approach in MIRL imply that modeling specific biases and heuristics may be useful. Flexibility and unbiased inference in the identified alternative solution concepts suggest that a solution concept which has both recursive and generalized characteristics may perform well at modeling realistic social interactions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑