arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.12966cs.LG
收藏

迈向逆强化学习的理论理解

Towards Theoretical Understanding of Inverse Reinforcement Learning

Politecnico di Milano(米兰理工大学)

Alberto Maria Metelli, Filippo Lazzati, Marcello Restelli

AI总结:

本文研究具备生成模型的有限视野IRL中可行奖励集的PAC估计,给出极小极大样本复杂度下界,并证明US-IRL达到至多相差对数因子的匹配上界。

摘要:

逆强化学习(IRL)是指一类功能强大的算法,用于恢复能够证明智能体专家所展示行为合理性的奖励函数。IRL的一个众所周知的局限是奖励函数选择上的模糊性,这是由于存在多种可解释所观察行为的奖励。最近,通过将IRL表述为估计可行奖励集(即与专家行为相容的奖励区域)的问题,这一局限已被规避。在本文中,我们针对具备生成模型的有限视野问题,朝着弥合IRL的理论空白迈出了一步。我们首先正式介绍估计可行奖励集的问题、相应的PAC要求,并讨论特定奖励类别的性质。随后,我们给出了估计可行奖励集问题的首个极小极大样本复杂度下界,其阶为 $Ω\Bigl( \frac{H^3SA}{ε^2} \bigl( \log \bigl(\frac{1}δ\bigl) + S \bigl)\Bigl)$,其中 $S$ 和 $A$ 分别为状态数和动作数,$H$ 为视野,$ε$ 为期望精度,$δ$ 为置信度。我们分析了均匀采样策略(US-IRL)的样本复杂度,证明了一个在对数因子范围内相匹配的上界。最后,我们概述了IRL中的若干开放问题,并提出了未来研究方向。

英文摘要:

Inverse reinforcement learning (IRL) denotes a powerful family of algorithms for recovering a reward function justifying the behavior demonstrated by an expert agent. A well-known limitation of IRL is the ambiguity in the choice of the reward function, due to the existence of multiple rewards that explain the observed behavior. This limitation has been recently circumvented by formulating IRL as the problem of estimating the feasible reward set, i.e., the region of the rewards compatible with the expert's behavior. In this paper, we make a step towards closing the theory gap of IRL in the case of finite-horizon problems with a generative model. We start by formally introducing the problem of estimating the feasible reward set, the corresponding PAC requirement, and discussing the properties of particular classes of rewards. Then, we provide the first minimax lower bound on the sample complexity for the problem of estimating the feasible reward set of order $Ω\Bigl( \frac{H^3SA}{ε^2} \bigl( \log \bigl(\frac{1}δ\bigl) + S \bigl)\Bigl)$, being $S$ and $A$ the number of states and actions respectively, $H$ the horizon, $ε$ the desired accuracy, and $δ$ the confidence. We analyze the sample complexity of a uniform sampling strategy (US-IRL), proving a matching upper bound up to logarithmic factors. Finally, we outline several open questions in IRL and propose future research directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑