arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.07996cs.LG
收藏

奖励兼容性:一种逆强化学习(IRL)框架

Reward Compatibility: A Framework for Inverse RL

Politecnico di Milano(米兰理工大学);Technion(以色列理工学院)

Filippo Lazzati, Mirco Mutti, Alberto Metelli

AI总结:

该研究提出奖励兼容性这一新颖IRL框架,推广了传统可行奖励集概念,实现了从表格型到大规模MDP的可证明高效IRL拓展,并针对多种演示与数据收集设定给出了可处理算法及样本复杂度分析。

摘要:

我们通过奖励兼容性这一量化奖励与给定专家演示兼容性的新颖框架,对逆强化学习(Inverse Reinforcement Learning,IRL)开展了原创理论研究。直观而言,若某奖励下计算得到的专家策略性能越接近该奖励下的最优性能,则该奖励与演示的兼容性越高。这一概念推广了理论IRL文献中最常见的可行奖励集框架——后者仅判定奖励兼容或不兼容。奖励兼容性引入的“灰度”是将可证明高效的IRL适用范围从表格型MDP拓展到大规模MDP的关键,远超可行奖励集所能达到的边界。我们在多种设定下分析了IRL问题,包括最优与次优专家演示、在线与离线数据收集模式。针对所有这些维度,我们提出了可处理的算法并给出了相应的样本复杂度分析,同时提供了关于奖励兼容性的诸多洞见,以及该框架如何为更通用的问题设定铺平道路的思考。

英文摘要:

We provide an original theoretical study of Inverse Reinforcement Learning (IRL) through the lens of reward compatibility, a novel framework to quantify the compatibility of a reward with the given expert's demonstrations. Intuitively, a reward is more compatible with the demonstrations the closer the performance of the expert's policy computed with that reward is to the optimal performance for that reward. This generalizes the notion of feasible reward set, the most common framework in the theoretical IRL literature, for which a reward is either compatible or not compatible. The grayscale introduced by the reward compatibility is the key to extend the realm of provably efficient IRL far beyond what is attainable with the feasible reward set: from tabular to large-scale MDPs. We analyze the IRL problem across various settings, including optimal and suboptimal expert's demonstrations and both online and offline data collection. For all of these dimensions, we provide a tractable algorithm and corresponding sample complexity analysis, as well as various insights on reward compatibility and how the framework can pave the way to yet more general problem settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑