arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.14599cs.LGcs.AIstat.ML
收藏

逆强化学习的快速收敛率

Fast Rates for Inverse Reinforcement Learning

EPFL(瑞士联邦理工学院)

Andreas Schlaginhaufen, Maryam Kamgarpour

AI总结:

本文研究了熵正则化min-max逆强化学习在有限时间 horizon MDPs中的结构和统计特性,证明了在有限样本下,轨迹级KL散度和参数误差以O(n^{-1})速度收敛。

摘要:

我们建立了熵正则化min-max逆强化学习(Min-Max-IRL)在有限时间 horizon MDPs中的新型结构和统计结果,其中奖励类线性且状态和动作空间为Borel空间。在结构方面,我们证明在总体水平和确定性动态下,最大似然估计(MLE)和Min-Max-IRL是等价的。在统计方面,利用Min-Max-IRL损失的伪自共轭性,我们证明轨迹级KL散度和Hessian范数下的平方参数误差以O(n^{-1})的速度衰减,其中n是专家轨迹数量。我们的保证适用于不规范情况且不需要探索假设。我们进一步将奖励可识别性结果扩展到一般Borel空间,并推导了软最优价值函数对奖励参数导数的新结果。

英文摘要:

We establish novel structural and statistical results for entropy-regularized min-max inverse reinforcement learning (Min-Max-IRL) in finite-horizon MDPs with Borel state and action spaces. We show that maximum likelihood estimation (MLE) and Min-Max-IRL are equivalent at the population level, and at the empirical level under deterministic dynamics. For linear reward classes, we leverage pseudo-self-concordance of the Min-Max-IRL loss to prove that both the excess trajectory-level KL divergence and the squared parameter error in the Hessian norm decay at the fast rate $O(n^{-1})$, where $n$ is the number of expert trajectories. A local minimax lower bound matches the parameter-error rate up to logarithmic factors in the well-specified deterministic setting. Our guarantees apply under misspecification and require no uniform state-coverage assumption. We further extend reward-identifiability results to general Borel spaces and compare our results with MLE-based guarantees.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑