逆向Q学习：在线模仿学习中的Q^π-可实现MDP
Inverse Q-Learning Done Right: Offline Imitation Learning in $Q^π$-Realizable MDPs
AI总结：
本文提出SPOIL算法，通过线性Q^π-可实现MDP的结构性假设，在有限样本下实现与专家策略性能匹配，并在非线性情况下扩展，展示其在深度模仿学习中的优越性能。
摘要：
我们研究了马尔可夫决策过程（MDP）中的在线模仿学习问题，目标是给定由专家策略生成的状态-动作对数据集，学习一个性能良好的策略。补充最近在该领域工作的线性Q^π-可实现MDP的假设，我们从新的角度出发，并利用对环境的不同结构性假设。具体而言，对于线性Q^π-可实现MDP类，我们引入了一种新的算法，称为鞍点在线模仿学习（SPOIL），该算法在访问O(ε^{-2})样本的情况下，能够保证与任何专家策略的性能匹配，最多误差为ε。此外，我们还将这一结果扩展到可能非线性的Q^π-可实现MDP，代价是样本复杂性为O(ε^{-4})。最后，我们的分析提出了一种新的损失函数，用于从专家数据中训练批评网络。在标准基准上的经验评估表明，SPOIL的神经网络实现优于行为克隆，并且与最先进的算法相竞争。
英文摘要：
We study the problem of offline imitation learning in Markov decision processes (MDPs), where the goal is to learn a well-performing policy given a dataset of state-action pairs generated by an expert policy. Complementing a recent line of work on this topic that assumes the expert belongs to a tractable class of known policies, we approach this problem from a new angle and leverage a different type of structural assumption about the environment. Specifically, for the class of linear $Q^π$-realizable MDPs, we introduce a new algorithm called saddle-point offline imitation learning (\SPOIL), which is guaranteed to match the performance of any expert up to an additive error $\varepsilon$ with access to $\mathcal{O}(\varepsilon^{-2})$ samples. Moreover, we extend this result to possibly nonlinear $Q^π$-realizable MDPs at the cost of a worse sample complexity of order $\mathcal{O}(\varepsilon^{-4})$. Finally, our analysis suggests a new loss function for training critic networks from expert data in deep imitation learning. Empirical evaluations on standard benchmarks demonstrate that the neural net implementation of \SPOIL is superior to behavior cloning and competitive with state-of-the-art algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
