模仿过去的成功可能非常不理想
Imitating Past Successes can be Very Suboptimal
摘要：
先前的工作提出了一种简单的强化学习（RL）策略：用该经验中获得的结果标记经验，然后模仿重新标记的经验。这些基于结果的模仿学习方法因其简单、性能强以及与监督学习的紧密联系而备受关注。然而，目前还不清楚这些方法如何与标准RL目标，奖励最大化。在本文中，我们正式与结果条件模仿学习奖励最大化，绘制一个精确的学习政策和Q值之间的关系，并解释这些方法和以前的EM为基础的政策搜索方法之间的密切联系。该分析表明，现有的以结果为条件的模仿学习方法不一定改善策略，但是在某些假设下，简单的修改导致确实保证策略改善的方法。
英文摘要：
Prior work has proposed a simple strategy for reinforcement learning (RL): label experience with the outcomes achieved in that experience, and then imitate the relabeled experience. These outcome-conditioned imitation learning methods are appealing because of their simplicity, strong performance, and close ties with supervised learning. However, it remains unclear how these methods relate to the standard RL objective, reward maximization. In this paper, we formally relate outcome-conditioned imitation learning to reward maximization, drawing a precise relationship between the learned policy and Q-values and explaining the close connections between these methods and prior EM-based policy search methods. This analysis shows that existing outcome-conditioned imitation learning methods do not necessarily improve the policy, but a simple modification results in a method that does guarantee policy improvement, under some assumptions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
