重审经验可回放条件
Revisiting Experience Replayable Conditions
AI总结：
该研究重审强化学习的经验可回放条件，指出策略提升不稳定性是关键因素，从度量学习视角提出两类稳定化技巧，让经验回放可用于同策略的优势演员-评论家算法，性能比肩顶尖离策略算法软演员-评论家。
摘要：
（深度）强化学习中使用的经验回放（ER）被认为仅适用于离策略算法。但已有部分将ER应用于同策略算法的案例，表明离策略性可能是应用ER的充分条件。本文重新考量了更严格的“经验可回放条件”（ERC），并提出了修改现有算法以满足ERC的方法。据此，本文假设策略提升的不稳定性是ERC的关键影响因素。研究从度量学习的视角揭示了两类不稳定因素：i）来自负样本的排斥力，以及ii）不当经验的回放。相应地，本文推导了对应的稳定化技巧。最终通过数值模拟证实，所提出的稳定化技巧使ER可应用于优势演员-评论家（一种同策略算法），且其学习性能与最先进的离策略算法软演员-评论家相当。
英文摘要：
Experience replay (ER) used in (deep) reinforcement learning is considered to be applicable only to off-policy algorithms. However, there have been some cases in which ER has been applied for on-policy algorithms, suggesting that off-policyness might be a sufficient condition for applying ER. This paper reconsiders more strict "experience replayable conditions" (ERC) and proposes the way of modifying the existing algorithms to satisfy ERC. In light of this, it is postulated that the instability of policy improvements represents a pivotal factor in ERC. The instability factors are revealed from the viewpoint of metric learning as i) repulsive forces from negative samples and ii) replays of inappropriate experiences. Accordingly, the corresponding stabilization tricks are derived. As a result, it is confirmed through numerical simulations that the proposed stabilization tricks make ER applicable to an advantage actor-critic, an on-policy algorithm. Moreover, its learning performance is comparable to that of a soft actor-critic, a state-of-the-art off-policy algorithm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
