Hindsight-DICE：面向深度强化学习的稳定信用分配
Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learning
AI总结：
针对强化学习中稀疏奖励导致的信用分配难题，现有后见之明策略方法存在不稳定问题，本研究提出Hindsight-DICE，通过适配离策略评估的重要性采样比率估计技术实现后见之明分布校正，提升了学习的稳定性与效率。
摘要：
在序列决策问题的环境中，用于指导强化学习智能体的评估反馈通常十分稀疏。在极端情况下，漫长的行为轨迹中仅会出现一个终端反馈信号，导致有意义的奖励观测与促成该奖励的各个行为步骤之间存在显著的时间延迟。应对这类信用分配挑战是强化学习的核心标志性难题之一。\n已有研究提出了后见之明策略（hindsight policies）的概念，构建了一种具备理论依据的方法，可根据对观测轨迹回报的影响对同策略数据进行重加权，但我们发现这类方法存在不稳定性，会导致复杂环境下的学习效率低下。\n本研究将现有的用于离策略评估的重要性采样比率估计技术加以适配，大幅提升了这类所谓后见之明策略方法的稳定性与效率。我们提出的后见之明分布校正方法，可在信用分配问题困扰基线方法的各类环境中，实现稳定、高效的学习。
英文摘要：
Oftentimes, environments for sequential decision-making problems can be quite sparse in the provision of evaluative feedback to guide reinforcement-learning agents. In the extreme case, long trajectories of behavior are merely punctuated with a single terminal feedback signal, leading to a significant temporal delay between the observation of a non-trivial reward and the individual steps of behavior culpable for achieving said reward. Coping with such a credit assignment challenge is one of the hallmark characteristics of reinforcement learning. While prior work has introduced the concept of hindsight policies to develop a theoretically moxtivated method for reweighting on-policy data by impact on achieving the observed trajectory return, we show that these methods experience instabilities which lead to inefficient learning in complex environments. In this work, we adapt existing importance-sampling ratio estimation techniques for off-policy evaluation to drastically improve the stability and efficiency of these so-called hindsight policy methods. Our hindsight distribution correction facilitates stable, efficient learning across a broad range of environments where credit assignment plagues baseline methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
