深度强化学习中时序信用分配综述
A Survey of Temporal Credit Assignment in Deep Reinforcement Learning
AI总结：
本文综述深度强化学习中的时序信用分配问题，提出统一形式化框架以公平比较现有算法，分析延迟效应、转置和动作影响缺失等挑战，并总结评估协议与未来研究方向。
摘要：
信用分配问题（CAP）指的是强化学习（RL）智能体将动作与其长期后果关联起来的长期挑战。解决CAP是RL在现实世界中成功部署的关键一步，因为大多数决策问题提供的反馈具有噪声大、延迟高且几乎不包含原因信息的特点。这些条件使得难以区分偶然结果与由明智决策所导致的结果。然而，信用的数学本质以及CAP仍然未被充分理解和定义。在本综述中，我们回顾了深度RL中时序信用分配（CA）的最新研究进展。我们提出了一种统一的信用形式化框架，使得对最先进算法的公平比较成为可能，并增进了我们对各种方法之间权衡的理解。我们将CAP表述为从有限经验中学习一个动作对某个结果的影响的问题。我们讨论了延迟效应、转置以及动作影响缺失所带来的挑战，并分析了现有方法如何旨在解决这些问题。最后，我们综述了评估信用分配方法的协议，并提出了诊断不同方法困难来源的方法。总体而言，本综述为初入该领域的从业者和研究人员提供了领域概览，为希望加快CAP新研究起步阶段的学者提供了一致的视角，并为未来研究提出了潜在方向。
英文摘要：
The Credit Assignment Problem (CAP) refers to the longstanding challenge of Reinforcement Learning (RL) agents to associate actions with their long-term consequences. Solving the CAP is a crucial step towards the successful deployment of RL in the real world since most decision problems provide feedback that is noisy, delayed, and with little or no information about the causes. These conditions make it hard to distinguish serendipitous outcomes from those caused by informed decision-making. However, the mathematical nature of credit and the CAP remains poorly understood and defined. In this survey, we review the state of the art of Temporal Credit Assignment (CA) in deep RL. We propose a unifying formalism for credit that enables equitable comparisons of state-of-the-art algorithms and improves our understanding of the trade-offs between the various methods. We cast the CAP as the problem of learning the influence of an action over an outcome from a finite amount of experience. We discuss the challenges posed by delayed effects, transpositions, and a lack of action influence, and analyse how existing methods aim to address them. Finally, we survey the protocols to evaluate a credit assignment method and suggest ways to diagnose the sources of struggle for different methods. Overall, this survey provides an overview of the field for new-entry practitioners and researchers, it offers a coherent perspective for scholars looking to expedite the starting stages of a new study on the CAP, and it suggests potential directions for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
