迈向因果信用分配
Towards Causal Credit Assignment
AI总结：
本论文实证研究了后见信用分配方法，将其应用于基于环境因果结构的因子化状态表示，并提出一种有效利用因果结构的变体，大幅提升了效率并在多种任务中优于基线方法。
摘要：
根据行动对未来结果的贡献为其充分分配信用，是强化学习中一个长期存在的开放性挑战。最常用的信用分配方法的假设在决策效果不能立即显现的任务中处于不利地位。此外，该方法只能评估智能体已选择的行动，使其效率极低。尽管如此，该领域尚未广泛采用其他替代方法。后见信用分配是一个有前景但尚未探索的候选方案，旨在解决长期信用分配和反事实信用分配的问题。在本论文中，我们对后见信用分配进行实证研究，以确定其主要优势和改进要点。然后，我们将其应用于因子化状态表示，特别是基于环境因果结构的状态表示。在此设置下，我们提出了一种后见信用分配的变体，能够有效利用给定的因果结构。我们表明，我们的修改大幅减少了后见信用分配的工作量，使其更高效，并使其能够在各种任务中优于基线信用分配方法。这为基于给定或学习到的因果结构的其他方法开辟了道路。
英文摘要：
Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are disadvantageous in tasks where the effects of decisions are not immediately evident. Furthermore, this method can only evaluate actions that have been selected by the agent, making it highly inefficient. Still, no alternative methods have been widely adopted in the field. Hindsight Credit Assignment is a promising, but still unexplored candidate, which aims to solve the problems of both long-term and counterfactual credit assignment. In this thesis, we empirically investigate Hindsight Credit Assignment to identify its main benefits, and key points to improve. Then, we apply it to factored state representations, and in particular to state representations based on the causal structure of the environment. In this setting, we propose a variant of Hindsight Credit Assignment that effectively exploits a given causal structure. We show that our modification greatly decreases the workload of Hindsight Credit Assignment, making it more efficient and enabling it to outperform the baseline credit assignment method on various tasks. This opens the way to other methods based on given or learned causal structures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
