arXiv 2306.168032023-10-31cs.LGstat.ML

我能获得那个奖励吗？基于反事实贡献分析的长期信用分配

Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis

Google DeepMind（谷歌DeepMind）；Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）；Princeton Neuroscience Institute, Princeton University（普林斯顿大学普林斯顿神经科学研究所）；Department of Psychology, Princeton University（普林斯顿大学心理学系）

Alexander Meulemans, Simon Schug, Seijin Kobayashi, Nathaniel Daw, Gregory Wayne

备注：: NeurIPS 2023 spotlight

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出反事实贡献分析（COCOA）方法，通过量化动作对后续奖励的贡献改进长期信用分配，实验证明其相比HCA具有更低偏差和方差，为样本高效强化学习开辟新路径。

摘要：

为了使强化学习更具样本效率，我们需要更好的信用分配方法来衡量某个动作对未来奖励的影响。在事后信用分配（HCA）的基础上，我们提出了反事实贡献分析（COCOA），这是一个新的基于模型的信用分配算法家族。我们的算法通过量化一个反事实查询来衡量动作在获得后续奖励时的贡献：“如果智能体采取了另一个动作，它是否仍能达到这个奖励？”我们表明，像HCA那样针对奖励状态来衡量贡献，会产生虚假的贡献估计，导致HCA在许多相关环境中退化为高方差的REINFORCE估计器。相反，我们针对奖励或奖励对象的学习表示来衡量贡献，从而得到方差更低的梯度估计。我们在一组专门设计用于评估长期信用分配能力的问题上进行了实验。通过使用动态规划，我们测量了真实策略梯度，并表明我们新的基于模型的信用分配方法的性能提升源于与HCA和常见基线相比更低的偏差和方差。我们的结果展示了如何利用对动作朝向奖励结果的贡献建模来进行信用分配，为样本高效的强化学习开辟了一条新路径。

英文摘要：

To make reinforcement learning more sample efficient, we need better credit assignment methods that measure an action's influence on future rewards. Building upon Hindsight Credit Assignment (HCA), we introduce Counterfactual Contribution Analysis (COCOA), a new family of model-based credit assignment algorithms. Our algorithms achieve precise credit assignment by measuring the contribution of actions upon obtaining subsequent rewards, by quantifying a counterfactual query: 'Would the agent still have reached this reward if it had taken another action?'. We show that measuring contributions w.r.t. rewarding states, as is done in HCA, results in spurious estimates of contributions, causing HCA to degrade towards the high-variance REINFORCE estimator in many relevant environments. Instead, we measure contributions w.r.t. rewards or learned representations of the rewarding objects, resulting in gradient estimates with lower variance. We run experiments on a suite of problems specifically designed to evaluate long-term credit assignment capabilities. By using dynamic programming, we measure ground-truth policy gradients and show that the improved performance of our new model-based credit assignment methods is due to lower bias and variance compared to HCA and common baselines. Our results demonstrate how modeling action contributions towards rewarding outcomes can be leveraged for credit assignment, opening a new path towards sample-efficient reinforcement learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。