arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.04841stat.MLcond-mat.dis-nncs.AIcs.LG
收藏

时序差分强化学习的损失动力学

Loss Dynamics of Temporal Difference Reinforcement Learning

John Paulson School of Engineering and Applied Sciences(约翰·保尔森工程与应用科学学院);Center for Brain Science(脑科学中心);Kempner Institute for the Study of Natural & Artificial Intelligence(肯普纳自然与人工智能研究所);Harvard University(哈佛大学)

Blake Bordelon, Paul Masset, Henry Kuo, Cengiz Pehlevan

AI总结:

该研究运用统计物理概念研究线性函数逼近器下时序差分强化学习的损失动力学,揭示半梯度噪声导致值误差平台期,分析相关影响因素与优化策略,为强化学习动力学理论提供新工具与新方向。

摘要:

强化学习已在多项智能体需在稀疏反馈环境中学习行动的应用中取得成功。然而,尽管有这些实证成果,人们仍缺乏对强化学习模型参数与用于表示状态的特征如何相互作用以调控学习动力学的理论理解。在本研究中,我们运用统计物理学的概念,研究采用线性函数逼近器的值函数时序差分学习的典型案例学习曲线。我们的理论基于高斯等价假设推导得出,该假设将随机轨迹上的平均替换为时间相关的高斯特征平均,并且我们在小规模马尔可夫决策过程(Markov Decision Processes)上验证了我们的假设。我们发现,与传统梯度下降动力学不同,由可能情节空间的子采样导致的随机半梯度噪声会在值误差中产生显著的平台期。我们研究了学习动力学和平台期如何依赖于特征结构、学习率、折扣因子和奖励函数。随后我们分析了学习率退火和奖励塑形等策略如何能有利地改变学习动力学和平台期。总而言之,我们的工作引入了新工具,为发展强化学习中的学习动力学理论开辟了新方向。

英文摘要:

Reinforcement learning has been successful across several applications in which agents have to learn to act in environments with sparse feedback. However, despite this empirical success there is still a lack of theoretical understanding of how the parameters of reinforcement learning models and the features used to represent states interact to control the dynamics of learning. In this work, we use concepts from statistical physics, to study the typical case learning curves for temporal difference learning of a value function with linear function approximators. Our theory is derived under a Gaussian equivalence hypothesis where averages over the random trajectories are replaced with temporally correlated Gaussian feature averages and we validate our assumptions on small scale Markov Decision Processes. We find that the stochastic semi-gradient noise due to subsampling the space of possible episodes leads to significant plateaus in the value error, unlike in traditional gradient descent dynamics. We study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function. We then analyze how strategies like learning rate annealing and reward shaping can favorably alter learning dynamics and plateaus. To conclude, our work introduces new tools to open a new direction towards developing a theory of learning dynamics in reinforcement learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑