arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01423cs.LGcs.AI
收藏

价值改进的Actor-Critic算法

Value Improved Actor Critic Algorithms

Delft University of Technology(代尔夫特理工大学);University of Southern Denmark(南部丹麦大学);Department of Intelligent Systems(智能系统系)

Yaniv Oren, Moritz A. Zanger, Pascal R. van der Vaart, Mustafa Mert Celikok, Matthijs T. J. Spaan, Wendelin Bohmer

AI总结:

本文提出一种解耦策略与批评者评估策略的Actor-Critic算法,通过价值改进提升性能,适用于不同环境且计算成本低。

摘要:

为了学习近似最优的决策策略,现代Actor-Critic算法依赖深度神经网络(DNN)来参数化策略,并利用贪心运算符迭代改进策略。依赖DNN暗示了一种基于梯度的改进,这种改进每一步都比通过更贪心的运算符(如Q学习算法中的贪心更新)所能实现的改进要少。另一方面,对策略的缓慢变化也可能对学习过程的稳定性有益,从而在贪心性和稳定性之间产生权衡。为更好地解决这一权衡,我们提出将策略与由批评者评估的策略解耦。这使智能体能够分别用更贪心的更新改进批评者的策略(例如价值改进),同时保持对参数化策略的缓慢梯度改进。我们使用通用策略迭代的流行分析方案在有限时间域中研究了该方法的收敛性。实验表明,将价值改进纳入流行的off-policy actor-critic算法TD3和SAC中,在不同环境(来自DeepMind连续控制领域)中显著提高了或与各自基线相当的性能,且计算和实现成本极低。

英文摘要:

To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms. On the other hand, slow changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability. To better address this tradeoff, we propose to decouple the acting policy from the policy evaluated by the critic. This allows the agent to separately improve the critic's policy (e.g. value improvement) with greedier updates while maintaining the slow gradient-based improvement to the parameterized acting policy. We investigate the convergence of this approach using the popular analysis scheme of generalized Policy Iteration in the finite-horizon domain. Empirically, incorporating value-improvement into the popular off-policy actor-critic algorithms TD3 and SAC significantly improves or matches performance over their respective baselines, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑