arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.14976cs.LGcs.AI
收藏

线性回归与拟牛顿方法在连续动作情形下的深度强化学习中的应用

Application of linear regression and quasi-Newton methods to the deep reinforcement learning in continuous action cases

Data Science and AI Innovation Research Promotion Center(数据科学与人工智能创新研究促进中心);Shiga University(Shiga大学)

Hisato Komatsu

AI总结:

本文提出DLS-DDPG方法,结合线性回归与拟牛顿方法改进连续动作的深度强化学习,在MuJoCo环境中部分任务性能提升。

摘要:

线性回归(LR)方法具有相对容易计算最优参数的优点,但其表示能力弱于深度学习技术。为了改进深度强化学习,Levine等人提出了最小二乘深度Q网络(LS-DQN)方法,该方法将深度Q网络(DQN)与LR方法相结合。然而,LS-DQN方法假设动作是离散的。在本研究中,我们提出双最小二乘深度确定性策略梯度(DLS-DDPG)方法以解决这一限制。该方法将LR方法与深度确定性策略梯度(DDPG)技术相结合,DDPG是连续动作情形下具有代表性的深度强化学习算法之一。对于评论者网络的LR更新,DLS-DDPG采用了类似于LS-DQN所采用的拟合Q迭代的算法。此外,我们使用拟牛顿方法计算最优动作,并将其同时用作智能体的动作和行动者网络LR更新的训练数据。在MuJoCo环境中进行的数值实验表明,尽管存在诸如无法使正则化项变小等困难,但所提出的方法至少在部分任务中提升了性能。

英文摘要:

The linear regression (LR) method offers the advantage that optimal parameters can be calculated relatively easily, although its representation capability is limited than that of the deep learning technique. To improve deep reinforcement learning, the Least Squares Deep Q Network (LS-DQN) method was proposed by Levine et al., which combines Deep Q Network (DQN) with LR method. However, the LS-DQN method assumes that the actions are discrete. In this study, we propose the Double Least Squares Deep Deterministic Policy Gradient (DLS-DDPG) method to address this limitation. This method combines the LR method with the Deep Deterministic Policy Gradient (DDPG) technique, one of the representative deep reinforcement learning algorithms for continuous action cases. For the LR update of the critic network, DLS-DDPG uses an algorithm similar to the Fitted Q iteration, the method which LS-DQN adopted. In addition, we calculated the optimal action using the quasi-Newton method and used it as both the agent's action and the training data for the LR update of the actor network. Numerical experiments conducted in MuJoCo environments showed that the proposed method improved performance at least in some tasks, although there are difficulties such as the inability to make the regularization terms small.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑