arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.01838cs.LGcs.AI
收藏

在函数空间中学习目标网络

Learning the Target Network in Function Space

Amazon(亚马逊);Technion(以色列理工学院);Princeton University(普林斯顿大学)

Kavosh Asadi, Yao Liu, Shoham Sabach, Ming Yin, Rasool Fakoor

AI总结:

提出Lookahead-Replicate算法,通过在函数空间而非参数空间维持等价性来更新目标网络,实现值函数学习的收敛,并在Atari基准上显著提升深度强化学习性能。

摘要:

我们关注强化学习(RL)设置中的值函数学习任务。该任务通常通过更新一对在线网络和目标网络,同时确保这两个网络的参数等价来解决。我们提出了Lookahead-Replicate(LR),这是一种与该参数空间等价性无关的新的值函数近似算法。相反,LR算法旨在维持两个网络在函数空间中的等价性。这种基于值的等价性是通过采用一种新的目标网络更新获得的。我们表明LR在值函数学习中会带来收敛行为。我们还给出了实验结果,证明基于LR的目标网络更新在Atari基准上显著改进了深度RL。

英文摘要:

We focus on the task of learning the value function in the reinforcement learning (RL) setting. This task is often solved by updating a pair of online and target networks while ensuring that the parameters of these two networks are equivalent. We propose Lookahead-Replicate (LR), a new value-function approximation algorithm that is agnostic to this parameter-space equivalence. Instead, the LR algorithm is designed to maintain an equivalence between the two networks in the function space. This value-based equivalence is obtained by employing a new target-network update. We show that LR leads to a convergent behavior in learning the value function. We also present empirical results demonstrating that LR-based target-network updates significantly improve deep RL on the Atari benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑