arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.12754cs.LGcs.AI
收藏

表示秩的自适应正则化作为贝尔曼方程的隐式约束

Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equation

Ruhr University Bochum(波鸿鲁尔大学);University of Maryland, College Park(马里兰大学帕克分校);University of Liverpool(利物浦大学)

Qiang He, Tianyi Zhou, Meng Fang, Setareh Maghsudi

AI总结:

本研究提出基于贝尔曼方程的自适应秩正则化器BEER,通过约束表示秩提升深度强化学习性能,在12个控制任务中大幅超越基线。

摘要:

表示秩是理解神经网络在深度强化学习中的作用的一个重要概念,它衡量了价值网络的表达能力。现有研究侧重于无界地最大化该秩;然而,这种方法会在学习过程中引入过于复杂的模型,从而损害性能。因此,微调表示秩是一个具有挑战性且关键的最优化问题。为了解决这个问题,我们找到了自适应控制表示秩的指导原则。我们以贝尔曼方程为理论基础,推导出价值网络连续状态-动作对表示的余弦相似度的上界。然后,我们利用这个上界提出了一种新颖的正则化器,即基于贝尔曼方程的自动秩正则化器(BEER)。该正则化器自适应地正则化表示秩,从而提升深度强化学习智能体的性能。我们首先在说明性实验中验证了秩自动控制的有效性。然后,通过将BEER与确定性策略梯度方法相结合,将其扩展到复杂的连续控制任务。在12个具有挑战性的DeepMind控制任务中,BEER大幅优于基线方法。此外,BEER在Q值近似方面展现出显著优势。我们的代码可在https://github.com/sweetice/BEER-ICLR2024获取。

英文摘要:

Representation rank is an important concept for understanding the role of Neural Networks (NNs) in Deep Reinforcement learning (DRL), which measures the expressive capacity of value networks. Existing studies focus on unboundedly maximizing this rank; nevertheless, that approach would introduce overly complex models in the learning, thus undermining performance. Hence, fine-tuning representation rank presents a challenging and crucial optimization problem. To address this issue, we find a guiding principle for adaptive control of the representation rank. We employ the Bellman equation as a theoretical foundation and derive an upper bound on the cosine similarity of consecutive state-action pairs representations of value networks. We then leverage this upper bound to propose a novel regularizer, namely BEllman Equation-based automatic rank Regularizer (BEER). This regularizer adaptively regularizes the representation rank, thus improving the DRL agent's performance. We first validate the effectiveness of automatic control of rank on illustrative experiments. Then, we scale up BEER to complex continuous control tasks by combining it with the deterministic policy gradient method. Among 12 challenging DeepMind control tasks, BEER outperforms the baselines by a large margin. Besides, BEER demonstrates significant advantages in Q-value approximation. Our code is available at https://github.com/sweetice/BEER-ICLR2024.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑