演员-评论家算法中的高估、过拟合与可塑性：强化学习的惨痛教训
Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
AI总结：
该研究通过实现60余种整合不同正则化技术的离策略强化学习智能体，在2个模拟基准的14项任务上测试，发现经适当正则化的简单Soft Actor-Critic智能体性能稳健优异，揭示了正则化对强化学习的作用机制。
摘要：
离策略强化学习（RL）近期的进展显著提升了样本效率，这主要得益于引入了多种形式的正则化，使得智能体能够进行比传统智能体更多的梯度更新步骤。然而，这些技术中有许多仅在有限的场景中接受了测试，通常是在单一模拟基准的任务上，且仅与知名算法进行对比，而非与一系列正则化方法比较。这限制了我们对推动强化学习性能提升的具体机制的理解。为解决这一问题，我们实现了60余种不同的离策略智能体，每一种都整合了近期最先进算法中已有的正则化技术。我们在来自2个模拟基准的14项不同任务上测试了这些智能体，测量了与高估、过拟合和可塑性损失相关的训练指标——这些问题正是所研究的正则化技术要解决的。我们的研究结果表明，尽管特定正则化设置的有效性随任务而变化，但某些组合始终表现出稳健且优越的性能。值得注意的是，一个经过适当正则化的简单Soft Actor-Critic智能体，能够在训练过程中可靠地找到性能更优的策略，而这在以前主要通过基于模型的方法才能实现。
英文摘要：
Recent advancements in off-policy Reinforcement Learning (RL) have significantly improved sample efficiency, primarily due to the incorporation of various forms of regularization that enable more gradient update steps than traditional agents. However, many of these techniques have been tested in limited settings, often on tasks from single simulation benchmarks and against well-known algorithms rather than a range of regularization approaches. This limits our understanding of the specific mechanisms driving RL improvements. To address this, we implemented over 60 different off-policy agents, each integrating established regularization techniques from recent state-of-the-art algorithms. We tested these agents across 14 diverse tasks from 2 simulation benchmarks, measuring training metrics related to overestimation, overfitting, and plasticity loss -- issues that motivate the examined regularization techniques. Our findings reveal that while the effectiveness of a specific regularization setup varies with the task, certain combinations consistently demonstrate robust and superior performance. Notably, a simple Soft Actor-Critic agent, appropriately regularized, reliably finds a better-performing policy within the training regime, which previously was achieved mainly through model-based approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
