基于推理的深度强化学习中理论创新与实现创新的协同适应
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
摘要：
近年来，人们提出了许多基于函数逼近的强化学习算法。虽然它们有明确的算法区别，但它们也有许多独立于算法的实现差异，有时会被忽视。算法新颖性和实现技巧的这种混合使得严格分析算法性能改进的来源变得困难。在这项工作中，我们专注于一系列基于非策略推理的演员-评论家算法- MPO，AWR和SAC -解耦算法创新和实施决策。我们通过一个单一的控制推理目标，我们可以将每个算法分类为基于期望最大化（EM）或直接Kullback-Leibler（KL）发散最小化，并将其余的规范作为实现细节。我们进行了广泛的消融研究，并确定了实质性的性能下降时，实现细节不匹配的算法选择。这些结果显示了哪些实现或代码细节与算法共同适应和共同进化，以及哪些可以在算法之间转移：例如，我们发现双曲正切高斯策略和网络大小高度适应算法类型，而层归一化和ELU对MPO的性能至关重要，但也可以在SAC中转移到显著的增益。我们希望我们的工作可以激发未来的工作，进一步揭开多种算法性能改进的来源，并允许研究人员建立在彼此的算法和实现创新的基础上。
英文摘要：
Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are algorithm-independent and sometimes under-emphasized. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources of performance improvements across algorithms difficult. In this work, we focus on a series of off-policy inference-based actor-critic algorithms -- MPO, AWR, and SAC -- to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single control-as-inference objective, where we can categorize each algorithm as based on either Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications as implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation or code details are co-adapted and co-evolved with algorithms, and which are transferable across algorithms: as examples, we identified that tanh Gaussian policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO's performances but also transfer to noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another's both algorithmic and implementational innovations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
