重新思考动态稀疏训练在可扩展深度强化学习中的作用
Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
AI总结：
本文重新思考动态稀疏训练在可扩展深度强化学习中的作用，发现其提供模块特定益处并补充架构改进，进而提出模块特定训练（MST）框架，在不修改算法的情况下显著提升多种RL算法的可扩展性。
摘要：
扩展神经网络规模推动了机器学习领域的突破性进展，然而这一范式在深度强化学习（DRL）中却遭遇失败，因为更大的模型往往由于可塑性丧失等独特的优化病理问题而导致性能下降。尽管近期研究表明，在训练过程中动态调整网络拓扑可以缓解这些问题，但现有研究存在三个关键局限性：（1）尽管编码器、评论家和演员遵循不同的学习范式，却对所有模块采用统一的动态训练策略；（2）评估集中于基础架构，未阐明动态训练与架构改进之间的相对重要性和相互作用；（3）缺乏对不同动态方法（包括稀疏到稀疏、稠密到稀疏和稀疏到稠密）的系统比较。通过对跨模块和跨架构的综合研究，我们揭示动态稀疏训练策略提供了模块特定的益处，这些益处补充了由架构改进所确立的主要可扩展性基础。最后，我们将这些见解提炼为模块特定训练（MST），这是一个实用框架，进一步利用架构改进的优势，并在无需算法修改的情况下，在多种强化学习算法中展示了显著的可扩展性提升。
英文摘要：
Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL), where larger models often degrade performance due to unique optimization pathologies such as plasticity loss. While recent works show that dynamically adapting network topology during training can mitigate these issues, existing studies have three critical limitations: (1) applying uniform dynamic training strategies across all modules despite encoder, critic, and actor following distinct learning paradigms, (2) focusing evaluation on basic architectures without clarifying the relative importance and interaction between dynamic training and architectural improvements, and (3) lacking systematic comparison between different dynamic approaches including sparse-to-sparse, dense-to-sparse, and sparse-to-dense. Through comprehensive investigation across modules and architectures, we reveal that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements. We finally distill these insights into Module-Specific Training (MST), a practical framework that further exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
