arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.04712cs.LG
收藏

SPHERE: 缓解混合专家在深度强化学习中的频谱可塑性损失

SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning

State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(人工智能国家重点实验室,智能科学与技术学院,北京大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)

Lirui Luo, Guoxi Zhang, Hongming Xu, Cong Fang, Qing Li

AI总结:

本文提出SPHERE,通过引入Parseval惩罚来缓解混合专家策略在持续学习中的频谱可塑性损失,提升了MetaWorld和HumanoidBench的持续强化学习性能。

摘要:

在深度强化学习(DRL)中,智能体从经验流中进行训练。在持续学习设置中,此类智能体可能面临可塑性损失:其从新经验中学习新技能的能力会随训练过程减弱。最近,混合专家(MoE)网络被报告能够实现扩展定律并促进多样化技能的学习。然而,在持续强化学习设置中,其性能会随着学习过程的进行而退化,表明可塑性损失。为了解决这个问题,基于神经切线核(NTK)理论,我们将MoE策略中的可塑性损失正式化为频谱可塑性损失。然后,我们推导出一个可计算的频谱可塑性代理,其可表示为单个专家特征矩阵的函数。利用这个代理,我们引入了SPHERE,一种针对基于MoE的策略的实用Parseval惩罚,以缓解频谱可塑性损失。在MetaWorld和HumanoidBench上,SPHERE在持续RL中将平均成功率提高了133%和50%,同时在训练过程中保持较高的频谱可塑性。

英文摘要:

In deep reinforcement learning (DRL), an agent is trained from a stream of experience. In a continual learning setting, such agents can suffer from plasticity loss: their ability to learn new skills from new experiences diminishes over training. Recently, Mixture-of-Experts (MoE) networks have been reported to enable scaling laws and facilitate the learning of diverse skills. However, in continual reinforcement learning settings, their performance can degenerate as learning proceeds, indicating a loss of plasticity. To address this, building on Neural Tangent Kernel (NTK) theory, we formalize the plasticity loss in MoE policies as a loss of spectral plasticity. We then derive a tractable proxy for spectral plasticity, one expressible in terms of individual expert feature matrices. Leveraging this proxy, we introduce SPHERE, a practical Parseval penalty tailored for MoE-based policies that alleviates the loss of spectral plasticity. On MetaWorld and HumanoidBench, SPHERE improves average success under continual RL by 133% and 50% over an unregularized MoE baseline, while maintaining higher spectral plasticity throughout training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑