基于不确定性的平滑策略正则化用于少样本示范的强化学习
Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
AI总结：
提出SPReD框架，通过集成方法量化不确定性，在少示范强化学习中实现连续且与不确定性成正比的模仿正则化，显著降低梯度方差并在复杂机器人任务中取得最高14倍的性能提升。
摘要：
在稀疏奖励的强化学习中，示范可以加速学习，但确定何时模仿示范仍然具有挑战性。我们提出了基于示范的平滑策略正则化（SPReD），该框架解决了一个基本问题：智能体何时应该模仿示范，而不是遵循自己的策略？SPReD使用集成方法显式建模示范动作和策略动作的Q值分布，从而量化不确定性以进行比较。我们开发了两种互补的感知不确定性的方法：一种估计示范优越性似然的概率方法，以及一种通过统计显著性来缩放模仿的基于优势的方法。与做出二元模仿决策的现有方法（例如Q-filter）不同，SPReD应用连续的、与不确定性成正比的正则化权重，从而减少了训练期间的梯度方差。尽管计算简单，SPReD在八个机器人任务的实验中取得了显著收益，在复杂任务中性能比现有方法高出14倍，同时对示范的质量和数量保持鲁棒性。
英文摘要：
In reinforcement learning with sparse rewards, demonstrations can accelerate learning, but determining when to imitate them remains challenging. We propose Smooth Policy Regularisation from Demonstrations (SPReD), a framework that addresses the fundamental question: when should an agent imitate a demonstration versus follow its own policy? SPReD uses ensemble methods to explicitly model Q-value distributions for both demonstration and policy actions, quantifying uncertainty for comparisons. We develop two complementary uncertainty-aware methods: a probabilistic approach estimating the likelihood of demonstration superiority, and an advantage-based approach scaling imitation by statistical significance. Unlike prevailing methods (e.g. Q-filter) that make binary imitation decisions, SPReD applies continuous, uncertainty-proportional regularisation weights, reducing gradient variance during training. Despite its computational simplicity, SPReD achieves remarkable gains in experiments across eight robotics tasks, outperforming existing approaches by up to a factor of 14 in complex tasks while maintaining robustness to demonstration quality and quantity. Our code is available at https://github.com/YujieZhu7/SPReD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
