D-SPEAR：双流优先经验自适应回放用于稳定机器人操作的强化学习
D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulation
AI总结：
本文提出D-SPEAR双流优先经验自适应回放框架，通过分离actor和critic采样并维护共享回放缓冲区，利用优先回放提升价值学习效率，低误差过渡稳定策略优化，在机器人操作任务中优于SAC、TD3等基线方法。
摘要：
机器人操作仍是强化学习的挑战，由于接触丰富动态、长时间尺度和训练不稳定。尽管离线策略actor-critic算法如SAC和TD3在模拟中表现良好，但在现实设置中常出现策略振荡和性能崩溃，部分原因在于经验回放策略忽略了actor和critic不同的数据需求。我们提出D-SPEAR：双流优先经验自适应回放，一种回放框架，将actor和critic采样解耦同时维护共享回放缓冲区。critic利用优先回放实现高效的值学习，而actor使用低误差过渡稳定策略优化。自适应锚机制根据TD误差的变异系数平衡均匀和优先采样，Huber基于的critic目标进一步在异质奖励尺度下提高鲁棒性。我们在robosuite基准的具有挑战性的机器人操作任务上评估D-SPEAR，包括Block-Lifting和Door-Opening。结果表明，D-SPEAR在最终性能和训练稳定性上一致优于SAC、TD3和DDPG等强离线策略基线，消融研究确认了actor侧和critic侧回放流的互补作用。
英文摘要：
Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor-critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actorside and critic-side replay streams.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
