Next-Future：面向机械臂任务的样本高效策略学习
Next-Future: Sample-Efficient Policy Learning for Robotic-Arm Tasks
AI总结：
针对HER缺乏原则性框架的局限，提出专注单步转移奖励的Next-Future回放策略，在8个机械臂任务中提升7个任务的样本效率与6个任务的成功率，且经真实世界实验验证可行性。
摘要：
事后经验回放（HER）被广泛认为是在具有二元奖励的机器人操作任务中实现样本高效多目标强化学习（RL）的最先进算法。HER通过重放重新定义目标的轨迹，促进从失败尝试中学习。然而，它依赖于基于启发式的回放方法，缺乏原则性框架。为解决这一局限，我们引入一种新颖的回放策略“Next-Future”，其专注于奖励单步转移。该方法显著提升了学习多目标马尔可夫决策过程（MDPs）的样本效率和准确性，尤其在严格的精度要求下——这是执行复杂精密机械臂任务的关键方面。我们通过强调单步学习如何在多目标RL框架内改进价值近似，证明了方法的有效性。所提回放策略的性能在8个具有挑战性的机器人操作任务中评估，训练使用10个随机种子。结果表明，8个任务中有7个的样本效率大幅提升，6个任务的成功率更高。此外，真实世界实验验证了所学策略的实际可行性，展示了“Next-Future”在解决复杂机械臂任务中的潜力。
英文摘要：
Hindsight Experience Replay (HER) is widely regarded as the state-of-the-art algorithm for achieving sample-efficient multi-goal reinforcement learning (RL) in robotic manipulation tasks with binary rewards. HER facilitates learning from failed attempts by replaying trajectories with redefined goals. However, it relies on a heuristic-based replay method that lacks a principled framework. To address this limitation, we introduce a novel replay strategy, "Next-Future", which focuses on rewarding single-step transitions. This approach significantly enhances sample efficiency and accuracy in learning multi-goal Markov decision processes (MDPs), particularly under stringent accuracy requirements -- a critical aspect for performing complex and precise robotic-arm tasks. We demonstrate the efficacy of our method by highlighting how single-step learning enables improved value approximation within the multi-goal RL framework. The performance of the proposed replay strategy is evaluated across eight challenging robotic manipulation tasks, using ten random seeds for training. Our results indicate substantial improvements in sample efficiency for seven out of eight tasks and higher success rates in six tasks. Furthermore, real-world experiments validate the practical feasibility of the learned policies, demonstrating the potential of "Next-Future" in solving complex robotic-arm tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
