arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.07110cs.RO
收藏

从失败中学习:利用片段记忆的高效强化学习控制

Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory

Xi’an Jiaotong University(西安交通大学)

Chenyang Miao

AI总结:

本文提出FEMA技术,通过存储失败经验防止机器人陷入不稳定状态,提升强化学习的样本效率和长期探索能力。

摘要:

强化学习在机器人学习中取得了显著成功。然而,在具有挑战性的探索和接触丰富的动态情况下,早期训练经常受到提前终止的影响,如碰撞和跌倒。因此,学习被短时间跨度、低回报轨迹所主导,这阻碍了收敛并限制了长时间跨度的探索。为了解决这个问题,我们提出了一种称为失败片段记忆警报(FEMA)的技术。FEMA通过一个片段记忆模块显式地存储短时间跨度的失败经验。在交互过程中,它检索相似的失败经验,防止机器人反复陷入不稳定状态,引导策略向具有更高长期价值的长时间跨度轨迹发展。FEMA可以轻松地与无模型强化学习算法结合,并在多个经典强化学习算法上实现了样本效率的显著提升,达到33.11%。此外,将FEMA整合到并行化的PPO训练流程中,在现实世界双足机器人任务中展示了其有效性。

英文摘要:

Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as collisions and falls. As a result, learning is overwhelmed by short-horizon, low-return trajectories, which hinder convergence and limit long-horizon exploration. To alleviate this issue, we propose a technique called Failure Episodic Memory Alert (FEMA). FEMA explicitly stores short-horizon failure experiences through an episodic memory module. During interactions, it retrieves similar failure experiences and prevents the robot from recurrently relapsing into unstable states, guiding the policy toward long-horizon trajectories with greater long-term value. FEMA can be combined easily with model-free reinforcement learning algorithms, and yields a substantial sample-efficiency improvement of 33.11% on MuJoCo tasks across several classical RL algorithms. Furthermore, integrating FEMA into a parallelized PPO training pipeline demonstrates its effectiveness on a real-world bipedal robot task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑