arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.13490cs.ROcs.LG
收藏

基于新颖性的样本重用用于连续机器人控制

Novelty-based Sample Reuse for Continuous Robotics Control

Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)

Ke Duan, Kai Yang, Houde Liu, Xueqian Wang

AI总结:

针对强化学习中样本利用不均导致训练耗时的问题,提出Novelty-guided Sample Reuse (NSR)方法,为罕见状态提供额外更新并跳过频繁状态更新,从而提高算法收敛率和成功率且不显著增加时间消耗。

摘要:

在强化学习中,智能体通过环境交互收集状态信息和奖励,这对于策略改进至关重要。这一过程非常耗时,特别是在复杂的机器人仿真和真实世界应用中。传统算法通常在处理完单个批次的样本后便重新与环境交互,从而未能充分利用历史数据。然而,频繁观察到的状态具有可靠的价值估计,只需极少的更新;相比之下,罕见观察到的状态则需要更密集的更新以实现准确的价值估计。为了解决样本利用不均的问题,我们提出了Novelty-guided Sample Reuse (NSR)。NSR为不频繁出现的新颖状态提供额外更新,并跳过对频繁状态的额外更新,从而在与环境再次交互之前最大化样本使用。我们的实验表明,NSR提高了算法的收敛率和成功率,且未显著增加时间消耗。我们的代码已在 https://github.com/ppksigs/NSR-DDPG-HER 公开。

英文摘要:

In reinforcement learning, agents collect state information and rewards through environmental interactions, essential for policy refinement. This process is notably time-consuming, especially in complex robotic simulations and real-world applications. Traditional algorithms usually re-engage with the environment after processing a single batch of samples, thereby failing to fully capitalize on historical data. However, frequently observed states, with reliable value estimates, require minimal updates; in contrast, rare observed states necessitate more intensive updates for achieving accurate value estimations. To address uneven sample utilization, we propose Novelty-guided Sample Reuse (NSR). NSR provides extra updates for infrequent, novel states and skips additional updates for frequent states, maximizing sample use before interacting with the environment again. Our experiments show that NSR improves the convergence rate and success rate of algorithms without significantly increasing time consumption. Our code is publicly available at https://github.com/ppksigs/NSR-DDPG-HER.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑