arXiv 2306.061922025-03-13cs.ROcs.AIcs.LG

置信度控制的探索：面向机器人导航的高效稀疏奖励策略学习

Confidence-Controlled Exploration: Efficient Sparse-Reward Policy Learning for Robot Navigation

U.S. Army Research Laboratory（美国陆军研究实验室）；JP Morgan AI Research（摩根大通人工智能研究院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Central Florida（中佛罗里达大学）

Bhrij Patel, Kasun Weerakoon, Wesley A. Suttle, Alec Koppel, Brian M. Sadler, Tianyi Zhou, Amrit Singh Bedi, Dinesh Manocha

备注：: 10 pages, 6 figures, 2 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对机器人导航中稀疏奖励导致的探索低效问题，提出CCE依据策略熵动态调节轨迹长度，并在多种RL算法、仿真及真实户外场景中验证其样本效率与性能提升。

摘要：

强化学习（RL）是一种很有前景的机器人导航方法，使机器人能够通过试错进行学习。然而，现实世界的机器人任务往往面临稀疏奖励问题，由于RL的样本效率低下，会导致探索效率不高和策略次优。在本工作中，我们提出了置信度控制探索（Confidence-Controlled Exploration，CCE），这是一种在不修改奖励函数的情况下提高基于RL的机器人导航样本效率的新方法。与熵正则化和奖励塑造等通过改变奖励而可能引入不稳定性的现有方法不同，CCE根据策略熵动态调整轨迹长度。具体而言，它在不确定性较高时缩短轨迹以增强探索，在置信度较高时延长轨迹以优先利用。CCE是一种原则性且实用的解决方案，其灵感来自策略熵与梯度估计之间的理论联系。它可以与同策略和异策略RL方法无缝集成，并且只需极少修改。我们在模拟和真实世界导航任务中，跨REINFORCE、PPO和SAC验证了CCE。在固定训练样本预算下，CCE优于固定轨迹和熵正则化基线，实现了高18%的成功率、短20–38%的路径以及低9.32%的高程成本。最后，我们将CCE部署在Clearpath Husky机器人上，展示了其在复杂户外环境中的有效性。

英文摘要：

Reinforcement learning (RL) is a promising approach for robotic navigation, allowing robots to learn through trial and error. However, real-world robotic tasks often suffer from sparse rewards, leading to inefficient exploration and suboptimal policies due to sample inefficiency of RL. In this work, we introduce Confidence-Controlled Exploration (CCE), a novel method that improves sample efficiency in RL-based robotic navigation without modifying the reward function. Unlike existing approaches, such as entropy regularization and reward shaping, which can introduce instability by altering rewards, CCE dynamically adjusts trajectory length based on policy entropy. Specifically, it shortens trajectories when uncertainty is high to enhance exploration and extends them when confidence is high to prioritize exploitation. CCE is a principled and practical solution inspired by a theoretical connection between policy entropy and gradient estimation. It integrates seamlessly with on-policy and off-policy RL methods and requires minimal modifications. We validate CCE across REINFORCE, PPO, and SAC in both simulated and real-world navigation tasks. CCE outperforms fixed-trajectory and entropy-regularized baselines, achieving an 18\% higher success rate, 20-38\% shorter paths, and 9.32\% lower elevation costs under a fixed training sample budget. Finally, we deploy CCE on a Clearpath Husky robot, demonstrating its effectiveness in complex outdoor environments.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。