HALO：面向机器人导航的基于人类偏好对齐的离线奖励学习
HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation
AI总结：
HALO是一种离线奖励学习算法，将人类导航直觉量化为视觉奖励函数，通过偏好排序训练奖励模型，并在真实机器人上显著提升导航成功率与轨迹质量。
摘要：
在本文中，我们提出了HALO，一种新颖的离线奖励学习算法，该算法将人类在导航中的直觉量化为基于视觉的奖励函数，用于机器人导航。HALO利用从移动机器人收集的专家轨迹，从离线数据中学习奖励模型。在训练过程中，动作围绕参考动作均匀采样，并使用基于以偏好动作中心的玻尔兹曼分布得出的偏好分数进行排序，并根据对直观导航查询的二元用户反馈进行塑形。奖励模型通过Plackett-Luce损失进行训练，以与这些排序的偏好对齐。为了展示HALO的有效性，我们在两个下游应用中部署了其奖励模型：（i）直接在HALO派生的奖励上训练的离线学习策略，以及（ii）基于模型预测控制（MPC）的规划器，该规划器将HALO奖励作为额外的成本项纳入。这展示了HALO在基于学习和经典导航框架中的多功能性。我们在Clearpath Husky上跨多种场景的真实世界部署表明，使用HALO训练的策略能够有效泛化到训练数据中未出现的未见环境和硬件设置。HALO优于最先进的基于视觉的导航方法，在成功率上至少提高了33.3%，归一化轨迹长度减少了12.9%，与人类专家轨迹相比，弗雷歇距离减少了26.6%。
英文摘要：
In this paper, we introduce HALO, a novel Offline Reward Learning algorithm that quantifies human intuition in navigation into a vision-based reward function for robot navigation. HALO learns a reward model from offline data, leveraging expert trajectories collected from mobile robots. During training, actions are uniformly sampled around a reference action and ranked using preference scores derived from a Boltzmann distribution centered on the preferred action, and shaped based on binary user feedback to intuitive navigation queries. The reward model is trained via the Plackett-Luce loss to align with these ranked preferences. To demonstrate the effectiveness of HALO, we deploy its reward model in two downstream applications: (i) an offline learned policy trained directly on the HALO-derived rewards, and (ii) a model-predictive-control (MPC) based planner that incorporates the HALO reward as an additional cost term. This showcases the versatility of HALO across both learning-based and classical navigation frameworks. Our real-world deployments on a Clearpath Husky across diverse scenarios demonstrate that policies trained with HALO generalize effectively to unseen environments and hardware setups not present in the training data. HALO outperforms state-of-the-art vision-based navigation methods, achieving at least a 33.3% improvement in success rate, a 12.9% reduction in normalized trajectory length, and a 26.6% reduction in Frechet distance compared to human expert trajectories.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
