利用最优传输增强手术机器人环境中的离线强化学习
Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environments
AI总结：
本文提出最优传输奖励（OTR）标注算法，利用少量专家演示为离线轨迹分配奖励，无需人工设计，并在手术机器人仿真平台SurRoL上验证其有效性，以加速离线强化学习部署。
摘要：
大多数强化学习（RL）方法传统上是在主动学习环境中研究的，其中智能体直接与环境交互，观察行动结果，并通过试错进行学习。然而，允许部分训练的智能体与真实物理系统交互会带来重大挑战，包括高成本、安全风险以及需要持续监督。离线强化学习通过利用现有数据集并减少对资源密集型实时交互的需求，解决了这些成本和安全问题。然而，一个重大挑战在于这些数据集需要被细致地标注奖励。在本文中，我们引入了最优传输奖励（OTR）标注算法，这是一种创新算法，旨在使用少量高质量专家演示为离线轨迹分配奖励。OTR的核心原理是利用最优传输（OT）计算数据集中未标注轨迹与专家演示之间的最优对齐。这种对齐产生一个相似性度量，可有效地解释为奖励信号。离线强化学习算法随后可以利用这些奖励信号来学习策略。这种方法规避了对人工设计奖励的需求，释放了利用大规模数据集进行策略学习的潜力。利用专为手术机器人学习设计的SurRoL仿真平台，我们生成数据集并使用OTR算法训练策略。通过在不同领域展示OTR的有效性，我们强调了其多功能性及其在广泛领域加速强化学习部署的潜力。
英文摘要：
Most Reinforcement Learning (RL) methods are traditionally studied in an active learning setting, where agents directly interact with their environments, observe action outcomes, and learn through trial and error. However, allowing partially trained agents to interact with real physical systems poses significant challenges, including high costs, safety risks, and the need for constant supervision. Offline RL addresses these cost and safety concerns by leveraging existing datasets and reducing the need for resource-intensive real-time interactions. Nevertheless, a substantial challenge lies in the demand for these datasets to be meticulously annotated with rewards. In this paper, we introduce Optimal Transport Reward (OTR) labelling, an innovative algorithm designed to assign rewards to offline trajectories, using a small number of high-quality expert demonstrations. The core principle of OTR involves employing Optimal Transport (OT) to calculate an optimal alignment between an unlabeled trajectory from the dataset and an expert demonstration. This alignment yields a similarity measure that is effectively interpreted as a reward signal. An offline RL algorithm can then utilize these reward signals to learn a policy. This approach circumvents the need for handcrafted rewards, unlocking the potential to harness vast datasets for policy learning. Leveraging the SurRoL simulation platform tailored for surgical robot learning, we generate datasets and employ them to train policies using the OTR algorithm. By demonstrating the efficacy of OTR in a different domain, we emphasize its versatility and its potential to expedite RL deployment across a wide range of fields.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
