arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.00746cs.LGcs.AIcs.RO
收藏

利用次优数据进行人在回路强化学习

Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning

University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute(阿尔伯塔机器智能研究所)

Calarina Muslimani, Matthew E. Taylor

AI总结:

本文提出次优数据预训练方法SDP,通过用环境最小奖励伪标记低质量数据来预训练奖励模型,从而减少人在回路强化学习对人类交互的需求,并在多种模拟机器人任务上显著提升性能。

摘要:

要创建有用的强化学习(RL)智能体,第一步是设计一个能够捕捉任务细微差别的合适奖励函数。然而,奖励工程可能是一个困难且耗时的过程。相反,人在回路强化学习方法有望从人类反馈中学习奖励函数。尽管近期取得了一些成功,许多人在回路强化学习方法仍然需要大量人类交互才能学习到成功的奖励函数。为了提高人在回路强化学习方法的反馈效率(即需要更少的人类交互),本文提出了次优数据预训练(Sub-optimal Data Pre-training,SDP),这是一种利用无奖励的次优数据来改进基于标量和基于偏好的强化学习算法的方法。在SDP中,我们首先用环境最小奖励对所有低质量数据进行伪标记。通过这一过程,我们获得了奖励标签,从而无需人类标注或偏好即可预训练我们的奖励模型。该预训练阶段为奖励模型提供了学习上的先发优势,使其能够认识到低质量转移应被赋予低奖励。通过在模拟教师和人类教师上的大量实验,我们发现SDP在多种模拟机器人任务上至少能达到、并且通常能显著提升最先进的人在回路强化学习性能。

英文摘要:

To create useful reinforcement learning (RL) agents, step zero is to design a suitable reward function that captures the nuances of the task. However, reward engineering can be a difficult and time-consuming process. Instead, human-in-the-loop RL methods hold the promise of learning reward functions from human feedback. Despite recent successes, many of the human-in-the-loop RL methods still require numerous human interactions to learn successful reward functions. To improve the feedback efficiency of human-in-the-loop RL methods (i.e., require less human interaction), this paper introduces Sub-optimal Data Pre-training, SDP, an approach that leverages reward-free, sub-optimal data to improve scalar- and preference-based RL algorithms. In SDP, we start by pseudo-labeling all low-quality data with the minimum environment reward. Through this process, we obtain reward labels to pre-train our reward model without requiring human labeling or preferences. This pre-training phase provides the reward model a head start in learning, enabling it to recognize that low-quality transitions should be assigned low rewards. Through extensive experiments with both simulated and human teachers, we find that SDP can at least meet, but often significantly improve, state of the art human-in-the-loop RL performance across a variety of simulated robotic tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑