arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.00894cs.LGcs.AI
收藏

多位教师的积极奖励学习

Active Reward Learning from Multiple Teachers

University of California, Berkeley(加州大学伯克利分校);Center for Human-Compatible AI(人类兼容人工智能中心)

Peter Barnett, Rachel Freedman, Justin Svegliato, Stuart Russell

摘要:

奖励学习算法利用人的反馈来推断奖励函数,然后将其用于训练AI系统。这种人类反馈通常是一种偏好比较,其中人类教师比较了几个人工智能行为样本,并选择他们认为最能实现目标的样本。虽然奖励学习通常假设所有反馈都来自单个教师,但在实践中,这些系统经常询问多个教师以收集足够的训练数据。在本文中,我们调查了这种差异,并发现这些不同来源的反馈的算法评估有利于更准确和有效的奖励学习。我们正式分析的信息(VOI)的价值时,奖励从教师学习不同程度的合理性,并定义和评估的算法,利用这个VOI积极选择教师查询反馈。令人惊讶的是,我们发现,它往往是更翔实的询问相对非理性的教师。通过将这个问题形式化并推导出分析解决方案,我们希望促进奖励学习方法的改进,使AI行为与人类价值观保持一致。

英文摘要:

Reward learning algorithms utilize human feedback to infer a reward function, which is then used to train an AI system. This human feedback is often a preference comparison, in which the human teacher compares several samples of AI behavior and chooses which they believe best accomplishes the objective. While reward learning typically assumes that all feedback comes from a single teacher, in practice these systems often query multiple teachers to gather sufficient training data. In this paper, we investigate this disparity, and find that algorithmic evaluation of these different sources of feedback facilitates more accurate and efficient reward learning. We formally analyze the value of information (VOI) when reward learning from teachers with varying levels of rationality, and define and evaluate an algorithm that utilizes this VOI to actively select teachers to query for feedback. Surprisingly, we find that it is often more informative to query comparatively irrational teachers. By formalizing this problem and deriving an analytical solution, we hope to facilitate improvement in reward learning approaches to aligning AI behavior with human values.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑