arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.22285cs.ROcs.AIcs.HCcs.LG
收藏

提出更智能的问题:针对欠规范奖励的主动偏好学习

Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards

Massachusetts Institute of Technology(麻省理工学院)

Helena Merker, Nick Walker, Andreea Bobu

AI总结:

该研究针对欠规范奖励的主动偏好学习,构建有信息先验并附解释引导人类,在低查询场景下性能优于基线,提升了奖励恢复效果。

摘要:

奖励学习通常假设人类的输入涵盖了任务的所有重要内容。但在实际应用中,人类一次只能关注有限的内容,因此他们的教学自然会对任务的某些特征或方面的捕捉优于其他方面。现有研究已表明机器人可直接从人类教学数据中识别出哪些特征监督不足,但当前的解决方案是要求人类重新尝试,将负担转嫁给了本就难以传达这些特征的人。我们建议将这一负担转移给机器人。我们的核心见解是,那些能告知机器人自身未知内容的信息,同样可帮助机器人提出更优质的问题。偏好查询是解决这种不确定性的低门槛方式,但前提是机器人知道该将查询指向何处,且人类在回答时知道该关注什么。我们从两方面落实这一见解:一是构建一个有信息先验,将偏好查询定向到需要解决的特征;二是为每个查询附上解释,引导人类的注意力精准聚焦于这些特征。在仿真实验中,该有信息先验在查询量少的场景下,比标准主动偏好学习基线的表现高出最多30.8%,而减少人类努力在此场景下最为关键。在用户研究中,该有信息先验和解释各自均显著提升了奖励恢复效果,且二者能相互增强。

英文摘要:

Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8\%$ in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑