arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.04717cs.LGcs.AI
收藏

奖励推断对人类模型误设定的敏感性研究

On the Sensitivity of Reward Inference to Misspecified Human Models

UC Berkeley(加州大学伯克利分校);Stanford(斯坦福大学)

Joey Hong, Kush Bhatia, Anca Dragan

AI总结:

本文从理论和实证角度研究人类模型误设定对奖励推断的影响,证明小的对抗性偏差可导致任意大的推断误差,同时识别出使误差线性有界的合理假设,并在控制任务中验证了理论结果。

摘要:

从人类行为中推断奖励函数是价值对齐的核心——即将人工智能目标与人类真正想要的目标对齐。但这样做依赖于在给定目标下人类如何行为的模型。经过认知科学、神经科学和行为经济学数十年的研究,获得准确的人类模型仍然是一个开放的研究课题。这引出了一个问题:为了使奖励推断准确,这些模型需要多准确?一方面,如果模型中的小误差可能导致推断中的灾难性错误,那么奖励学习的整个框架似乎注定失败,因为我们永远不会有完美的人类行为模型。另一方面,如果随着我们的模型改进,我们能够保证奖励准确性也随之提高,这将表明在建模方面更多工作的益处。我们从理论和实证两方面研究这个问题。我们确实表明,不幸的是,有可能在行为中构造小的对抗性偏差,导致推断出的奖励出现任意大的误差。然而,也许更重要的是,我们还能够识别出合理的假设,在这些假设下,奖励推断误差可以线性地受人类模型误差的约束。最后,我们在离散和连续控制任务中用模拟数据和人类数据验证了我们的理论见解。

英文摘要:

Inferring reward functions from human behavior is at the center of value alignment - aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order for the reward inference to be accurate? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have a guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we are also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑