策略优化中的不变性与奖励学习中的部分可辨识性
Invariance in Policy Optimisation and Partial Identifiability in Reward Learning
AI总结：
本文研究奖励学习中奖励函数仅部分可辨识的问题，形式化刻画专家演示、轨迹比较等数据源下的不可辨识空间，并基于不变性框架分析其对策略优化等下游任务的影响。
摘要：
为复杂的现实世界任务手动设计奖励函数通常极具挑战性。为解决这一问题，可以转而使用奖励学习从数据中推断奖励函数。然而，即使在无限数据的极限情况下，也往往存在多个同样良好拟合数据的奖励函数。这意味着奖励函数仅具有部分可辨识性。在本工作中，我们针对若干常用的奖励学习数据源（包括专家演示和轨迹比较），形式化刻画了给定这些数据源时奖励函数的部分可辨识性。我们还分析了这种部分可辨识性对若干下游任务（例如策略优化）的影响。我们将各项结果统一到一个框架中，该框架依据数据源和下游任务的不变性对其进行比较，并对奖励学习中数据源的设计与选择具有启示意义。
英文摘要：
It is often very challenging to manually design reward functions for complex, real-world tasks. To solve this, one can instead use reward learning to infer a reward function from data. However, there are often multiple reward functions that fit the data equally well, even in the infinite-data limit. This means that the reward function is only partially identifiable. In this work, we formally characterise the partial identifiability of the reward function given several popular reward learning data sources, including expert demonstrations and trajectory comparisons. We also analyse the impact of this partial identifiability for several downstream tasks, such as policy optimisation. We unify our results in a framework for comparing data sources and downstream tasks by their invariances, with implications for the design and selection of data sources for reward learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
