图上的转导奖励推断
Transductive Reward Inference on Graph
AI总结：
本研究提出一种基于图信息传播的转导奖励推断方法，利用有限人类标注推断未标记数据奖励，并证明收敛性，在运动控制和机器人操作任务中提升离线强化学习性能。
摘要：
在本研究中，我们提出了一种在奖励信息传播图上的转导推断方法，该方法能够在离线强化学习中有效估计未标记数据的奖励。奖励推断是在实际场景中学习有效策略的关键，而直接与环境交互要么代价过高，要么不符合伦理，且奖励函数很少可获取，例如在医疗和机器人领域。我们的研究专注于开发一种基于图上信息传播的上下文属性的奖励推断方法，该方法利用有限数量的人类奖励标注来推断未标记数据的奖励。我们利用可用数据和有限的奖励标注来构建一个奖励传播图，其中边的权重包含了与奖励相关的多种影响因素。随后，我们使用构建的图进行转导奖励推断，从而估计未标记数据的奖励。此外，我们证明了在转导推断过程的多次迭代中存在一个不动点，并展示了其至少收敛到局部最优。在运动控制和机器人操作任务上的实证评估验证了我们方法的有效性。应用我们推断出的奖励提高了离线强化学习任务的性能。
英文摘要：
In this study, we present a transductive inference approach on that reward information propagation graph, which enables the effective estimation of rewards for unlabelled data in offline reinforcement learning. Reward inference is the key to learning effective policies in practical scenarios, while direct environmental interactions are either too costly or unethical and the reward functions are rarely accessible, such as in healthcare and robotics. Our research focuses on developing a reward inference method based on the contextual properties of information propagation on graphs that capitalizes on a constrained number of human reward annotations to infer rewards for unlabelled data. We leverage both the available data and limited reward annotations to construct a reward propagation graph, wherein the edge weights incorporate various influential factors pertaining to the rewards. Subsequently, we employ the constructed graph for transductive reward inference, thereby estimating rewards for unlabelled data. Furthermore, we establish the existence of a fixed point during several iterations of the transductive inference process and demonstrate its at least convergence to a local optimum. Empirical evaluations on locomotion and robotic manipulation tasks validate the effectiveness of our approach. The application of our inferred rewards improves the performance in offline reinforcement learning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
