通过分治法简化奖励设计
Simplifying Reward Design through Divide-and-Conquer
摘要：
设计一个好的奖励函数对于机器人规划和强化学习至关重要，但它也可能具有挑战性和令人沮丧。奖励需要在多个不同的环境中工作，这通常需要多次迭代调优。我们引入了一种新颖的分治方法，使设计师能够为每个环境单独指定奖励。通过将这些单独的奖励函数视为对潜在真实奖励的观察，我们推导出一种在所有环境中推断共同奖励的方法。我们进行用户研究，在一个抽象的网格世界域和运动规划域的7自由度机械手，衡量用户的努力和解决方案的质量。我们表明，我们的方法更快，更容易使用，并产生更高质量的解决方案比典型的方法设计一个奖励联合在所有环境中。我们还进行了一系列实验，测量这些结果对奖励设计任务的不同属性的敏感性，例如环境的数量，每个环境的可行解决方案的数量，以及每个环境中变化的总特征的比例。我们发现，独立的奖励设计优于标准的，联合的，奖励设计过程中，但最好的工作时，设计问题可以分为更简单的子问题。
英文摘要：
Designing a good reward function is essential to robot planning and reinforcement learning, but it can also be challenging and frustrating. The reward needs to work across multiple different environments, and that often requires many iterations of tuning. We introduce a novel divide-and-conquer approach that enables the designer to specify a reward separately for each environment. By treating these separate reward functions as observations about the underlying true reward, we derive an approach to infer a common reward across all environments. We conduct user studies in an abstract grid world domain and in a motion planning domain for a 7-DOF manipulator that measure user effort and solution quality. We show that our method is faster, easier to use, and produces a higher quality solution than the typical method of designing a reward jointly across all environments. We additionally conduct a series of experiments that measure the sensitivity of these results to different properties of the reward design task, such as the number of environments, the number of feasible solutions per environment, and the fraction of the total features that vary within each environment. We find that independent reward design outperforms the standard, joint, reward design process but works best when the design problem can be divided into simpler subproblems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
