标量化多目标强化学习中通过内插奖励函数预测最优价值函数
Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning
摘要:
定义多目标强化学习(MORL)问题的奖励函数的常见方法是多个目标的加权和。然后,权重被视为设计参数,取决于执行学习的人的专业知识(和偏好),典型的结果是,这些设置中的任何变化都需要新的解决方案。本文研究了MORL的奖励函数和最优价值函数之间的关系;特别是解决了如何近似最优价值函数的问题,远远超出了一组权重的优化问题,从而避免了需要重新计算任何特定的选择。我们证明了,价值函数变换平滑的奖励函数的权重的变换(从而在政策空间中的平滑插值)。一个高斯过程是用来获得一个光滑的插值超过三个著名的例子:GridWorld,Objectworld和Pendulum的最佳价值函数的奖励函数权重。结果表明,该插值方法在离散域和连续域问题中均能为样本状态和动作空间提供非常稳健的值。在自动驾驶汽车领域使用这种插值技术具有显著的优势:在驾驶时轻松、即时地调整用户偏好,以及在训练期间真正随机化障碍车辆行为偏好。
英文摘要:
A common approach for defining a reward function for Multi-objective Reinforcement Learning (MORL) problems is the weighted sum of the multiple objectives. The weights are then treated as design parameters dependent on the expertise (and preference) of the person performing the learning, with the typical result that a new solution is required for any change in these settings. This paper investigates the relationship between the reward function and the optimal value function for MORL; specifically addressing the question of how to approximate the optimal value function well beyond the set of weights for which the optimization problem was actually solved, thereby avoiding the need to recompute for any particular choice. We prove that the value function transforms smoothly given a transformation of weights of the reward function (and thus a smooth interpolation in the policy space). A Gaussian process is used to obtain a smooth interpolation over the reward function weights of the optimal value function for three well-known examples: GridWorld, Objectworld and Pendulum. The results show that the interpolation can provide very robust values for sample states and action space in discrete and continuous domain problems. Significant advantages arise from utilizing this interpolation technique in the domain of autonomous vehicles: easy, instant adaptation of user preferences while driving and true randomization of obstacle vehicle behavior preferences during training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。