策略正则化的离线多目标强化学习
Policy-regularized Offline Multi-objective Reinforcement Learning
AI总结：
本文将离线策略正则化方法扩展至多目标强化学习，提出过滤偏好不一致示范和高表达能力正则化两种方案，并集成偏好条件标量化更新与正则化权重自适应，利用单网络高效学习多偏好策略。
摘要：
在本文中，我们旨在仅利用离线轨迹数据来训练多目标强化学习的策略。我们将单目标离线强化学习问题中广泛采用的离线策略正则化方法扩展到多目标设置中，以实现上述目标。然而，此类方法在离线多目标强化学习设置中面临一个新挑战，即偏好不一致的示范问题。我们针对该问题提出了两种解决方案：1）通过近似行为偏好来过滤掉偏好不一致的示范；2）采用具有高策略表达能力的正则化技术。此外，我们将偏好条件的标量化更新方法集成到策略正则化的离线强化学习中，以便使用单个策略网络同时学习一组策略，从而减少为各种偏好训练大量独立策略所产生的计算成本。最后，我们引入了正则化权重自适应方法，以在部署期间为任意目标偏好动态确定合适的正则化权重。在各种多目标数据集上的实证结果证明了我们的方法在解决离线多目标强化学习问题上的能力。
英文摘要：
In this paper, we aim to utilize only offline trajectory data to train a policy for multi-objective RL. We extend the offline policy-regularized method, a widely-adopted approach for single-objective offline RL problems, into the multi-objective setting in order to achieve the above goal. However, such methods face a new challenge in offline MORL settings, namely the preference-inconsistent demonstration problem. We propose two solutions to this problem: 1) filtering out preference-inconsistent demonstrations via approximating behavior preferences, and 2) adopting regularization techniques with high policy expressiveness. Moreover, we integrate the preference-conditioned scalarized update method into policy-regularized offline RL, in order to simultaneously learn a set of policies using a single policy network, thus reducing the computational cost induced by the training of a large number of individual policies for various preferences. Finally, we introduce Regularization Weight Adaptation to dynamically determine appropriate regularization weights for arbitrary target preferences during deployment. Empirical results on various multi-objective datasets demonstrate the capability of our approach in solving offline MORL problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
