遍历帕累托最优策略：可证明高效的多目标强化学习
Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning
AI总结：
本文研究多目标强化学习，提出基于Tchebycheff标量化的min-max-max重构算法，设计了单偏好UCB算法和无偏好探索框架，并扩展至平滑Tchebycheff标量化，实现了可证明高效的帕累托最优策略学习。
摘要：
本文研究了多目标强化学习（MORL），其重点是在存在多个奖励函数的情况下学习帕累托最优策略。尽管MORL在经验上取得了显著成功，但对各种MORL优化目标及高效学习算法仍缺乏令人满意的理解。我们的工作对几种优化目标进行了系统分析，以评估它们寻找所有帕累托最优策略的能力，以及通过不同目标的偏好对所学策略的控制能力。随后，我们确定Tchebycheff标量化是MORL的一种有利标量化方法。考虑到Tchebycheff标量化的非平滑性，我们将其最小化问题重新表述为一个新的极小-极大-极大优化问题。接着，针对随机策略类，我们利用该重构提出了高效算法来学习帕累托最优策略。我们首先提出一种基于在线UCB的算法，针对单个给定偏好实现了 $\varepsilon$ 的学习误差，其样本复杂度为 $\tilde{\mathcal{O}}(\varepsilon^{-2})$。为了进一步降低不同偏好下环境探索的成本，我们提出了一个无偏好框架，该框架首先在无预定义偏好的情况下探索环境，然后为任意数量的偏好生成解决方案。我们证明，它在探索阶段仅需 $\tilde{\mathcal{O}}(\varepsilon^{-2})$ 的探索复杂度，且此后无需额外探索。最后，我们分析了平滑Tchebycheff标量化（Tchebycheff标量化的扩展），基于偏好向量的条目值，证明其在将帕累托最优策略与其他弱帕累托最优策略区分开来方面更具优势。此外，我们将算法和理论分析扩展以适应该优化目标。
英文摘要：
This paper investigates multi-objective reinforcement learning (MORL), which focuses on learning Pareto optimal policies in the presence of multiple reward functions. Despite MORL's significant empirical success, there is still a lack of satisfactory understanding of various MORL optimization targets and efficient learning algorithms. Our work offers a systematic analysis of several optimization targets to assess their abilities to find all Pareto optimal policies and controllability over learned policies by the preferences for different objectives. We then identify Tchebycheff scalarization as a favorable scalarization method for MORL. Considering the non-smoothness of Tchebycheff scalarization, we reformulate its minimization problem into a new min-max-max optimization problem. Then, for the stochastic policy class, we propose efficient algorithms using this reformulation to learn Pareto optimal policies. We first propose an online UCB-based algorithm to achieve an $\varepsilon$ learning error with an $\tilde{\mathcal{O}}(\varepsilon^{-2})$ sample complexity for a single given preference. To further reduce the cost of environment exploration under different preferences, we propose a preference-free framework that first explores the environment without pre-defined preferences and then generates solutions for any number of preferences. We prove that it only requires an $\tilde{\mathcal{O}}(\varepsilon^{-2})$ exploration complexity in the exploration phase and demands no additional exploration afterward. Lastly, we analyze the smooth Tchebycheff scalarization, an extension of Tchebycheff scalarization, which is proved to be more advantageous in distinguishing the Pareto optimal policies from other weakly Pareto optimal policies based on entry values of preference vectors. Furthermore, we extend our algorithms and theoretical analysis to accommodate this optimization target.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
