通过离线多目标强化学习扩展帕累托高效决策
Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL
AI总结：
针对多目标强化学习中偏好未知的离线决策问题，本文构建了含180万条演示的D4MORL离线数据集，并提出基于偏好与回报条件策略的PEDA算法，可高效逼近帕累托前沿。
摘要：
多目标强化学习（MORL）的目标是学习可同时优化多个相互冲突目标的策略。在实际场景中，智能体对各目标的偏好可能无法预先获知，因此我们需要能在测试时泛化到任意偏好的策略。本文提出一种全新的数据驱动型离线MORL设置，旨在仅利用包含其他智能体离线演示及其偏好的有限数据集，学习一个与偏好无关的策略智能体。本研究的核心贡献分为两方面：其一，我们推出了D4MORL，即专为离线场景设计的多目标强化学习数据集（(D)atasets for MORL）。该数据集包含180万条带标注的演示数据，这些数据通过在6个MuJoCo环境（每个环境含2-3个目标）中执行针对随机采样偏好优化的参考策略滚动生成。其二，我们提出了帕累托高效决策智能体（PEDA），这是一类离线MORL算法，通过一种新颖的偏好与回报条件策略构建并扩展了Decision Transformer。实验结果表明，在D4MORL基准上，PEDA能够紧密逼近行为策略，并且通过适当的条件设置，可实现对帕累托前沿的极佳近似，该结论由超体积和稀疏度指标验证。
英文摘要：
The goal of multi-objective reinforcement learning (MORL) is to learn policies that simultaneously optimize multiple competing objectives. In practice, an agent's preferences over the objectives may not be known apriori, and hence, we require policies that can generalize to arbitrary preferences at test time. In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences. The key contributions of this work are two-fold. First, we introduce D4MORL, (D)atasets for MORL that are specifically designed for offline settings. It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for randomly sampled preferences on 6 MuJoCo environments with 2-3 objectives each. Second, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that builds and extends Decision Transformers via a novel preference-and-return-conditioned policy. Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides an excellent approximation of the Pareto-front with appropriate conditioning, as measured by the hypervolume and sparsity metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
