arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.12110cs.LG
收藏

用于生成多样化专家策略的帕累托逆强化学习

Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation

Sungkyunkwan University(成均馆大学)

Woo Kyung Kim, Minjong Yoo, Honguk Woo

AI总结:

本文提出 ParIRL 框架,利用奖励距离估计正则化判别器,从有限专家数据集生成适应多目标偏好的帕累托策略集,并将其蒸馏为偏好条件扩散模型,在多任务和 CARLA 自动驾驶中优于其他 IRL 算法。

摘要:

数据驱动的离线强化学习和模仿学习方法在解决序列决策问题中日益受到关注。然而,这些方法很少考虑从有限的专家数据集中学习帕累托最优策略。由于获取所有偏好的全面数据集在实际中存在局限,这一问题尤为突出,尤其是在存在多个冲突目标且每个专家可能对这些目标持有独特优化偏好的情况下。本文通过使用奖励距离估计来正则化判别器,从而对逆强化学习(IRL)进行了改进。这使得在仅使用两个不同数据集(每个数据集关联不同的专家偏好)的情况下,能够渐进式地生成一组适应多目标上不同偏好的策略。为此,我们提出了一个 Pareto IRL 框架(ParIRL),从这些有限的数据集中建立帕累托策略集。在该框架中,帕累托策略集随后被蒸馏为一个单一的、偏好条件的扩散模型,从而允许用户立即指定他们偏好哪种专家模式。通过实验,我们表明 ParIRL 在各种多目标控制任务中优于其他 IRL 算法,实现了对帕累托前沿的密集近似。我们还展示了 ParIRL 在 CARLA 中自动驾驶的适用性。

英文摘要:

Data-driven offline reinforcement learning and imitation learning approaches have been gaining popularity in addressing sequential decision-making problems. Yet, these approaches rarely consider learning Pareto-optimal policies from a limited pool of expert datasets. This becomes particularly marked due to practical limitations in obtaining comprehensive datasets for all preferences, where multiple conflicting objectives exist and each expert might hold a unique optimization preference for these objectives. In this paper, we adapt inverse reinforcement learning (IRL) by using reward distance estimates for regularizing the discriminator. This enables progressive generation of a set of policies that accommodate diverse preferences on the multiple objectives, while using only two distinct datasets, each associated with a different expert preference. In doing so, we present a Pareto IRL framework (ParIRL) that establishes a Pareto policy set from these limited datasets. In the framework, the Pareto policy set is then distilled into a single, preference-conditioned diffusion model, thus allowing users to immediately specify which expert's patterns they prefer. Through experiments, we show that ParIRL outperforms other IRL algorithms for various multi-objective control tasks, achieving the dense approximation of the Pareto frontier. We also demonstrate the applicability of ParIRL with autonomous driving in CARLA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑