arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.00799cs.LG
收藏

论多目标强化学习中的跨环境泛化

On Generalization Across Environments In Multi-Objective Reinforcement Learning

Singapore Management University(新加坡管理大学);Federation University Australia(澳大利亚联邦大学)

Jayden Teoh, Pradeep Varakantham, Peter Vamplew

AI总结:

本文形式化多目标强化学习的跨环境泛化问题,提出含参数化配置的多目标基准,并通过基线实验揭示现有 MORL 算法泛化不足及标量奖励表达局限。

摘要:

现实世界中的序贯决策任务往往需要在多个相互冲突的目标之间权衡取舍,这使得多目标强化学习(MORL)成为一个日益重要的研究领域。尽管近期取得了进展,但现有 MORL 文献仍狭隘地关注静态环境中的性能,忽视了在多样化设置之间进行泛化的重要性。相反,现有的 RL 泛化研究始终假设标量奖励,忽略了现实问题固有的多目标性。多目标情境下的泛化从根本上说更具挑战性,因为它需要学习一个 Pareto 策略集,以应对跨多个目标的不同偏好。在本文中,我们形式化了 MORL 中泛化的概念及其评估方式。随后,我们贡献了一个新颖的基准,该基准具有多样化的多目标域和参数化环境配置,以促进该领域未来的研究。我们在该基准上对最先进 MORL 算法进行的基线评估揭示了其有限的泛化能力,表明仍有显著的改进空间。我们的实证发现还暴露了标量奖励在表达能力方面的局限,强调需要多目标规范才能实现有效泛化。我们进一步分析了当前 MORL 方法中的算法复杂性,这些复杂性可能阻碍性能从单环境设置向多环境设置迁移。本工作填补了一个关键空白,并为未来结合强化学习两大关键领域的研究奠定基础:求解多目标决策问题以及跨多样化环境泛化。我们的代码可在 https://github.com/JaydenTeoh/MORL-Generalization 获取。

英文摘要:

Real-world sequential decision-making tasks often require balancing trade-offs between multiple conflicting objectives, making Multi-Objective Reinforcement Learning (MORL) an increasingly prominent field of research. Despite recent advances, existing MORL literature has narrowly focused on performance within static environments, neglecting the importance of generalizing across diverse settings. Conversely, existing research on generalization in RL has always assumed scalar rewards, overlooking the inherent multi-objectivity of real-world problems. Generalization in the multi-objective context is fundamentally more challenging, as it requires learning a Pareto set of policies addressing varying preferences across multiple objectives. In this paper, we formalize the concept of generalization in MORL and how it can be evaluated. We then contribute a novel benchmark featuring diverse multi-objective domains with parameterized environment configurations to facilitate future studies in this area. Our baseline evaluations of state-of-the-art MORL algorithms on this benchmark reveals limited generalization capabilities, suggesting significant room for improvement. Our empirical findings also expose limitations in the expressivity of scalar rewards, emphasizing the need for multi-objective specifications to achieve effective generalization. We further analyzed the algorithmic complexities within current MORL approaches that could impede the transfer in performance from the single- to multiple-environment settings. This work fills a critical gap and lays the groundwork for future research that brings together two key areas in reinforcement learning: solving multi-objective decision-making problems and generalizing across diverse environments. We make our code available at https://github.com/JaydenTeoh/MORL-Generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑