Real-world sequential decision-making tasks often require balancing trade-offs between multiple conflicting objectives, making Multi-Objective Reinforcement Learning (MORL) an increasingly prominent field of research. Despite recent advances, existing MORL literature has narrowly focused on performance within static environments, neglecting the importance of generalizing across diverse settings. Conversely, existing research on generalization in RL has always assumed scalar rewards, overlooking the inherent multi-objectivity of real-world problems. Generalization in the multi-objective context is fundamentally more challenging, as it requires learning a Pareto set of policies addressing varying preferences across multiple objectives. In this paper, we formalize the concept of generalization in MORL and how it can be evaluated. We then contribute a novel benchmark featuring diverse multi-objective domains with parameterized environment configurations to facilitate future studies in this area. Our baseline evaluations of state-of-the-art MORL algorithms on this benchmark reveals limited generalization capabilities, suggesting significant room for improvement. Our empirical findings also expose limitations in the expressivity of scalar rewards, emphasizing the need for multi-objective specifications to achieve effective generalization. We further analyzed the algorithmic complexities within current MORL approaches that could impede the transfer in performance from the single- to multiple-environment settings. This work fills a critical gap and lays the groundwork for future research that brings together two key areas in reinforcement learning: solving multi-objective decision-making problems and generalizing across diverse environments. We make our code available at https://github.com/JaydenTeoh/MORL-Generalization.
Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning
Federation University Australia(澳大利亚联邦大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);University of Galway(高威大学);Deakin University(迪肯大学);Linköping University(林雪平大学);Universidade do Vale do Rio dos Sinos(南里约 dos Sinos大学);Vrije Universiteit Brussel(布鲁塞尔自由大学)
Divide and Conquer: Provably Unveiling the Pareto Front with Multi-Objective Reinforcement Learning
Vrije Universiteit Brussel(布鲁塞尔自由大学);Université de Montréal(蒙特利尔大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);University of Galway(戈尔韦大学);City of Amsterdam(阿姆斯特丹市);Utrecht University(乌得勒支大学)
DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learning
Department of Computing Science, University of Alberta(阿尔伯塔大学计算机科学系);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔人工智能研究所);Canada CIFAR AI Chair(加拿大 CIFAR 人工智能主席)
Controllability in preference-conditioned multi-objective reinforcement learning
LTCI, Télécom Paris, Institut Polytechnique de Paris(LTCI,巴黎电信学院,巴黎理工学院);University of California, Berkeley(加州大学伯克利分校);TU Delft(代尔夫特理工大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm
School of Electical Engineering, Korea Advanced Institute of Science & Technology(韩国科学技术院电气工程学院);Faculty of Engineering, Bar-Ilan University(巴伊兰大学工程学院)
Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignment
Institute of Innovation, Science and Sustainability(创新、科学与可持续发展研究所);Federation University Australia(澳大利亚联邦大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);School of IT(信息技术学院);Deakin University(迪肯大学)
A Practical Guide to Multi-Objective Reinforcement Learning and Planning
National University of Ireland Galway(爱尔兰国立高威大学);Vrije Universiteit Brussel(布鲁塞尔自由大学);Linköping University(林雪平大学);University of Amsterdam(阿姆斯特丹大学);University of Oxford(牛津大学);Deakin University(迪肯大学);University of Washington(华盛顿大学);Universidade do Vale do Rio dos Sinos(南里奥格兰德河谷大学);Politecnico di Milano(米兰理工大学);Federation University Australia(澳大利亚联邦大学)
What is the objective of reasoning with reinforcement learning?
Department of Statistics and Data Science, The Wharton School, University of Pennsylvania(统计与数据科学系,沃森商学院,宾夕法尼亚大学);Department of Electrical Engineering and Computer Sciences, University of California, Berkeley(电气工程与计算机科学系,加州大学伯克利分校)