一种面向约束多目标强化学习的离线适配框架
An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning
AI总结：
本研究提出一种离线适配框架，无需人工设定目标偏好，仅通过演示即可隐式推断多目标强化学习的预期策略，还可扩展至满足安全约束，经实验验证效果良好。
摘要：
近年来，多目标强化学习（RL）研究取得了显著进展，该领域旨在通过纳入各目标的偏好来平衡多个目标。在现有大多数研究中，部署阶段必须提供特定偏好，以明确指示所需的策略。然而，这些偏好的设计高度依赖人类先验知识，而这类知识通常通过对具备预期行为的高性能演示进行大量观察来获取。本研究针对多目标RL问题提出了一种简单却有效的离线适配框架，该框架无需假设人工设定的目标偏好，仅需若干演示即可隐式指示预期策略的偏好。此外，我们证明该框架可自然扩展，通过利用安全演示满足安全关键目标的约束，即便安全阈值未知也可实现。在离线多目标任务与安全任务上的实验结果表明，该框架能够推断出与真实偏好一致的策略，同时满足所提供演示隐含的约束条件。
英文摘要：
In recent years, significant progress has been made in multi-objective reinforcement learning (RL) research, which aims to balance multiple objectives by incorporating preferences for each objective. In most existing studies, specific preferences must be provided during deployment to indicate the desired policies explicitly. However, designing these preferences depends heavily on human prior knowledge, which is typically obtained through extensive observation of high-performing demonstrations with expected behaviors. In this work, we propose a simple yet effective offline adaptation framework for multi-objective RL problems without assuming handcrafted target preferences, but only given several demonstrations to implicitly indicate the preferences of expected policies. Additionally, we demonstrate that our framework can naturally be extended to meet constraints on safety-critical objectives by utilizing safe demonstrations, even when the safety thresholds are unknown. Empirical results on offline multi-objective and safe tasks demonstrate the capability of our framework to infer policies that align with real preferences while meeting the constraints implied by the provided demonstrations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
