arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.25311cs.LGcs.AI
收藏

密集且多样的多目标强化学习中的目标覆盖

Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning

Sagalpreet Singh, Rishi Saket, Aravindan Raghuveer

AI总结:

本文提出了一种新的多目标强化学习算法,旨在在最大化预期回报的同时,使状态分布均匀覆盖目标状态,通过自定义奖励函数和离线强化学习算法实现高效收敛。

摘要:

强化学习算法主要关注学习一个最大化预期回报的策略。因此,学习到的策略可能只利用一个或少数奖励源。然而,在许多自然情况下,学习一个诱导分散的边际状态分布覆盖奖励状态,同时最大化预期回报(通常与达到目标状态相关)是理想的。现有技术基于熵正则化和内在奖励使用随机性鼓励探索以找到最优策略,但可能不必然导致分散的边际状态分布覆盖奖励状态。其他RL算法假设目标分布已知。这在大规模系统中可能不可行,因为无法枚举所有状态,且只有在到达状态时才确定其为目标状态。我们正式将最大化预期回报同时均匀访问目标状态的问题定义为多目标强化学习(Multi Goal RL),其中一个Oracle分类器确定目标状态。我们提出了一种新的算法,学习一个高回报策略混合,其边际状态分布分散在目标状态集上。我们的算法基于优化一个自定义的RL奖励函数,该函数在每次迭代中基于当前策略混合计算一组采样轨迹。后者通过离线RL算法更新策略混合。我们证明了该算法的性能保证,展示了优化自然目标的高效收敛界,该目标捕捉了预期回报以及边际状态分布在目标状态上的分散性。我们设计并进行了合成MDP和标准RL环境的实验以评估算法的有效性。

英文摘要:

Reinforcement Learning algorithms are primarily focused on learning a policy that maximizes expected return. As a result, the learned policy can exploit one or few reward sources. However, in many natural situations, it is desirable to learn a policy that induces a dispersed marginal state distribution over rewarding states, while maximizing the expected return which is typically tied to reaching a goal state. This aspect remains relatively unexplored. Existing techniques based on entropy regularization and intrinsic rewards use stochasticity for encouraging exploration to find an optimal policy which may not necessarily lead to dispersed marginal state distribution over rewarding states. Other RL algorithms which match a target distribution assume the latter to be available apriori. This may be infeasible in large scale systems where enumeration of all states is not possible and a state is determined to be a goal state only upon reaching it. We formalize the problem of maximizing the expected return while uniformly visiting the goal states as Multi Goal RL in which an oracle classifier over the state space determines the goal states. We propose a novel algorithm that learns a high-return policy mixture with marginal state distribution dispersed over the set of goal states. Our algorithm is based on optimizing a custom RL reward which is computed - based on the current policy mixture - at each iteration for a set of sampled trajectories. The latter are used via an offline RL algorithm to update the policy mixture. We prove performance guarantees for our algorithm, showing efficient convergence bounds for optimizing a natural objective which captures the expected return as well as the dispersion of the marginal state distribution over the goal states. We design and perform experiments on synthetic MDPs and standard RL environments to evaluate the effectiveness of our algorithm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑