arXiv 2604.021512026-04cs.LG

基于拍卖的在线策略适应用于演化的目标

Auction-Based Online Policy Adaptation for Evolving Objectives

University of Pennsylvania（宾夕法尼亚大学）；IMDEA Software Institute（IMDEA软件研究所）

Guruprerana Shabadi, Kaushik Mallik

备注：: 22 pages, 8 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于拍卖的模块化框架，通过自利局部策略和动态拍卖机制实现多目标强化学习中目标变化时的策略适应，证明了纳什均衡的存在性和有效性。

摘要：

我们考虑多目标强化学习问题，其中目标来自相同家族（如可达性目标）并在运行时出现或消失。我们的目标是设计能高效调整行为的适应策略。为此，我们提出一个模块化框架，每个目标由一个自利局部策略支持，通过新颖的拍卖机制协调：策略根据当前状态的紧迫性进行投标，最高投标者选择动作，实现动态且可解释的目标权衡。当目标变化时，系统通过添加或移除相应策略进行适应。由于目标来自相同家族，可部署相同参数化的策略以实现运行时的即时适应。我们通过将问题转化为一般和博弈，证明了自利局部策略的计算方法。每个策略必须不仅优化自身目标，还考虑其他目标的存在并学习产生校准的投标以反映相对优先级。在温和假设下，我们证明了纳什均衡的存在性，其中不诚实投标导致次优结果，最紧迫的目标自动获得控制权。在实现中，策略通过近端策略优化（PPO）同时训练。我们在两个Atari游戏和基于网格世界的路径规划任务中进行了评估，证明了方法在PPO训练的单体策略上的显著优势。

英文摘要：

We consider multi-objective reinforcement learning problems where objectives come from an identical family -- such as the class of reachability objectives -- and may appear or disappear at runtime. Our goal is to design adaptive policies that can efficiently adjust their behaviors as the set of active objectives changes. To solve this problem, we propose a modular framework where each objective is supported by a selfish local policy, and coordination is achieved through a novel auction-based mechanism: policies bid for the right to execute their actions, with bids reflecting the urgency of the current state. The highest bidder selects the action, enabling a dynamic and interpretable trade-off among objectives. Going back to the original adaptation problem, when objectives change, the system adapts by simply adding or removing the corresponding policies. Moreover, as objectives arise from the same family, identical copies of a parameterized policy can be deployed, facilitating immediate adaptation at runtime. We show how the selfish local policies can be computed by turning the problem into a general-sum Markov game, where the policies compete against each other to fulfill their own objectives. To succeed, each policy must not only optimize its own objective, but also reason about the presence of other goals and learn to produce calibrated bids that reflect relative priority. Under mild assumptions, we prove the existence of Nash equilibria where dishonest bidding leads to suboptimal outcome, and the most urgent objectives win control automatically. In our implementation, the policies are trained concurrently using proximal policy optimization (PPO). We evaluate on two Atari games and a gridworld-based path-planning task with dynamic targets. Our method achieves substantially better performance than monolithic policies trained with PPO.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文