面向激励对齐的上下文双层强化学习
Contextual Bilevel Reinforcement Learning for Incentive Alignment
AI总结：
该研究提出上下文双层强化学习CB-RL模型及随机超策略梯度下降HPGD算法，拓展传统双层优化范畴，可应用于RLHF、税收设计等多领域，且在奖励塑形和税收设计任务中验证了性能。
摘要：
在各类现实战略决策问题中，最优策略既取决于环境配置，也受外生事件影响。针对这类场景，我们提出上下文双层强化学习（Contextual Bilevel Reinforcement Learning，CB-RL）——一种随机双层决策模型，其下层问题为求解上下文马尔可夫决策过程（Contextual Markov Decision Process，CMDP）。CB-RL可视为一种Stackelberg博弈：领导者与不受其控制的随机上下文共同决定多个MDP的设置，而潜在的多个跟随者会对这些设置做出最优响应。该框架拓展了传统双层优化的范畴，在RLHF、税收设计、奖励塑形、契约理论和机制设计等多个领域均有应用价值。我们提出随机超策略梯度下降（Hyper Policy Gradient Descent，HPGD）算法求解CB-RL，并证明了其收敛性。值得注意的是，HPGD基于跟随者的轨迹观测使用随机超梯度估计，因此允许跟随者采用任意训练流程，且领导者无需知晓具体算法，这与多种现实场景相契合。我们进一步考虑了领导者可影响跟随者训练的场景，并提出了一种加速算法。我们通过实验验证了该算法在奖励塑形和税收设计任务中的性能。
英文摘要：
The optimal policy in various real-world strategic decision-making problems depends both on the environmental configuration and exogenous events. For these settings, we introduce Contextual Bilevel Reinforcement Learning (CB-RL), a stochastic bilevel decision-making model, where the lower level consists of solving a contextual Markov Decision Process (CMDP). CB-RL can be viewed as a Stackelberg Game where the leader and a random context beyond the leader's control together decide the setup of many MDPs that potentially multiple followers best respond to. This framework extends beyond traditional bilevel optimization and finds relevance in diverse fields such as RLHF, tax design, reward shaping, contract theory and mechanism design. We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm to solve CB-RL, and demonstrate its convergence. Notably, HPGD uses stochastic hypergradient estimates, based on observations of the followers' trajectories. Therefore, it allows followers to use any training procedure and the leader to be agnostic of the specific algorithm, which aligns with various real-world scenarios. We further consider the setting when the leader can influence the training of followers and propose an accelerated algorithm. We empirically demonstrate the performance of our algorithm for reward shaping and tax design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
