arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.03236cs.LGstat.ML
收藏

具有约束异质性的联邦强化学习

Federated Reinforcement Learning with Constraint Heterogeneity

Peking University(北京大学)

Hao Jin, Liangyu Zhang, Zhihua Zhang

AI总结:

本文针对约束信号访问受限的多智能体联邦强化学习问题,提出基于原始-对偶策略优化的 FedNPG 和 FedPPO 方法,并证明 FedNPG 具有全局收敛性,FedPPO 可高效处理复杂任务。

摘要:

我们研究了一个具有约束异质性的联邦强化学习(FedRL)问题。在该设定中,我们旨在解决一个带有多重约束的强化学习问题,同时有 $N$ 个训练智能体分别位于 $N$ 个不同的环境中,它们对约束信号的访问有限,并且期望通过协作学习到一个满足所有约束信号的策略。这类学习问题在大语言模型(LLM)微调和医疗健康应用中普遍存在。为了解决该问题,我们基于传统策略梯度方法提出了联邦原始-对偶策略优化方法。具体来说,我们为智能体引入 $N$ 个局部拉格朗日函数以进行局部策略更新,然后这些智能体被调度为周期性地就其局部策略进行通信。以自然策略梯度(NPG)和近端策略优化(PPO)作为策略优化方法,我们主要关注所提算法的两个实例,即 FedNPG 和 FedPPO。我们证明 FedNPG 以 $\tilde{O}(1/\sqrt{T})$ 的速率实现全局收敛,并且 FedPPO 能够借助深度神经网络高效求解复杂的学习任务。

英文摘要:

We study a Federated Reinforcement Learning (FedRL) problem with constraint heterogeneity. In our setting, we aim to solve a reinforcement learning problem with multiple constraints while $N$ training agents are located in $N$ different environments with limited access to the constraint signals and they are expected to collaboratively learn a policy satisfying all constraint signals. Such learning problems are prevalent in scenarios of Large Language Model (LLM) fine-tuning and healthcare applications. To solve the problem, we propose federated primal-dual policy optimization methods based on traditional policy gradient methods. Specifically, we introduce $N$ local Lagrange functions for agents to perform local policy updates, and these agents are then scheduled to periodically communicate on their local policies. Taking natural policy gradient (NPG) and proximal policy optimization (PPO) as policy optimization methods, we mainly focus on two instances of our algorithms, ie, {FedNPG} and {FedPPO}. We show that FedNPG achieves global convergence with an $\tilde{O}(1/\sqrt{T})$ rate, and FedPPO efficiently solves complicated learning tasks with the use of deep neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑