具有双重正则化的联邦离线策略优化
Federated Offline Policy Optimization with Dual Regularization
AI总结：
本文提出离线联邦策略优化算法DRPO，通过双重正则化应对离线FRL中的双层分布偏移，使智能体无需环境交互即可协作学习，并确保每轮联邦学习中的严格策略改进。
摘要：
联邦强化学习（FRL）被视为人工智能物联网时代智能决策的极具前景的解决方案。然而，现有的 FRL 方法在本地更新期间通常需要与环境进行反复交互，这在许多现实场景中可能成本极高甚至不可行。为了克服这一挑战，本文提出了一种新颖的离线联邦策略优化算法，名为 $\ exttt{DRPO}$，该算法使分布式智能体能够仅从私有静态数据中协作学习决策策略，而无需进一步的环境交互。$\ exttt{DRPO}$ 利用双重正则化，结合了局部行为策略和全局聚合策略，以审慎应对离线 FRL 中固有的双层分布偏移。理论分析刻画了双重正则化对性能的影响，证明了通过实现适当的平衡，$\ exttt{DRPO}$ 能够有效抵消分布偏移，并确保在每一轮联邦学习轮次中实现严格的策略改进。大量实验验证了 $\ exttt{DRPO}$ 相较于基线方法的显著性能提升。
英文摘要：
Federated Reinforcement Learning (FRL) has been deemed as a promising solution for intelligent decision-making in the era of Artificial Internet of Things. However, existing FRL approaches often entail repeated interactions with the environment during local updating, which can be prohibitively expensive or even infeasible in many real-world domains. To overcome this challenge, this paper proposes a novel offline federated policy optimization algorithm, named $\texttt{DRPO}$, which enables distributed agents to collaboratively learn a decision policy only from private and static data without further environmental interactions. $\texttt{DRPO}$ leverages dual regularization, incorporating both the local behavioral policy and the global aggregated policy, to judiciously cope with the intrinsic two-tier distributional shifts in offline FRL. Theoretical analysis characterizes the impact of the dual regularization on performance, demonstrating that by achieving the right balance thereof, $\texttt{DRPO}$ can effectively counteract distributional shifts and ensure strict policy improvement in each federative learning round. Extensive experiments validate the significant performance gains of $\texttt{DRPO}$ over baseline methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
