FedGuide：面向异构联邦强化学习的扩散先验对齐与价值基线引导
FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning
浏览论文内容
中文总结 AI 辅助
针对异构联邦强化学习中的分布不匹配问题，提出FedGuide框架，利用扩散先验作为行为模型，通过OT-MoE聚合保留异构模式，并采用DICE价值基线引导策略改进，实验证明其在回报和鲁棒性上优于现有方法。
中文摘要 AI 辅助
联邦强化学习（FRL）使得在具有异构环境的分布式智能体之间进行协作策略学习成为可能。尽管近期基于方差缩减、散度惩罚和动量优化的方法在异构设置下提升了FRL的性能，但它们仍主要同步策略或价值网络参数，并未明确解决异构客户端之间的分布不匹配问题。为此，我们提出FedGuide，一种FRL框架，利用扩散先验作为行为模型，为异构局部策略学习提供个性化的数据支持分布。FedGuide并非直接平均局部策略，而是通过最优传输专家混合（OT-MoE）聚合这些扩散先验，在分布空间中保留异构行为模式。它进一步开发了分布校正估计（DICE）价值基线，为局部策略改进提供低方差、回报感知的引导。在异构环境中的实验表明，FedGuide在客户端平均回报、最终轮次性能和最差轮次鲁棒性方面优于代表性FRL方法，同时在更强的异构性下保持稳定的学习。
英文摘要
Federated Reinforcement Learning (FRL) enables collaborative policy learning across distributed agents with heterogeneous environments. While recent methods based on variance reduction, divergence penalization, and momentum optimization improve FRL under heterogeneous settings, they still primarily synchronize policy or value-network parameters and do not explicitly address distributional mismatch among heterogeneous clients. Therefore, we propose \textbf{FedGuide}, a FRL framework that uses diffusion priors as behavior models to provide personalized data supported distributions for heterogeneous local policy learning. Instead of directly averaging local policies, FedGuide aggregates those diffusion priors through Optimal-Transport Mixture-of-Experts (OT-MoE), preserving heterogeneous behavior modes in distribution space. It further develops a Distribution Correction Estimation (DICE) value baseline to provide low-variance, return-aware guidance for local policy improvement. Experiments across heterogeneous environments show that FedGuide outperforms representative FRL methods in client-average returns, final-round performance, and worst-round robustness, while maintaining stable learning under stronger heterogeneity.
发表机构
- Carnegie Mellon University（卡内基梅隆大学）
机构由 AI 辅助整理，请以论文原文为准。
