FORLER: 联邦离线强化学习中的Q-集成与演员校正
FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification
AI总结：
FORLER通过Q-集成和演员校正，解决离线联邦RL中的策略污染问题，提升鲁棒性和性能。
摘要：
在物联网系统中，联邦学习通过允许并行策略训练而提升了在线强化学习（RL），无需共享原始数据。然而，与真实环境在线交互可能具有风险和成本，推动了离线联邦RL（FRL）的发展，其中本地设备从固定数据集学习。尽管其前景广阔，离线FRL在低质量、异质数据下可能失效。离线RL容易陷入局部最优，而在FRL中，一个设备的次优策略会降级聚合模型，即策略污染。我们提出了FORLER，结合服务器上的Q-集成聚合与设备上的演员校正。服务器稳健地合并设备Q函数以遏制策略污染，并将计算转移到资源受限硬件上，而不影响隐私。本地方面，演员校正通过零阶搜索寻找高Q动作以及定制的正则化器，将策略推向这些动作。Δ-周期性策略进一步减少本地计算。我们理论上提供了安全策略改进的性能保证。大量实验表明，FORLER在不同数据质量和异质性下均优于强基线。
英文摘要：
In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be risky and costly, motivating offline federated RL (FRL), where local devices learn from fixed datasets. Despite its promise, offline FRL may break down under low-quality, heterogeneous data. Offline RL tends to get stuck in local optima, and in FRL, one device's suboptimal policy can degrade the aggregated model, i.e., policy pollution. We present FORLER, combining Q-ensemble aggregation on the server with actor rectification on devices. The server robustly merges device Q-functions to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy. Locally, actor rectification enriches policy gradients via a zeroth-order search for high-Q actions plus a bespoke regularizer that nudges the policy toward them. A $δ$-periodic strategy further reduces local computation. We theoretically provide safe policy improvement performance guarantees. Extensive experiments show FORLER consistently outperforms strong baselines under varying data quality and heterogeneity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
