FedRLHF：一种面向隐私保护与个性化RLHF的收敛保证联邦框架
FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF
AI总结：
本文提出FedRLHF框架，通过联邦学习将RLHF去中心化，在保护隐私的同时实现个性化策略学习，并提供收敛保证与样本复杂度界，实验证明其性能可与集中式RLHF相媲美。
摘要：
在隐私担忧日益增加和个性化体验需求日益增长的背景下，传统的基于人类反馈的强化学习（RLHF）框架由于依赖集中式数据而面临重大挑战。我们提出联邦人类反馈强化学习（FedRLHF），这是一种将RLHF过程去中心化的新型框架。FedRLHF能够在多个客户端之间实现协作策略学习，而无需共享原始数据或人类反馈，从而确保稳健的隐私保护。利用联邦强化学习，每个客户端在本地将人类反馈整合到其奖励函数中，并通过个性化RLHF过程更新其策略。我们为FedRLHF建立了严格的理论基础，提供了收敛保证，并推导出随客户端数量高效扩展的样本复杂度界。在MovieLens和IMDb数据集上的实证评估表明，FedRLHF不仅保护了用户隐私，还实现了与集中式RLHF相当的性能，同时在不同客户端环境中增强了个性化。
英文摘要：
In the era of increasing privacy concerns and demand for personalized experiences, traditional Reinforcement Learning with Human Feedback (RLHF) frameworks face significant challenges due to their reliance on centralized data. We introduce Federated Reinforcement Learning with Human Feedback (FedRLHF), a novel framework that decentralizes the RLHF process. FedRLHF enables collaborative policy learning across multiple clients without necessitating the sharing of raw data or human feedback, thereby ensuring robust privacy preservation. Leveraging federated reinforcement learning, each client integrates human feedback locally into their reward functions and updates their policies through personalized RLHF processes. We establish rigorous theoretical foundations for FedRLHF, providing convergence guarantees, and deriving sample complexity bounds that scale efficiently with the number of clients. Empirical evaluations on the MovieLens and IMDb datasets demonstrate that FedRLHF not only preserves user privacy but also achieves performance on par with centralized RLHF, while enhancing personalization across diverse client environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
