FedMOA: 基于异质奖励的联邦GRPO用于个性化推理大语言模型
FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards
AI总结：
FedMOA是一种联邦GRPO框架，用于在异质奖励下实现个性化推理大语言模型的多目标对齐，通过自适应加权机制和任务感知聚合策略提升性能和个性化效果。
摘要：
组相对策略优化（GRPO）最近出现，成为通过在线多目标强化学习提高大型语言模型推理能力的有效方法。虽然在私人数据上的个性化越来越重要，但传统的强化学习（RL）对齐往往在设备端联邦学习中由于维护单独的批评者网络而变得内存不可行。GRPO的批评者自由架构使设备端训练成为可能，但转向联邦设置会引入系统性挑战：异质奖励定义、不平衡的多目标优化以及高训练成本。我们提出FedMOA，一种用于异质奖励下的多目标对齐的联邦GRPO框架。FedMOA通过超梯度下降的在线自适应加权机制稳定本地训练，优先处理主要推理，当辅助目标饱和时。在服务器端，它利用任务和准确度意识的聚合策略来优先考虑高质量的更新。在数学推理和代码生成基准测试中，FedMOA在联邦平均上持续表现更优，实现了高达2.2%的准确性提升，同时改善了全局性能、个性化和多目标平衡。
英文摘要：
Group Relative Policy Optimization (GRPO) has recently emerged as an effective approach for improving the reasoning capabilities of large language models through online multi-objective reinforcement learning. While personalization on private data is increasingly vital, traditional Reinforcement Learning (RL) alignment is often memory-prohibitive for on-device federated learning due to the overhead of maintaining a separate critic network. GRPO's critic-free architecture enables feasible on-device training, yet transitioning to a federated setting introduces systemic challenges: heterogeneous reward definitions, imbalanced multi-objective optimization, and high training costs. We propose FedMOA, a federated GRPO framework for multi-objective alignment under heterogeneous rewards. FedMOA stabilizes local training through an online adaptive weighting mechanism via hypergradient descent, which prioritizes primary reasoning as auxiliary objectives saturate. On the server side, it utilizes a task- and accuracy-aware aggregation strategy to prioritize high-quality updates. Experiments on mathematical reasoning and code generation benchmarks demonstrate that FedMOA consistently outperforms federated averaging, achieving accuracy gains of up to 2.2% while improving global performance, personalization, and multi-objective balance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
