PluralLLM：通过联邦学习实现 LLM 的多元对齐
PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
AI总结：
本文提出 PluralLLM，利用 FedAvg 联邦学习让多用户群体在不共享敏感数据的情况下训练偏好预测器，在问答对齐任务中实现更快收敛、更高对齐分且保持公平性。
摘要：
确保大语言模型（LLM）在保护隐私和公平性的同时，契合多样化的人类偏好，仍是一项挑战。现有方法（如基于人类反馈的强化学习，RLHF）依赖集中式数据收集，导致计算成本高昂且侵犯隐私。我们提出 PluralLLM，这是一种基于联邦学习的方法，使多个用户群体能够协同训练一个基于 transformer 的偏好预测器，而无需共享敏感数据；该预测器也可作为奖励模型用于对齐 LLM。我们的方法利用 Federated Averaging（FedAvg，联邦平均）高效聚合偏好更新，实现了 46% 的更快收敛、对齐分数提升 4%，并且群体公平性指标与集中式训练几乎相同。在一项问答（Q/A）偏好对齐任务上的评估表明，PluralLLM 证明联邦偏好学习为使 LLM 对齐多样化人类价值观提供了一种可扩展且保护隐私的替代方案。
英文摘要：
Ensuring Large Language Models (LLMs) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as Reinforcement Learning from Human Feedback (RLHF), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM a federated learning-based approach that enables multiple user groups to collaboratively train a transformer-based preference predictor without sharing sensitive data, which can also serve as a reward model for aligning LLMs. Our method leverages Federated Averaging (FedAvg) to aggregate preference updates efficiently, achieving 46% faster convergence, a 4% improvement in alignment scores, and nearly the same group fairness measure as in centralized training. Evaluated on a Q/A preference alignment task, PluralLLM demonstrates that federated preference learning offers a scalable and privacy-preserving alternative for aligning LLMs with diverse human values.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
