面向大语言模型的基于聚合客户端偏好的联邦RLHF
Towards Federated RLHF with Aggregated Client Preference for LLMs
AI总结：
本文提出联邦RLHF方法FedBis和FedBiscuit，通过聚合客户端二元选择器捕获共同偏好，并建立首个异构偏好联邦RLHF基准，显著提升LLM生成内容的专业性和可读性。
摘要：
基于人类反馈的强化学习（RLHF）利用用户偏好数据微调预训练大语言模型（LLM），使其能够生成符合人类偏好的内容。然而，出于隐私顾虑，用户可能不愿分享敏感的偏好数据。为解决这一问题，我们提出利用联邦学习（FL）技术，允许从多样化的真实用户中进行大规模偏好收集，而无需他们将数据传输到中央服务器。我们的联邦RLHF方法（即FedBis和FedBiscuit）将每个客户端的偏好编码为二元选择器，并对其进行聚合以捕获共同偏好。特别是，FedBiscuit通过创新性方案克服了偏好异质性和奖励黑客等关键挑战，例如将具有相似偏好的客户端分组以降低异质性，并使用多个二元选择器来提升LLM输出质量。为评估所提方法的性能，我们建立了首个带有异构人类偏好数据集的联邦RLHF基准。实验结果表明，通过将LLM与聚合的客户端偏好相结合，FedBis和FedBiscuit显著提升了生成内容的专业性和可读性。
英文摘要：
Reinforcement learning with human feedback (RLHF) fine-tunes a pretrained large language model (LLM) using user preference data, enabling it to generate content aligned with human preferences. However, due to privacy concerns, users may be reluctant to share sensitive preference data. To address this, we propose utilizing Federated Learning (FL) techniques, allowing large-scale preference collection from diverse real-world users without requiring them to transmit data to a central server. Our federated RLHF methods (i.e., FedBis and FedBiscuit) encode each client's preferences into binary selectors and aggregate them to capture common preferences. In particular, FedBiscuit overcomes key challenges, such as preference heterogeneity and reward hacking, through innovative solutions like grouping clients with similar preferences to reduce heterogeneity and using multiple binary selectors to enhance LLM output quality. To evaluate the performance of the proposed methods, we establish the first federated RLHF benchmark with a heterogeneous human preference dataset. Experimental results show that by integrating the LLM with aggregated client preferences, FedBis and FedBiscuit significantly enhance the professionalism and readability of the generated content.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
