arXiv 2603.197412026-03-20cs.LGcs.CL

FedPDPO：联邦个性化直接偏好优化用于大语言模型对齐

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

College of Intelligence and Computing, Tianjin University, Tianjin, China（智能与计算学院，天津大学，天津，中国）；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）

Kewen Zhu, Liping Yi, Zhiming Zhao, Zhuang Qi, Han Yu, Qinghua Hu

备注：: under review

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出FedPDPO，一种联邦个性化直接偏好优化框架，通过参数高效微调和个性化奖励头解决非iid数据下的大语言模型对齐问题，实验显示在联邦内域和跨域设置中平均准确率提升达4.80%。

摘要：

在联邦学习中对齐大语言模型（LLMs）与人类偏好具有挑战性，因为数据是分散的、隐私敏感的且高度非iid。直接偏好优化（DPO）为强化学习与人类反馈（RLHF）提供了一种高效替代方案，但在联邦学习中的直接应用在非iid数据下性能下降严重，且隐式奖励泛化能力有限。为弥合这一差距，我们提出了FedPDPO（联邦个性化直接偏好优化），一种用于LLM偏好评对齐的个性化联邦框架。它采用参数高效的微调架构，其中每个客户端维护一个冻结的预训练LLM主干，附加一个低秩适应（LoRA）适配器，以实现通信高效的聚合。为解决非iid异质性，我们设计了（1）全局共享的LoRA适配器与个性化客户端特定的LLM头部。此外，我们引入了（2）一种个性化DPO训练策略，带有客户端特定的显式奖励头部以补充隐式奖励并进一步缓解非iid异质性，以及（3）一个瓶颈适配器以平衡全局和本地特征。我们提供了理论分析，建立了概率基础和严谨性。在多个偏好数据集上的广泛实验表明，该方法在联邦内域和跨域设置中实现了最先进的性能，平均准确率提升达4.80%。

英文摘要：

Aligning large language models (LLMs) with human preferences in federated learning (FL) is challenging due to decentralized, privacy-sensitive, and highly non-IID preference data. Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning with human feedback (RLHF), but its direct application in FL suffers from severe performance degradation under non-IID data and limited generalization of implicit rewards. To bridge this gap, we propose FedPDPO (Federated Personalized Direct Preference Optimization), a personalized federated framework for preference alignment of LLMs. It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation. To address non-IID heterogeneity, we devise (1) the globally shared LoRA adapter with the personalized client-specific LLM head. Moreover, we introduce (2) a personalized DPO training strategy with a client-specific explicit reward head to complement implicit rewards and further alleviate non-IID heterogeneity, and (3) a bottleneck adapter to balance global and local features. We provide theoretical analysis establishing the probabilistic foundation and soundness. Extensive experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。