arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01787cs.CL
收藏

DEFT:基于分布的高效微调以实现人类对齐

DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment

Southern University of Science and Technology(南方科技大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Science and Technology of China(中国科学技术大学);University of Copenhagen(哥本哈根大学)

Liang Zhu, Feiteng Fang, Yuelin Bai, Longze Chen, Zhexiang Zhang, Minghuan Tan, Min Yang

AI总结:

本文提出DEFT框架,通过数据过滤和分布引导提升模型对齐效率与泛化能力,减少训练时间。

摘要:

基于人类反馈的强化学习(RLHF)通过如近端策略优化(PPO)等算法对大语言模型(LLMs)进行对齐,但成本高且不稳定。已有方法试图替代PPO或结合监督微调(SFT)和对比学习进行直接微调和价值对齐。然而,这些方法仍需大量数据学习偏好,可能削弱LLMs的泛化能力。为提高对齐效率和性能并缓解泛化能力的损失,本文引入了基于分布的高效微调(DEFT),一种结合数据过滤和分布引导的对齐框架。通过计算语言模型输出分布与偏好数据差异分布之间的差分分布奖励,从原始数据中过滤出一小部分高质量子集,并将其整合到现有对齐方法中以引导模型输出分布。实验结果表明,DEFT增强的方法在对齐能力和泛化能力上均优于原始方法,且训练时间显著减少。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF), using algorithms like Proximal Policy Optimization (PPO), aligns Large Language Models (LLMs) with human values but is costly and unstable. Alternatives have been proposed to replace PPO or integrate Supervised Fine-Tuning (SFT) and contrastive learning for direct fine-tuning and value alignment. However, these methods still require voluminous data to learn preferences and may weaken the generalization ability of LLMs. To further enhance alignment efficiency and performance while mitigating the loss of generalization ability, this paper introduces Distribution-guided Efficient Fine-Tuning (DEFT), an efficient alignment framework incorporating data filtering and distributional guidance by calculating the differential distribution reward based on the output distribution of language model and the discrepancy distribution of preference data. A small yet high-quality subset is filtered from the raw data using a differential distribution reward, which is then incorporated into existing alignment methods to guide the model's output distribution. Experimental results demonstrate that the methods enhanced by DEFT outperform the original methods in both alignment capability and generalization ability, with significantly reduced training time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑