arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.16944cs.LGcs.CL
收藏

Zephyr:LM对齐的直接蒸馏

Zephyr: Direct Distillation of LM Alignment

HuggingFace;HuggingFaceH4
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Hab… 展开作者

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf

AI总结:

本文提出Zephyr-7B,通过蒸馏直接偏好优化(dDPO)利用AI反馈偏好数据,以少量训练实现7B模型意图对齐,无需人工标注并在聊天基准上超越Llama2-Chat-70B。

摘要:

我们的目标是产生一个更小的、与用户意图对齐的语言模型。先前的研究表明,在较大的模型上应用蒸馏监督微调(dSFT)可以显著提高任务准确性;然而,这些模型并未对齐,即它们不能很好地响应自然提示。为了蒸馏这一特性,我们尝试使用来自AI反馈(AIF)的偏好数据。从一个由教师模型排序的输出数据集开始,我们应用蒸馏直接偏好优化(dDPO)来学习一个意图对齐显著改善的聊天模型。该方法只需要几个小时的训练,在微调期间无需任何额外采样。最终结果Zephyr-7B在7B参数模型的聊天基准上设立了最先进水平,并且不需要人工标注。特别是,MT-Bench上的结果表明,Zephyr-7B超越了Llama2-Chat-70B,后者是基于RLHF的最佳开放获取模型。系统的代码、模型、数据和教程可在https://github.com/huggingface/alignment-handbook获取。

英文摘要:

We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets the state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑