arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2309.11235cs.CL
收藏

OpenChat:利用混合质量数据推进开源语言模型

OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, Yang Liu

AI总结:

OpenChat提出C-RLFT框架,利用混合质量数据训练开源语言模型,通过类别条件策略实现单阶段监督学习,使13B模型在多个基准上达到最优性能。

摘要:

如今,像LLaMA这样的开源大型语言模型已经出现。近期的发展引入了监督微调(SFT)和强化学习微调(RLFT)来使这些模型与人类目标对齐。然而,SFT方法将所有混合质量的训练数据同等对待,而RLFT方法则需要高质量的成对或基于排名的偏好数据。在本研究中,我们提出了一个名为OpenChat的新框架,以利用混合质量数据推进开源语言模型。具体来说,我们考虑一般的SFT训练数据,其中包含少量专家数据与大量次优数据混合,且没有任何偏好标签。我们提出了C(条件)-RLFT,它将不同的数据来源视为粗粒度的奖励标签,并学习一个类别条件策略以利用互补的数据质量信息。有趣的是,C-RLFT中的最优策略可以通过单阶段、无需强化学习的监督学习轻松求解,这既轻量又避免了昂贵的人类偏好标注。通过在三个标准基准上的广泛实验,我们使用C-RLFT微调的openchat-13b在所有13b开源语言模型中取得了最高的平均性能。此外,我们使用AGIEval来验证模型的泛化性能,其中只有openchat-13b超过了基础模型。最后,我们进行了一系列分析以阐明OpenChat的有效性和鲁棒性。我们的代码、数据和模型可在https://github.com/imoneoi/openchat和https://huggingface.co/openchat公开获取。

英文摘要:

Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at https://github.com/imoneoi/openchat and https://huggingface.co/openchat.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑