arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.13385cs.CLcs.LG
收藏

Tuna:利用大型语言模型反馈进行指令微调

Tuna: Instruction Tuning using Feedback from Large Language Models

Singapore University of Technology and Design(新加坡科技设计大学);Microsoft Research Asia(微软亚洲研究院);Tsinghua University(清华大学)

Haoran Li, Yiran Liu, Xingxing Zhang, Wei Lu, Furu Wei

AI总结:

本文提出Tuna模型,通过概率排序和上下文排序方法对指令微调后的LLM进行微调,以增加生成更好响应的可能性,在多个基准测试中持续提升性能并超越强化学习基线。

摘要:

使用 Instruct-GPT 和 GPT-4 等更强大的大型语言模型(LLM)的直接输出,对 LLaMA 等开源大型语言模型进行指令微调,已被证明是使模型行为与人类偏好对齐的成本效益高的方法。然而,指令微调后的模型每个指令只见过一个响应,缺乏对潜在更好响应的知识。在本文中,我们提出使用我们新颖的概率排序和上下文排序方法对指令微调后的 LLM 进行微调,以增加生成更好响应的可能性。概率排序使指令微调后的模型能够从教师 LLM 继承高质量和低质量响应的相对排序。另一方面,使用上下文排序进行学习允许模型利用更强 LLM 的上下文理解能力来优化其自身的响应分布。此外,我们将概率排序和上下文排序依次应用于指令微调后的 LLM。我们称之为 Tuna 的最终模型,在 Super Natural Instructions(119 个测试任务)、LMentry(25 个测试任务)和 Vicuna QA 上持续提升性能,甚至可以获得比几个强强化学习基线更好的结果。我们的代码和数据可在 https://github.com/microsoft/LMOps 获取。

英文摘要:

Instruction tuning of open-source large language models (LLMs) like LLaMA, using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4, has proven to be a cost-effective way to align model behaviors with human preferences. However, the instruction-tuned model has only seen one response per instruction, lacking the knowledge of potentially better responses. In this paper, we propose finetuning an instruction-tuned LLM using our novel \textit{probabilistic ranking} and \textit{contextual ranking} approaches to increase the likelihood of generating better responses. Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs. Furthermore, we apply probabilistic ranking and contextual ranking sequentially to the instruction-tuned LLM. The resulting model, which we call \textbf{Tuna}, consistently improves the performance on Super Natural Instructions (119 test tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results than several strong reinforcement learning baselines. Our code and data are available at \url{ https://github.com/microsoft/LMOps}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑