arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.02861cs.CLcs.LG
收藏

使用异构反馈微调 LLM 的框架

A Framework for Fine-Tuning LLMs using Heterogeneous Feedback

Carnegie Mellon University(卡内基梅隆大学);Adobe Research(奥多比研究院)

Ryan Aponte, Ryan A. Rossi, Shunan Guo, Franck Dernoncourt, Tong Yu, Xiang Chen, Subrata Mitra, Nedim Lipka

AI总结:

本文提出一种异构反馈 LLM 微调框架,通过统一监督格式并筛选高质量多样化子集,使模型在指令遵循和减少偏见等方面同时获得提升。

摘要:

大型语言模型(LLM)已被应用于广泛的任务,包括文本摘要、网页导航和聊天机器人。在无监督预训练之后,它们已从监督微调(SFT)和基于人类反馈的强化学习(RLHF)中受益。这些数据集可能难以收集、范围有限,并且样本质量参差不齐。此外,数据集在监督格式上可能存在很大差异,从数值型到二值型,以及具有许多不同取值的多维格式。我们提出了一个使用异构反馈微调 LLM 的框架,该框架包含两个主要组成部分。首先,我们将异构反馈数据组合成单一的监督格式,使其与 SFT 和 RLHF 等方法兼容。接下来,在给定这一统一反馈数据集的情况下,我们提取一个高质量且多样化的子集,以获得可能超过完整数据集的性能提升。我们进行了大量实验,以理解这些用于整合异构反馈的技术的有效性,并展示了使用高质量且多样化的数据子集所带来的改进。我们发现,我们的框架能够同时在多个方面改进模型,例如指令遵循和偏见减少。

英文摘要:

Large language models (LLMs) have been applied to a wide range of tasks, including text summarization, web navigation, and chatbots. They have benefitted from supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) following an unsupervised pretraining. These datasets can be difficult to collect, limited in scope, and vary in sample quality. Additionally, datasets can vary extensively in supervision format, from numerical to binary as well as multi-dimensional with many different values. We present a framework for fine-tuning LLMs using heterogeneous feedback, which has two main components. First, we combine the heterogeneous feedback data into a single supervision format, compatible with methods like SFT and RLHF. Next, given this unified feedback dataset, we extract a high-quality and diverse subset to obtain performance increases potentially exceeding the full dataset. We conduct extensive experiments to understand the effectiveness of these techniques for incorporating heterogeneous feedback, and demonstrate improvements from using a high-quality and diverse subset of the data. We find that our framework is able to improve models in multiple areas simultaneously, such as in instruction following and bias reduction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑