arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.14922cs.CLcs.AI
收藏

RobustFT:噪声响应下大型语言模型的鲁棒监督微调

RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response

Peking University(北京大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Northwestern University(西北大学);University of Washington(华盛顿大学)

Junyu Luo, Xiao Luo, Kaize Ding, Jingyang Yuan, Zhiping Xiao, Ming Zhang

AI总结:

针对监督微调数据中不可避免的噪声问题,提出RobustFT框架,通过多专家协作系统进行噪声检测,结合上下文增强策略去噪重标注,并利用基于响应熵的数据选择机制保留高质量样本,显著提升了LLMs在噪声场景下的下游任务性能。

摘要:

监督微调(SFT)在将大型语言模型(LLMs)适应到特定领域或任务中起着至关重要的作用。然而,正如经验实验所证明的那样,收集的数据在实际应用中不可避免地包含噪声,这对模型在下游任务上的性能构成了重大挑战。因此,迫切需要一个噪声鲁棒的SFT框架来增强模型在下游任务中的能力。为了应对这一挑战,我们引入了一个鲁棒的SFT框架(RobustFT),该框架对下游任务数据执行噪声检测和重新标记。在噪声识别方面,我们的方法采用了一个多专家协作系统,结合推理增强模型,以实现卓越的噪声检测。在去噪阶段,我们利用上下文增强策略,该策略整合了最相关且置信的知识,随后进行仔细评估以生成可靠的标注。此外,我们引入了一种基于响应熵的有效数据选择机制,确保仅保留高质量样本用于微调。在多个LLMs和五个数据集上进行的广泛实验表明,RobustFT在噪声场景下具有卓越的性能。

英文摘要:

Supervised fine-tuning (SFT) plays a crucial role in adapting large language models (LLMs) to specific domains or tasks. However, as demonstrated by empirical experiments, the collected data inevitably contains noise in practical applications, which poses significant challenges to model performance on downstream tasks. Therefore, there is an urgent need for a noise-robust SFT framework to enhance model capabilities in downstream tasks. To address this challenge, we introduce a robust SFT framework (RobustFT) that performs noise detection and relabeling on downstream task data. For noise identification, our approach employs a multi-expert collaborative system with inference-enhanced models to achieve superior noise detection. In the denoising phase, we utilize a context-enhanced strategy, which incorporates the most relevant and confident knowledge followed by careful assessment to generate reliable annotations. Additionally, we introduce an effective data selection mechanism based on response entropy, ensuring only high-quality samples are retained for fine-tuning. Extensive experiments conducted on multiple LLMs across five datasets demonstrate RobustFT's exceptional performance in noisy scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑