arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.00907cs.LGcs.AIcs.CL
收藏

LaFFi:利用混合自然语言反馈微调语言模型

LaFFi: Leveraging Hybrid Natural Language Feedback for Fine-tuning Language Models

University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所);Huawei Noah’s Ark Laboratory(华为诺亚方舟实验室)

Qianxi Li, Yingyue Cao, Jikun Kang, Tianpei Yang, Xi Chen, Jun Jin, Matthew E. Taylor

AI总结:

本文提出LaFFi方法,通过让LLM直接预测标注者的自然语言反馈来进行反思微调,显著提升了领域内问答任务的准确性,并发现人工标注数据比例影响微调性能。

摘要:

微调大型语言模型(LLMs)使训练好的模型适应特定下游任务,显著提升特定任务的性能。监督微调(SFT)是一种常见方法,即训练LLM以产生期望的答案。然而,使用SFT训练的LLM有时会犯简单的错误,并导致在问答等推理任务上产生幻觉。如果没有外部反馈,SFT很难学习到问题和期望答案之间的良好映射,尤其是在数据集较小的情况下。本文提出了一种SFT的替代方法,称为用于微调LLMs的自然语言反馈(LaFFi)。LaFFi让LLM直接预测它们将从标注者那里收到的反馈。我们发现,要求这种反思可以显著提高领域内问答任务的准确性,为自然语言反馈在SFT LLMs领域的应用提供了一个有前景的方向。额外的消融研究表明,标注数据集中人工标注数据的比例会影响微调性能。

英文摘要:

Fine-tuning Large Language Models (LLMs) adapts a trained model to specific downstream tasks, significantly improving task-specific performance. Supervised Fine-Tuning (SFT) is a common approach, where an LLM is trained to produce desired answers. However, LLMs trained with SFT sometimes make simple mistakes and result in hallucinations on reasoning tasks such as question-answering. Without external feedback, it is difficult for SFT to learn a good mapping between the question and the desired answer, especially with a small dataset. This paper introduces an alternative to SFT called Natural Language Feedback for Finetuning LLMs (LaFFi). LaFFi has LLMs directly predict the feedback they will receive from an annotator. We find that requiring such reflection can significantly improve the accuracy in in-domain question-answering tasks, providing a promising direction for the application of natural language feedback in the realm of SFT LLMs. Additional ablation studies show that the portion of human-annotated data in the annotated datasets affects the fine-tuning performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑