arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05119cs.CLcs.AI
收藏

深入探究指令微调(Instruction Tuning)的局限性

A Closer Look at the Limitations of Instruction Tuning

University of Maryland(马里兰大学);Adobe(奥多比公司);NVIDIA(英伟达公司)

Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha

AI总结:

该研究通过实验分析指令微调(IT)对大语言模型的影响,揭示了IT无法增强模型知识、全参数微调会引发知识退化与幻觉、主流改进方法未优于简单LoRA微调等局限性,指出仅靠预训练知识生成的回复表现更优。

摘要:

指令微调(Instruction Tuning,IT)是使用指令-响应对训练大语言模型(large language models,LLMs)的过程,已成为将基础预训练LLMs转化为开放域对话智能体的主流方法。尽管IT已取得显著成功并得到广泛应用,但其局限性和不足之处仍未得到充分探究。本文通过严谨的实验,深入分析LLMs在IT过程中发生的变化,揭示了IT的多种局限性。具体而言,我们发现:(1)IT无法增强LLMs的知识或技能。LoRA微调仅局限于学习回复的起始和风格标记,而全参数微调会导致知识退化。(2)从源自知识型来源的IT数据集中复制回复模式,会导致回复质量下降。(3)全参数微调会从IT数据集中概念相似的实例中不准确地借用标记来生成回复,从而增加幻觉。(4)现有主流的IT改进方法,其性能并未优于简单的LoRA微调模型。我们的研究结果表明,在开源数据集上,仅依靠预训练知识生成的回复,始终优于从IT中学习任何形式新知识的模型生成的回复。我们希望本文揭示的洞见与挑战能为相关方向的未来研究提供启发。

英文摘要:

Instruction Tuning (IT), the process of training large language models (LLMs) using instruction-response pairs, has emerged as the predominant method for transforming base pre-trained LLMs into open-domain conversational agents. While IT has achieved notable success and widespread adoption, its limitations and shortcomings remain underexplored. In this paper, through rigorous experiments and an in-depth analysis of the changes LLMs undergo through IT, we reveal various limitations of IT. In particular, we show that (1) IT fails to enhance knowledge or skills in LLMs. LoRA fine-tuning is limited to learning response initiation and style tokens, and full-parameter fine-tuning leads to knowledge degradation. (2) Copying response patterns from IT datasets derived from knowledgeable sources leads to a decline in response quality. (3) Full-parameter fine-tuning increases hallucination by inaccurately borrowing tokens from conceptually similar instances in the IT dataset for generating responses. (4) Popular methods to improve IT do not lead to performance improvements over a simple LoRA fine-tuned model. Our findings reveal that responses generated solely from pre-trained knowledge consistently outperform responses by models that learn any form of new knowledge from IT on open-source datasets. We hope the insights and challenges revealed in this paper inspire future work in related directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑