指令微调(Instruction Tuning,IT)是使用指令-响应对训练大语言模型(large language models,LLMs)的过程,已成为将基础预训练LLMs转化为开放域对话智能体的主流方法。尽管IT已取得显著成功并得到广泛应用,但其局限性和不足之处仍未得到充分探究。本文通过严谨的实验,深入分析LLMs在IT过程中发生的变化,揭示了IT的多种局限性。具体而言,我们发现:(1)IT无法增强LLMs的知识或技能。LoRA微调仅局限于学习回复的起始和风格标记,而全参数微调会导致知识退化。(2)从源自知识型来源的IT数据集中复制回复模式,会导致回复质量下降。(3)全参数微调会从IT数据集中概念相似的实例中不准确地借用标记来生成回复,从而增加幻觉。(4)现有主流的IT改进方法,其性能并未优于简单的LoRA微调模型。我们的研究结果表明,在开源数据集上,仅依靠预训练知识生成的回复,始终优于从IT中学习任何形式新知识的模型生成的回复。我们希望本文揭示的洞见与挑战能为相关方向的未来研究提供启发。
英文摘要:
Instruction Tuning (IT), the process of training large language models (LLMs) using instruction-response pairs, has emerged as the predominant method for transforming base pre-trained LLMs into open-domain conversational agents. While IT has achieved notable success and widespread adoption, its limitations and shortcomings remain underexplored. In this paper, through rigorous experiments and an in-depth analysis of the changes LLMs undergo through IT, we reveal various limitations of IT. In particular, we show that (1) IT fails to enhance knowledge or skills in LLMs. LoRA fine-tuning is limited to learning response initiation and style tokens, and full-parameter fine-tuning leads to knowledge degradation. (2) Copying response patterns from IT datasets derived from knowledgeable sources leads to a decline in response quality. (3) Full-parameter fine-tuning increases hallucination by inaccurately borrowing tokens from conceptually similar instances in the IT dataset for generating responses. (4) Popular methods to improve IT do not lead to performance improvements over a simple LoRA fine-tuned model. Our findings reveal that responses generated solely from pre-trained knowledge consistently outperform responses by models that learn any form of new knowledge from IT on open-source datasets. We hope the insights and challenges revealed in this paper inspire future work in related directions.
Demystifying Instruction Mixing for Fine-tuning Large Language Models
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);LibrAI;Monash University(莫纳什大学);University of British Columbia(不列颠哥伦比亚大学);The University of Melbourne(墨尔本大学)
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory(协调科学实验室);Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Mathematics(数学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
An Empirical Study of Instruction-tuning Large Language Models in Chinese
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);APUS AiLMe Lab(APUS AiLMe实验室)
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Department of Linguistics and Translation, City University of Hong Kong(香港城市大学语言学及翻译学系)
Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning
School of Cyber Science and Engineering Nanjing University of Science and Technology China(南京理工大学信息科学与工程学院);Meituan China(美团);Zhipu.AI China(智谱AI);Institute of Information Engineering Chinese Academy of Sciences China(中国科学院信息工程研究所)
Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning
DataArc Tech Ltd.(DataArc科技有限公司);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));IDEA Research, International Digital Economy Academy(IDEA研究所,国际数字经济学院);Hithink RoyalFlush Information Network Co., Ltd(Hithink RoyalFlush信息网络有限公司)
COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
Shenzhen Institute of Advanced Technology, CAS(中国科学院深圳先进技术研究院);M-A-P;Institute of Automation, CAS(中国科学院自动化研究所);Peking University(北京大学);University of Science and Technology of China(中国科学技术大学);HKUST(香港科技大学);University of Manchester(曼彻斯特大学);University of Waterloo(滑铁卢大学);Vector Institute(向量研究所)
LongForm: Effective Instruction Tuning with Reverse Instructions
Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning(慕尼黑机器学习中心);Language Technology Lab, University of Cambridge(剑桥大学语言技术实验室)
On the Effect of Instruction Tuning Loss on Generalization
Dept. of Electrical Engineering(电气工程系);Indian Institute of Technology Delhi(印度理工学院德里);Media and Data Science Research(媒体与数据科学研究);Adobe Inc., India(Adobe公司,印度)
University of Southern California(南加州大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);The Pennsylvania State University(宾夕法尼亚州立大学);University of California, Davis(加利福尼亚大学戴维斯分校)