arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11192cs.CLcs.AI
收藏

我如果能用我的语言说话会学得更好:理解通过微调大型语言模型与LLM生成响应的优越性能

I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses

University of Adelaide(阿德莱德大学);University of Technology Sydney(悉尼科技大学)

Xuan Ren, Biao Wu, Lingqiao Liu

AI总结:

本文研究了LLM生成响应在微调大型语言模型时的优越性能,发现LLM对自身生成响应的熟悉性是提升学习效果的关键因素。

摘要:

本文探讨了一个引人入胜的观察:用LLM生成的响应来微调大型语言模型(LLM)通常比用人类生成的响应获得更好的结果,特别是在推理任务中。我们进行了深入的研究以了解为何会发生这种情况。与人们普遍认为这些实例是由于LLM生成内容更详细的观点相反,我们的研究发现另一个有贡献的因素:LLM本质上更熟悉LLM生成的响应。这种熟悉性通过微调前的低困惑度得到证明。我们设计了一系列实验以了解“熟悉性”的影响,我们的结论表明这种“熟悉性”显著影响学习性能。使用LLM生成的响应进行训练不仅提高了性能,而且有助于在微调特定任务后保持模型在其他推理任务中的能力。

英文摘要:

This paper explores an intriguing observation: fine-tuning a large language model (LLM) with responses generated by a LLM often yields better results than using responses generated by humans, particularly in reasoning tasks. We conduct an in-depth investigation to understand why this occurs. Contrary to the common belief that these instances is due to the more detailed nature of LLM-generated content, our study identifies another contributing factor: an LLM is inherently more "familiar" with LLM generated responses. This familiarity is evidenced by lower perplexity before fine-tuning. We design a series of experiments to understand the impact of the "familiarity" and our conclusion reveals that this "familiarity" significantly impacts learning performance. Training with LLM-generated responses not only enhances performance but also helps maintain the model's capabilities in other reasoning tasks after fine-tuning on a specific task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑