arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.01930cs.LG
收藏

高效的大语言模型上下文蒸馏

Efficient LLM Context Distillation

School of Computer Science, Georgia Institute of Technology(计算机科学学院,佐治亚理工学院)

Rajesh Upadhayaya, Manish Raj Osti, Zachary Smith, Chritopher Kottmyer

AI总结:

本文研究上下文蒸馏方法,通过将任务示例内部化来高效适配大语言模型,实验表明其域内外准确率可与上下文学习相当,且计算需求更低,适合小数据集场景。

摘要:

大语言模型(LLMs)在多种任务中展现出卓越能力,但针对特定应用通常需要有针对性的适配。已有多种方法被提出以促进这种适配,包括少样本微调、上下文学习以及上下文蒸馏。本文专门研究上下文蒸馏——一种通过将任务特定示例内部化来扩展其效用,从而扩充可用于模型推理的示例集合的方法。我们对上下文蒸馏与上下文学习(ICL)和少样本微调(FT)进行了比较分析,旨在确定上下文蒸馏在使用极少量上下文示例适配模型方面的有效性。利用来自Mobach的匹配数据集,我们的实验采用了不同规模的OPT模型。结果表明,上下文蒸馏能够有效适配模型,学生模型在域内和域外准确率上与上下文学习达到了相当水平。尽管上下文蒸馏在域外泛化上优于ICL,但它并未达到FT的性能水平。然而,减少的数据集规模和计算需求使上下文蒸馏成为一种可行的替代方案,尤其是对于较小的数据集。总体而言,本研究将上下文蒸馏呈现为一种高效且有力的方法,用于将大语言模型定制到特定任务。

英文摘要:

Large Language Models (LLMs) demonstrate proficiency across diverse tasks but often require targeted adaptations for specific applications. Various methods have been proposed to facilitate this adaptation, including fewshot fine-tuning, in-context learning, and context distillation. This paper specifically investigates context distillation a method that extends the utility of task-specific examples by internalizing them, thus augmenting the example set accessible for model inference. We conduct a comparative analysis of context distillation with in-context learning (ICL) and few-shot fine-tuning (FT), aiming to ascertain the efficacy of context distillation in adapting models using minimal in-context examples. Employing matched datasets from Mobach, our experiments leverage OPT models of various sizes. The results indicate that context distillation effectively adapts models, with student models attaining comparable in-domain and out-of-domain accuracies to in-context learning. Although context distillation surpasses ICL in out-of-domain generalization, it does not achieve the performance levels of FT. However, the reduced dataset size and computational demands position context distillation as a viable alternative, especially for smaller datasets. Overall, this study presents context distillation as an efficient and potent method for customizing LLMs to specific tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑