arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.07013cs.CL
收藏

黑盒大型语言模型的知识蒸馏

Knowledge Distillation of Black-Box Large Language Models

School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Alibaba Group(阿里巴巴集团)

Hongzhan Chen, Ruijun Chen, Yuqi Yi, Xiaojun Quan, Chenliang Li, Ming Yan, Ji Zhang

AI总结:

针对黑盒大模型知识蒸馏中内部状态不可访问的问题,提出使用代理模型的Proxy-KD方法,实验表明其性能优于传统白盒蒸馏,为从先进LLMs蒸馏知识提供了新途径。

摘要:

鉴于像GPT-4这样的专有大型语言模型(LLMs)的卓越性能,近期研究越来越关注通过从这些强大但黑盒的教师模型进行知识蒸馏(KD)来提升较小模型的能力。虽然利用这些教师模型的高质量输出是有利的,但其内部状态不可访问常常限制了有效的知识迁移。为了克服这一限制,我们引入了Proxy-KD,一种新颖的方法,使用代理模型促进从黑盒LLMs向较小模型的高效知识迁移。我们的实验表明,Proxy-KD不仅提升了从黑盒教师模型进行知识蒸馏的性能,而且超越了传统的白盒知识蒸馏技术。这种方法为从先进LLMs中蒸馏知识提供了一条引人注目的新途径。

英文摘要:

Given the exceptional performance of proprietary large language models (LLMs) like GPT-4, recent research has increasingly focused on boosting the capabilities of smaller models through knowledge distillation (KD) from these powerful yet black-box teachers. While leveraging the high-quality outputs of these teachers is advantageous, the inaccessibility of their internal states often limits effective knowledge transfer. To overcome this limitation, we introduce Proxy-KD, a novel method that uses a proxy model to facilitate the efficient transfer of knowledge from black-box LLMs to smaller models. Our experiments show that Proxy-KD not only enhances the performance of KD from black-box teacher models but also surpasses traditional white-box KD techniques.~This approach presents a compelling new avenue for distilling knowledge from advanced LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑