混合蒸馏帮助更小的语言模型更好地推理
Mixed Distillation Helps Smaller Language Model Better Reasoning
AI总结：
本文提出混合蒸馏（MD）框架，融合大语言模型的思维程序与思维链能力并蒸馏到较小模型，显著提升其单路径和多路径推理性能，在SVAMP上LLaMA2-7B和CodeLlama-7B分别超过GPT-3.5-Turbo 2.5%和3.5%。
摘要：
尽管大语言模型（LLMs）在近期自然语言处理（NLP）任务中表现卓越，但由于实际应用中高计算和内存需求，其部署面临重大挑战。近期研究聚焦于通过从大语言模型进行知识蒸馏来增强较小模型，取得了有前景的结果。然而，这些模型往往难以匹敌大语言模型的性能，尤其是在需要推理的任务中。本文提出混合蒸馏（Mixed Distillation，MD）框架，该框架利用大语言模型中思维程序（Program of Thought，PoT）和思维链（Chain of Thought，CoT）能力的优势，结合多种提示技术并将这些能力蒸馏到较小模型中。实验结果表明，MD显著增强了较小模型在多种任务中的单路径和多路径推理能力。在推理任务的准确性和泛化性方面，由其生成的模型超过两个单独蒸馏模型的综合性能。值得注意的是，使用MD的LLaMA2-7B和CodeLlama-7B在SVAMP基准上分别取得了84.5%和85.5%的显著提升，分别超过GPT-3.5-Turbo 2.5%和3.5%。
英文摘要：
While large language models (LLMs) have demonstrated exceptional performance in recent natural language processing (NLP) tasks, their deployment poses substantial challenges due to high computational and memory demands in real-world applications. Recent studies have focused on enhancing smaller models through knowledge distillation from LLMs, yielding promising results. However, these models often struggle to match the performance of LLMs, especially in tasks that require reasoning. In this work, we introduce Mixed Distillation (MD) framework, which capitalizes on the strengths of Program of Thought (PoT) and Chain of Thought (CoT) capabilities within LLMs, combining multiple prompting techniques and distilling these capabilities into smaller models. Our experimental results show that MD significantly enhances the single-path and multi-path reasoning ability of smaller models in various tasks. In terms of accuracy and generality of reasoning tasks, the model generated by it exceeds the comprehensive performance of two individually distilled models. Notably, LLaMA2-7B and CodeLlama-7B using MD achieved remarkable improvements of (84.5%) and (85.5%), respectively, outperforming GPT-3.5-Turbo by (2.5%) and (3.5%), on the SVAMP benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
