MCC-KD：多思维链一致性知识蒸馏
MCC-KD: Multi-CoT Consistent Knowledge Distillation
AI总结：
提出MCC-KD方法，通过生成多思维链并最小化双向KL散度增强一致性，有效蒸馏大模型推理能力至小模型，在数学与常识推理基准上表现优越且泛化性强。
摘要：
大型语言模型（LLMs）通过思维链（CoT）提示在复杂推理中展现了卓越的能力。近年来，将这种推理能力从大型语言模型迁移到较小模型的兴趣日益增长。然而，在推理依据上实现多样性与一致性并存是一个挑战。本文聚焦于增强这两个方面，并提出了多思维链一致性知识蒸馏（MCC-KD）方法，以高效地蒸馏推理能力。在MCC-KD中，我们为每个问题生成多个推理依据，并通过最小化答案分布之间的双向KL散度来强制相应预测之间的一致性。我们在数学推理和常识推理基准上，使用不同的模型架构（LLaMA/FlanT5）和不同的模型规模（3B/7B/11B/13B）研究了MCC-KD的有效性。实证结果不仅证实了MCC-KD在分布内数据集上的优越性能，还突出了其在分布外数据集上的强大泛化能力。
英文摘要：
Large language models (LLMs) have showcased remarkable capabilities in complex reasoning through chain of thought (CoT) prompting. Recently, there has been a growing interest in transferring these reasoning abilities from LLMs to smaller models. However, achieving both the diversity and consistency in rationales presents a challenge. In this paper, we focus on enhancing these two aspects and propose Multi-CoT Consistent Knowledge Distillation (MCC-KD) to efficiently distill the reasoning capabilities. In MCC-KD, we generate multiple rationales for each question and enforce consistency among the corresponding predictions by minimizing the bidirectional KL-divergence between the answer distributions. We investigate the effectiveness of MCC-KD with different model architectures (LLaMA/FlanT5) and various model scales (3B/7B/11B/13B) on both mathematical reasoning and commonsense reasoning benchmarks. The empirical results not only confirm MCC-KD's superior performance on in-distribution datasets but also highlight its robust generalization ability on out-of-distribution datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
