蒸馏对比解码：利用对比解码与蒸馏改进LLMs的推理能力
Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation
AI总结：
提出Distillation Contrastive Decoding（DCD）方法，结合对比提示与Dropout、Quantization等蒸馏技术，免除业余模型需求并降低内存，在GSM8K和StrategyQA等基准上显著提升LLMs推理性能。
摘要：
我们提出了一种名为Distillation Contrastive Decoding（DCD）的直接方法，以在推理期间增强大型语言模型（LLMs）的推理能力。与以往依赖较小的业余模型或分析隐藏状态差异的方法不同，DCD采用了Contrastive Chain-of-thought Prompting以及包括Dropout和Quantization在内的先进蒸馏技术。这种方法有效解决了Contrastive Decoding（CD）的局限性，CD通常需要同时使用专家模型和业余模型，从而增加了计算资源需求。通过将对比提示与蒸馏相结合，DCD免除了对业余模型的需求并降低了内存使用。我们的评估表明，DCD在一系列推理基准上显著提升了LLMs的性能，在GSM8K和StrategyQA数据集上超越了CD及现有方法。
英文摘要：
We propose a straightforward approach called Distillation Contrastive Decoding (DCD) to enhance the reasoning capabilities of Large Language Models (LLMs) during inference. In contrast to previous approaches that relied on smaller amateur models or analysis of hidden state differences, DCD employs Contrastive Chain-of-thought Prompting and advanced distillation techniques, including Dropout and Quantization. This approach effectively addresses the limitations of Contrastive Decoding (CD), which typically requires both an expert and an amateur model, thus increasing computational resource demands. By integrating contrastive prompts with distillation, DCD obviates the need for an amateur model and reduces memory usage. Our evaluations demonstrate that DCD significantly enhances LLM performance across a range of reasoning benchmarks, surpassing both CD and existing methods in the GSM8K and StrategyQA datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
