EpiCoDe：通过外推与对比解码提升模型在训练之外的表现
EpiCoDe: Boosting Model Performance Beyond Training with Extrapolation and Contrastive Decoding
AI总结：
本文提出 EpiCoDe 方法，在数据稀缺场景下无需额外训练，通过模型外推与对比解码提升 LLM 性能，并在多任务多模型实验中验证其显著且稳健的优势。
摘要：
大型语言模型（LLMs）的卓越性能在很大程度上依赖于丰富的高质量训练数据。然而，获取标注数据的高昂成本往往使模型难以获得处理下游任务的能力。在本文中，我们提出了一种新方法 EpiCoDe，在数据稀缺场景下无需额外训练即可提升模型性能。我们首先利用模型外推，用较弱的版本来增强微调后的模型，然后采用对比解码，通过比较外推模型与原始微调模型给出的 logit 分数，进一步减少预测误差。在四个不同 LLMs 上的三项任务实验表明，EpiCoDe 始终优于现有方法，并取得了显著且稳健的提升。我们还提出了一个新的理论框架，以揭示数据稀缺场景下对比解码背后的机制，这进一步帮助我们更好地理解 EpiCoDe 的有效性。
英文摘要：
The remarkable performance of Large language models (LLMs) relies heavily on the availability of abundant high-quality training data. However, the high cost of acquiring annotated data often prevents models from obtaining capabilities to tackle downstream tasks. In this paper, we introduce a novel method, EpiCoDe that boosts model performance in data-scarcity scenarios without extra training. We first employ model extrapolation to enhance a finetuned model with its inferior version, and then adopt contrastive decoding to further reduce predicted errors, by comparing the logit scores given by the extrapolated and the vanilla finetuned model. Experiments across three tasks over four different LLMs show that EpiCoDe consistently outperforms existing methods with significant and robust improvement. We also propose a new theoretical framework to reveal the mechanism behind contrastive decoding in data-scarcity scenarios, which further helps us better understand the effectiveness of EpiCoDe.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
