$\mathbb{USCD}$：通过不确定性感知的选择性对比解码提升大语言模型的代码生成能力
$\mathbb{USCD}$: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding
AI总结：
本文提出仅推理、即插即用的 USCD 机制，利用预测不确定性选择性抵消 lame prompt 产生的输出噪声，在多个代码 LLM 和基准上平均提升 pass@1 达16.59%。
摘要：
大语言模型（LLM）在代码生成方面已展现出卓越能力。然而，幻觉（例如输出噪声）的影响使得 LLM 尤其难以在一次生成中产出高质量代码。在本工作中，我们提出一种简单有效的不确定性感知选择性对比解码（uncertainty-aware selective contrastive decoding，$\mathbb{USCD}$）机制，以提升 LLM 一次通过代码生成的质量，并降低输出噪声的影响。具体而言，我们首先精心设计了一个负向提示（称为 lame prompt），通过从标准少样本提示中移除输入–输出示例来输出噪声。我们的初步研究表明，token 分布不确定性与输出噪声之间的 Jensen–Shannon 散度（JS divergence）相对较低（约为 $0.25$），表明二者具有高度相关性。随后，我们根据标准提示所产生预测分布的不确定性，选择性地消除由 lame prompt 诱发的输出噪声。值得注意的是，我们提出的即插即用机制是一种仅用于推理的方法，具有良好的灵活性。在多个 LLM（即 Inocder-6b、CodeLlama-7b、WizardCoder-15b、StarCoder 和 Llama2-7b）上，针对广泛使用的基准（例如 HumanEval、MBPP 和 MultiPL-E）进行的大量实验表明，我们提出的 USCD 显著提升了一次通过代码生成性能，平均 pass@$1$ 分数提高了 16.59%。我们将在 GitHub 上发布代码和数据。
英文摘要：
Large language models (LLMs) have shown remarkable capabilities in code generation. However, the effects of hallucinations (e.g., output noise) make it particularly challenging for LLMs to generate high-quality code in one pass. In this work, we propose a simple and effective \textbf{u}ncertainty-aware \textbf{s}elective \textbf{c}ontrastive \textbf{d}ecoding ($\mathbb{USCD}$) mechanism to improve the quality of one-pass code generation in LLMs and reduce the impact of output noise. To be specific, we first elaborately designed a negative prompt (namely lame prompt) to output noise by removing input-output examples from the standard few-shot prompt. Our preliminary study shows that the Jensen-Shannon divergence (JS divergence) between token distribution uncertainty and the output noise is relatively low (approximately $0.25$), indicating their high relevance. Then, we selectively eliminate output noise induced by lame prompts based on the uncertainty of the prediction distribution from the standard prompt. Notably, our proposed plug-and-play mechanism is an inference-only method, enjoying appealing flexibility. Extensive experiments on widely used benchmarks, e.g., HumanEval, MBPP, and MultiPL-E, upon several LLMs (i.e., Inocder-6b, CodeLlama-7b, WizardCoder-15b, StarCoder, and Llama2-7b), demonstrate that our proposed USCD significantly improves one-pass code generation, with an average \textit{pass@$1$} scores increase of 16.59\%. We will release code and data on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
