傻瓜们确信；聪明人则怀疑：探索LLM在代码补全中的信心
The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completion
AI总结：
本文研究LLM在代码补全中的信心，通过测量不同编程语言、模型和数据集的代码困惑度，发现强类型语言比动态类型语言困惑度更低，脚本语言困惑度更高，Java困惑度较低。
摘要：
代码补全任务是根据上下文提供缺失的令牌。它能提升开发者生产力并提供强大的代码发现工具。在大语言模型（LLM）浪潮中，代码补全已被各种在代码上微调的LLM（代码LLM）所探索。代码LLM的性能可以通过下游和内在指标评估。下游指标通常用于评估模型的实用性，但可能不可靠且需要复杂的计算和领域知识。相反，像困惑度、熵和互信息等内在指标，用于衡量模型的置信度或不确定性，是简单、通用且适用于所有LLM和任务的。这些指标可以作为LLM生成代码的功能正确性和幻觉风险的代理。受此启发，我们通过使用各种LLM测量不同编程语言、模型和数据集的代码困惑度，评估LLM在生成代码时的信心，并分析了2254个文件来自881个GitHub项目的样本。我们发现强类型语言的困惑度低于动态类型语言。脚本语言也表现出更高的困惑度。Shell在困惑度上普遍较高，而Java则较低。代码困惑度取决于所使用的LLM；在固定模型下，相对语言级别的排名在评估数据集中相对稳定。尽管代码注释通常会增加困惑度，但基于困惑度的语言排名几乎不受其存在影响。LLM研究人员、开发者和用户可以根据语言、模型选择和代码特性如何影响模型置信度，利用我们的发现来评估LLM在特定软件项目中的代码补全的益处和适用性。
英文摘要：
Code completion entails the task of providing missing tokens given a surrounding context. It can boost developer productivity and serve as a code discovery tool. Code completion has recently been approached with Large Language Models (LLMs) fine-tuned on code (code LLMs). The performance of code LLMs can be assessed with downstream and intrinsic metrics. Downstream metrics are usually employed to evaluate the practical utility of a model, but can be unreliable and require complex calculations and domain-specific knowledge. In contrast, intrinsic metrics such as perplexity, entropy, and mutual information, which measure model confidence or uncertainty, are simple, versatile, and universal across LLMs and tasks, and have been proposed as proxies for functional correctness and hallucination risk in LLM-generated code. Motivated by this, we evaluate the confidence of LLMs when generating code by measuring code perplexity across programming languages, models, and datasets using various LLMs, and a sample of 2254 files from 881 GitHub projects. We find that strongly-typed languages exhibit lower perplexity than dynamically typed languages. Scripting languages also demonstrate higher perplexity. Shell appears universally high in perplexity, whereas Java appears low. Code perplexity depends on the employed LLM; under a fixed model, relative language-level rankings are largely stable across evaluation corpora. Although code comments modestly increase perplexity, the language ranking based on perplexity is barely affected by their presence. LLM researchers, developers, and users can use our findings to assess the suitability of LLM-based code completion in specific software projects based on how language, model choice, and code characteristics impact model confidence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
