代码不仅仅是文本:代码生成的不确定性估计
Code Is More Than Text: Uncertainty Estimation for Code Generation
AI总结:
针对代码生成中错误程序的可靠性问题,提出基于词法、算法和功能三个正交轴的不确定性估计方法,在五个代码LLM上将AUROC提升8.1个百分点。
摘要:
大型语言模型(LLMs)越来越多地被部署为代码生成器,其中静默错误的程序会带来真实的安全和可靠性风险。可靠的不确定性估计(UE)对于选择性预测、人在回路审查和下游智能体决策至关重要。然而,现有的大多数代码UE方法继承自自然语言(NL)生成,忽略了使代码独特的属性。我们认为代码在三个方面与NL不同:单个错误标记可能破坏整个程序(标记脆弱性);算法意图和具体实现可能独立不一致(意图-代码差距);程序可以被执行(可执行性)。我们将这些属性实例化为三个正交的不确定性轴:词法(Top-K标记熵)、算法(伪代码一致性)和功能(行为一致性)。在五个代码LLM上,我们的三轴集成将平均AUROC从最强NL衍生基线的0.696提高到0.776(+8.1点)。值得注意的是,在Qwen3-14B上,我们的单次Top-K标记熵匹配了最强多次基线,同时成本降低超过3倍;在各模型上,它仍然是一个有竞争力的低成本信号。这些结果表明,代码UE需要特定于代码的设计,而不是直接移植NL方法。
英文摘要:
Large language models (LLMs) are increasingly deployed as code generators, where silently wrong programs pose real safety and reliability risks. Reliable uncertainty estimation (UE) is essential for selective prediction, human-in-the-loop review, and downstream agentic decisions. Yet most existing code UE methods are inherited from natural language (NL) generation and ignore properties that make code distinct. We argue that code differs from NL in three ways: a single wrong token can break an entire program (token fragility); algorithmic intent and concrete implementation can disagree independently (intent-code gap); and programs can be executed (executability). We instantiate these properties as three orthogonal uncertainty axes: lexical (Top-K token entropy), algorithmic (pseudo-code consistency), and functional (behavioral consistency). Across five code LLMs, our three-axis ensemble improves average AUROC from 0.696 for the strongest NL-derived baseline to 0.776 (+8.1 points). Notably, on Qwen3-14B, our single-pass Top-K token entropy matches the strongest multi-pass baseline while being over 3x cheaper; across models, it remains a competitive low-cost signal. These results suggest that code UE deserves code-specific design rather than direct NL ports.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。