arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.07463cs.PLcs.AIcs.SE
收藏

LLM生成代码的动态稳定性

Dynamic Stability of LLM-Generated Code

University of Luxembourg(卢森堡大学)

Prateek Rajput, Abdoul Aziz Bonkoungou, Yewei Song, Abdoul Kader Kabore, Iyiola E. Olatunji, Jacques Klein, Tegewende Bissyande

AI总结:

针对当前LLM代码生成评估忽视功能正确代码算法复杂度差异的局限,提出基于操作码分布的动态稳定性评估框架及SCTD、DCTD指标,实证揭示LLM正确代码存在显著算法差异及正确性与稳定性的权衡,为评估提供新方向。

摘要:

当前对LLM代码生成的评估强调功能正确性,却忽视了功能正确的解决方案在算法复杂度上可能存在显著差异的事实。例如,一个O(n²)的排序算法与O(n log n)的排序算法可能产生相似的输出,但在生产环境中会带来截然不同的性能成本。这种差异揭示了当前评估方法的一个关键局限性:它们未能捕捉正确解决方案之间的行为和性能多样性。为解决这一问题,我们引入了一个用于评估生成代码动态稳定性的原则性框架。我们提出了两个基于操作码分布的指标:静态规范轨迹差异(SCTD),用于捕捉不同生成解决方案之间的算法结构多样性;动态规范轨迹差异(DCTD),用于量化运行时行为差异。它们的比值——行为表达因子(BEF)——作为诊断信号:当BEF远小于1时,表明存在严重的运行时不稳定性;当BEF远大于1时,表明存在功能冗余。在BigOBench和CodeContests上的实证结果表明,即使在功能正确的输出中,最先进的LLM也表现出显著的算法差异。值得注意的是,提高采样温度会提高pass@1率,但会降低稳定性,揭示了一个未被认识到的权衡:在多样化的输出空间中搜索正确解决方案会在正确性和行为一致性之间引入“不稳定性惩罚”。我们的研究结果呼吁在代码生成中加入稳定性感知的目标,并为稳健的真实世界LLM评估提供带有渐近测试用例的新基准。

英文摘要:

Current evaluations of LLMs for code generation emphasize functional correctness, overlooking the fact that functionally correct solutions can differ significantly in algorithmic complexity. For instance, an $(O(n^2))$ versus $(O(n \log n))$ sorting algorithm may yield similar output but incur vastly different performance costs in production. This discrepancy reveals a critical limitation in current evaluation methods: they fail to capture the behavioral and performance diversity among correct solutions. To address this, we introduce a principled framework for evaluating the dynamic stability of generated code. We propose two metrics derived from opcode distributions: Static Canonical Trace Divergence (SCTD), which captures algorithmic structure diversity across generated solutions, and Dynamic Canonical Trace Divergence (DCTD), which quantifies runtime behavioral variance. Their ratio, the Behavioral Expression Factor (BEF), serves as a diagnostic signal: it indicates critical runtime instability when BEF $\ll$ 1 and functional redundancy when BEF $\gg$ 1. Empirical results on BigOBench and CodeContests show that state-of-the-art LLMs exhibit significant algorithmic variance even among functionally correct outputs. Notably, increasing sampling temperature improves pass@1 rates but degrades stability, revealing an unrecognized trade-off: searching for correct solutions in diverse output spaces introduces a "penalty of instability" between correctness and behavioral consistency. Our findings call for stability-aware objectives in code generation and new benchmarks with asymptotic test cases for robust, real-world LLM evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑