arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.24835cs.AIcs.PLcs.SE
收藏

模式驱动的大语言模型代码优化评估:分离编译器不可实现的语义推理

Pattern-Driven Evaluation of LLM Code Optimization: Isolating Compiler-Transcendent Semantic Reasoning

Georgia Institute of Technology(佐治亚理工学院);University of Virginia(弗吉尼亚大学)

Wesley Lu, Runzhou Liu, Hongru Du, Xinyao Yi

AI总结:

该研究提出PatternBench基准,结合运行时评估与AST分析,发现仅加速比无法反映LLM代码优化能力,需同时评估优化忠实性。

摘要:

现有基于大语言模型(LLM)的代码优化评估主要测量端到端加速比,掩盖了模型是否真正执行了预期优化的情况。对于编译型语言,这一问题尤为突出:观测到的性能提升可能反映了现代优化编译器已能实现的变换,而非模型自身的优化行为。我们提出PatternBench,这是一个用于研究编译器不可实现的优化(即经过激进编译器优化后仍保留的性能提升)的基准。PatternBench针对七类变换中的剩余优化机会,将运行时评估与基于抽象语法树(AST)的优化忠实性分析相结合,该分析用于验证生成程序中是否实现了预期优化。对前沿、开放权重及微调模型的实验表明,仅运行时加速比无法完整反映优化能力;优化忠实性与实际加速比密切相关,且微调可同时提升忠实性与优化有效性。这些结果提示,对基于LLM的代码优化的评估,不仅应测量性能结果,还应测量目标变换是否成功执行。

英文摘要:

Existing evaluations of LLM-based code optimization primarily measure end-to-end speedup, obscuring whether a model actually performs the intended optimization. For compiled languages, this is especially problematic: observed gains may reflect transformations already recoverable by modern optimizing compilers rather than optimization behavior attributable to the model itself. We present PatternBench, a benchmark for studying compiler-transcendent optimization, i.e., performance improvements that remain after aggressive compiler optimization. PatternBench targets residual optimization opportunities across seven classes of transformations and pairs runtime evaluation with an AST-based optimization faithfulness analysis that verifies whether the intended optimization is realized in the generated program. Experiments with frontier, open-weight, and fine-tuned models show that runtime speedup alone provides an incomplete account of optimization capability. Optimization faithfulness is strongly associated with realized speedup, and fine-tuning improves both faithfulness and optimization effectiveness. These results suggest that evaluation of LLM-based code optimization should measure not only performance outcomes, but also whether the target transformation was successfully executed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑