模式驱动的大语言模型代码优化评估:分离编译器不可实现的语义推理
Pattern-Driven Evaluation of LLM Code Optimization: Isolating Compiler-Transcendent Semantic Reasoning
AI总结:
该研究提出PatternBench基准,结合运行时评估与AST分析,发现仅加速比无法反映LLM代码优化能力,需同时评估优化忠实性。
摘要:
现有基于大语言模型(LLM)的代码优化评估主要测量端到端加速比,掩盖了模型是否真正执行了预期优化的情况。对于编译型语言,这一问题尤为突出:观测到的性能提升可能反映了现代优化编译器已能实现的变换,而非模型自身的优化行为。我们提出PatternBench,这是一个用于研究编译器不可实现的优化(即经过激进编译器优化后仍保留的性能提升)的基准。PatternBench针对七类变换中的剩余优化机会,将运行时评估与基于抽象语法树(AST)的优化忠实性分析相结合,该分析用于验证生成程序中是否实现了预期优化。对前沿、开放权重及微调模型的实验表明,仅运行时加速比无法完整反映优化能力;优化忠实性与实际加速比密切相关,且微调可同时提升忠实性与优化有效性。这些结果提示,对基于LLM的代码优化的评估,不仅应测量性能结果,还应测量目标变换是否成功执行。
英文摘要:
Existing evaluations of LLM-based code optimization primarily measure end-to-end speedup, obscuring whether a model actually performs the intended optimization. For compiled languages, this is especially problematic: observed gains may reflect transformations already recoverable by modern optimizing compilers rather than optimization behavior attributable to the model itself. We present PatternBench, a benchmark for studying compiler-transcendent optimization, i.e., performance improvements that remain after aggressive compiler optimization. PatternBench targets residual optimization opportunities across seven classes of transformations and pairs runtime evaluation with an AST-based optimization faithfulness analysis that verifies whether the intended optimization is realized in the generated program. Experiments with frontier, open-weight, and fine-tuned models show that runtime speedup alone provides an incomplete account of optimization capability. Optimization faithfulness is strongly associated with realized speedup, and fine-tuning improves both faithfulness and optimization effectiveness. These results suggest that evaluation of LLM-based code optimization should measure not only performance outcomes, but also whether the target transformation was successfully executed.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。