arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02684cs.SE
收藏

编码代理能否实现错过的编译器优化?在LLVM窥孔优化上评估大语言模型代理

Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations

University of Waterloo(滑铁卢大学)

Hongxu Xu, Chunhao Liao, Xintong Zhou, Chengnian Sun

AI总结:

研究编码代理能否开发编译器优化,引入PeepholeBench框架,其任务来自LLVM的实际窥孔优化问题,通过对比衡量编码代理与人工修复,发现正确性和收益性间存在矛盾及主要失败模式,为编码代理提供了现实且具挑战性的基准。

摘要:

基于大语言模型构建的编码代理如今能够修补大型现实世界代码库,但它们能否开发编译器优化仍是个悬而未决的问题。为研究此问题,我们引入了PeepholeBench,这是一个评估框架,其任务由针对LLVM的InstCombine pass报告的实际错过的窥孔优化构建而成。由于错过的窥孔优化通常通过小的局部补丁来修复,它们为编码代理提供了一个范围明确但要求苛刻的测试平台:正确的修复需要对程序语义进行严格推理以及熟悉优化器特定的约定。PeepholeBench从21个已解决的LLVM问题和19个合并的拉取请求(PR)中派生其任务,仅向代理提供每个修复之前存在的问题上下文,并评估生成的补丁的正确性和收益性。使用PeepholeBench,我们在修复LLVM的InstCombine pass中错过的窥孔优化方面对最先进的编码代理进行基准测试,将它们的补丁与相应的人工编写的修复进行比较。我们观察到正确性和收益性之间存在矛盾,并且没有代理在这两个维度上同时与人类开发者匹配。主要的失败模式是过度狭窄的转换和对LLVM特定机制的误用,这些错误是现有测试套件很少暴露的。这些结果共同确立了PeepholeBench作为编码代理的一个现实且具有挑战性的基准,并为构建能够更可靠地协助编译器优化开发的代理提出了未来方向。

英文摘要:

Coding agents built on large language models are now capable of patching sizable real-world codebases, yet whether they can develop compiler optimizations remains an open question. To study this question, we introduce PeepholeBench, an evaluation framework whose tasks are constructed from real-world missed peephole optimizations reported against LLVM's InstCombine pass. Since missed peephole optimizations are typically fixed with small, localized patches, they offer a well-scoped but demanding testbed for coding agents: a correct fix demands rigorous reasoning about program semantics along with familiarity with optimizer-specific conventions. PeepholeBench derives its tasks from 21 resolved LLVM issues and 19 merged pull requests (PRs), supplies agents with only the issue context that existed before each fix, and assesses the resulting patches for both correctness and profitability. With PeepholeBench, we benchmark state-of-the-art coding agents on fixing missed peephole optimizations in LLVM's InstCombine pass, measuring their patches against the corresponding human-written fixes. We observe a tension between correctness and profitability, and no agent matches human developers on both dimensions at once. The dominant failure modes are overly narrow transformations and misuse of LLVM-specific mechanisms, errors that existing test suites rarely expose. Together, these results establish PeepholeBench as a realistic and challenging benchmark for coding agents, and suggest future directions for building agents that can more dependably assist compiler optimization development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑