arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.00700cs.SEcs.AIcs.PL
收藏

LLVM-Bench:面向LLVM编译器问题解决的大语言模型基准测试与推进

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang, Junjie Chen

AI总结:

提出LLVM-Bench基准和LLVM-Gym平台,评估大语言模型在LLVM编译器问题解决上的表现,并设计集成方法LLVM-Ens提升解决率至21.99%。

摘要:

LLVM是一个广泛使用的编译器基础设施,其规模和复杂性使得问题解决劳动密集且具有挑战性。尽管大语言模型(LLMs)最近在问题解决方面取得了显著成功,但它们在复杂的系统级LLVM编译器上的有效性尚未得到充分探索。为了解决这一差距,我们引入了LLVM-Bench,这是第一个用于LLVM问题解决的大规模基准测试,包含从LLVM项目收集的423个真实世界、经过验证的任务。我们进一步开发了LLVM-Gym,一个可扩展的评估平台,自动化问题复现、补丁应用、编译器构建和测试执行。使用LLVM-Bench和LLVM-Gym,我们对四个代表性LLMs、六种检索配置和三个智能体进行了全面研究。我们的结果表明,当前基于LLM的问题解决技术在LLVM-Bench上仍然有限,补丁无效和构建失败是主要的失败模式。我们进一步揭示了不同LLMs和智能体之间的强互补性,从而提出了LLVM-Ens,一种轻量级集成方法,通过整合不同技术生成的补丁来扩展补丁空间,过滤错误和冗余候选,并识别最有希望的解决方案。我们的结果表明,LLVM-Ens实现了高达21.99%的解决率,进一步改进了LLVM问题解决。

英文摘要:

LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑