arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.04023cs.SEcs.AI
收藏

CodegenBench: 大型语言模型能否跨架构编写高效代码?

CodegenBench: Can LLMs Write Efficient Code Across Architectures?

Sun Yat-sen University(中山大学);National Supercomputing Center in Wuxi(无锡国家超级计算机中心);National Supercomputing Center in Shenzhen(深圳国家超级计算机中心);University of the Chinese Academy of Sciences(中国科学院大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院)

Jie Li, Wenzhao Wu, Junqi Hu, Qinrui Zheng, Bowen Wu, Juepeng Zheng, Yutong Lu, Haohuan Fu

AI总结:

提出CodegenBench基准测试,评估LLMs在x86_64、Sunway和Kunpeng三种架构上生成高效并行代码的能力,发现其在通用架构上表现良好,但在领域特定架构上性能显著下降。

摘要:

尽管大型语言模型(LLMs)在通用编程和GPU加速环境(如PyTorch、CUDA)的代码生成任务中得到了广泛评估,但它们在面向CPU的高性能计算(HPC)跨不同架构上的能力仍未充分探索。为填补这一空白,我们引入了CodegenBench,这是一个全面的基准测试套件,旨在评估在三种不同硬件平台(x86_64、Sunway和Kunpeng)上生成高效并行代码的能力。我们的基准测试包含106个标准基本线性代数子程序(BLAS)例程,建立了一个基础基线,以及20个针对每个独特超级计算架构(LeetSunway和LeetKunpeng)改编的专门计算内核。我们的广泛评估揭示,虽然最先进的LLMs能够为像x86_64这样的普遍架构生成优化代码,但在公共文档和训练数据有限的领域特定架构上,它们表现出显著的性能下降,突显了跨平台泛化的关键局限性。此外,我们对影响代码质量的因素(如实现长度和任务复杂度)的分析表明,当前LLMs在需要简洁代码片段的中等难度问题上最为有效。我们开源了我们的数据集和自动化评估基础设施,以促进未来在LLM驱动的高性能代码生成方面的研究。资源可在https://anonymous.4open.science/r/CodegenBench-EDE1/和https://anonymous.4open.science/r/CodegenBenchDataset-2551/获取。

英文摘要:

While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments (e.g., PyTorch, CUDA), their capabilities in CPU-oriented high-performance computing (HPC) across diverse architectures remain underexplored. To bridge this gap, we introduce CodegenBench, a comprehensive benchmark suite designed to evaluate the generation of efficient parallel code across three distinct hardware platforms: x86_64, Sunway, and Kunpeng. Our benchmark comprises 106 standard Basic Linear Algebra Subprograms (BLAS) routines establishing a fundamental baseline, alongside 20 specialized computational kernels adapted for each of the unique supercomputing architectures (LeetSunway and LeetKunpeng). Our extensive evaluation reveals that while state-of-the-art LLMs can generate optimized code for ubiquitous architectures like x86_64, they exhibit significant performance degradation on domain-specific architectures with limited public documentation and training data, highlighting critical limitations in cross-platform generalization. Furthermore, our analysis of factors influencing code quality such as implementation length and task complexity indicates that current LLMs are most effective for moderately difficult problems requiring concise code snippets. We open-source our dataset and automated evaluation infrastructure to facilitate future research in LLM-driven high-performance code generation. The resources are available at https://anonymous.4open.science/r/CodegenBench-EDE1/ and https://anonymous.4open.science/r/CodegenBenchDataset-2551.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑