arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.02236cs.LGcs.AI
收藏

CUDABench: 评估用于文本到CUDA生成的LLM

CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

Shanghai Jiao Tong University(上海交通大学)

Jiace Zhu, Wentao Chen, Qi Fan, Zhixing Ren, Junying Wu, Xing Zhe Chai, Chotiwit Rungrueangwutthinon, Yehan Ma, An Zou

AI总结:

CUDABench是一个评估LLM文本到CUDA生成能力的基准测试,通过编译正确性、功能一致性和性能指标评估LLM在生成GPU程序中的表现。

摘要:

最近的研究已经证明了大型语言模型(LLMs)在生成GPU内核方面的潜力。当前的基准测试侧重于将高级语言转换为CUDA,忽视了更通用且更具挑战性的文本到CUDA生成任务。此外,鉴于GPU编程的硬件特定性和性能关键特性,准确评估LLM生成的GPU程序性能是困难的。在本工作中,我们介绍了CUDABench,一个全面的基准测试,用于评估LLM的文本到CUDA能力。首先,我们构建了CUDABench-Set,它涵盖了广泛的应用领域,包括人工智能、科学计算和数据分析等,涵盖了广度、深度和难度的评估空间。此外,我们提出了CUDABench-Score和生成验证流水线,评估(1)编译正确性,(2)通过执行验证的功能一致性,以及(3)一个新的基于屋顶线的度量标准,性能分数。对最先进的LLM进行基准测试揭示了文本到CUDA的洞察发现和挑战,例如高编译成功率与低功能正确性的显著不匹配,缺乏领域特定的算法知识,以及GPU硬件资源利用的次优化。我们的基准测试可在https://github.com/CUDA-Bench/CUDABench上获得。

英文摘要:

Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPU Kernels. Current benchmarks focus on the translation of high-level languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-specific and performance-critical features of GPU programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial. In this work, we introduce CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs. First, we construct CUDABench-Set, which covers Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics, etc. Furthermore, we propose CUDABench-Score and Generative Verification Pipeline that assess (1) compilation correctness, (2) functional consistency through execution-based verification, and (3) a novel roofline-based metric, Performance-Score. Benchmarking state-of-the-art LLMs reveals insightful findings and challenges of text-to-CUDA, such as a notable mismatch between high compilation success rates and low functional correctness, a lack of domain-specific algorithmic knowledge, and suboptimal utilization of GPU hardware resources. Our benchmark is available at https://github.com/CUDA-Bench/CUDABench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑