arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.14752cs.CLcs.LG
收藏

TritonBench:基准测试大型语言模型生成 Triton 算子的能力

TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators

Tianjin University(天津大学);Harbin Institute of Technology(哈尔滨工业大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tsinghua University(清华大学)

Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, Haojie Wang, Jianrong Wang, Xu Han, Zhiyuan Liu, Maosong Sun

AI总结:

本文提出首个 Triton 算子生成综合基准 TritonBench,通过 GitHub 真实算子与 PyTorch 对齐算子双通道,评估代码 LLM 的正确性和 GPU 性能。

摘要:

Triton 是一种类似 Python 的高级语言,旨在构建高效的 GPU kernel,凭借可移植性、灵活性和易用性被深度学习框架广泛采用。然而,Triton 开发者在编程和并行优化方面仍需进行大量试错。尽管大型语言模型(LLM)在传统代码生成方面取得了进展,但由于不了解 Triton 的规范以及 GPU 编程的复杂性,这些模型难以生成准确且经过性能优化的 Triton 代码。更关键的是,目前迫切需要针对 Triton 的系统性评估。在本工作中,我们推出 TritonBench,这是首个面向 Triton 算子生成的综合基准。TritonBench 包含两个评估通道:一组精心整理的来自 GitHub 的 184 个真实世界算子,以及一批与 PyTorch 接口对齐的算子。不同于优先关注功能正确性的传统代码基准,TritonBench 还在契合行业应用、被广泛部署的 GPU 上剖析效率性能。研究表明,当前最先进的代码 LLM 难以生成高效的 Triton 算子,凸显出高性能代码生成方面的显著差距。TritonBench 将在 https://github.com/thunlp/TritonBench 提供。

英文摘要:

Triton, a high-level Python-like language designed for building efficient GPU kernels, is widely adopted in deep learning frameworks due to its portability, flexibility, and accessibility. However, programming and parallel optimization still require considerable trial and error from Triton developers. Despite advances in large language models (LLMs) for conventional code generation, these models struggle to generate accurate, performance-optimized Triton code, as they lack awareness of its specifications and the complexities of GPU programming. More critically, there is an urgent need for systematic evaluations tailored to Triton. In this work, we introduce TritonBench, the first comprehensive benchmark for Triton operator generation. TritonBench features two evaluation channels: a curated set of 184 real-world operators from GitHub and a collection of operators aligned with PyTorch interfaces. Unlike conventional code benchmarks prioritizing functional correctness, TritonBench also profiles efficiency performance on widely deployed GPUs aligned with industry applications. Our study reveals that current state-of-the-art code LLMs struggle to generate efficient Triton operators, highlighting a significant gap in high-performance code generation. TritonBench will be available at https://github.com/thunlp/TritonBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑