arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11715cs.LGcs.CL
收藏

DICE:扩散大语言模型在生成CUDA内核方面表现出色

DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels

Westlake University(西湖大学);Hong Kong University of Science and Technology(香港科技大学);Rochester Institute of Technology(罗切斯特理工学院)

Haolei Bai, Lingcheng Kong, Xueyi Chen, Jianmian Wang, Zhiqiang Tao, Huan Wang

AI总结:

提出CuKe数据集和BiC-RL训练框架,构建DICE系列扩散大语言模型(1.7B/4B/8B),在KernelBench上显著优于同类自回归和扩散模型,实现CUDA内核生成新SOTA。

摘要:

扩散大语言模型(dLLMs)因其并行生成令牌的能力,已成为自回归(AR)LLMs的有力替代方案。这一范式特别适用于代码生成,其中整体结构规划和非顺序优化至关重要。尽管有这种潜力,但针对CUDA内核生成定制dLLMs仍然具有挑战性,不仅因为高度专业化,还因为严重缺乏高质量的训练数据。为了解决这些挑战,我们构建了CuKe,一个针对高性能CUDA内核优化的增强监督微调数据集。在此基础上,我们提出了一个双阶段策划强化学习(BiC-RL)框架,包括CUDA内核填充阶段和端到端CUDA内核生成阶段。利用这一训练框架,我们推出了DICE,一系列专为CUDA内核生成设计的扩散大语言模型,涵盖1.7B、4B和8B三个参数规模。在KernelBench上的大量实验表明,DICE显著优于同等规模的自回归和扩散LLMs,为CUDA内核生成建立了新的最先进水平。

英文摘要:

Diffusion large language models (dLLMs) have emerged as a compelling alternative to autoregressive (AR) LLMs, owing to their capacity for parallel token generation. This paradigm is particularly well-suited for code generation, where holistic structural planning and non-sequential refinement are critical. Despite this potential, tailoring dLLMs for CUDA kernel generation remains challenging, obstructed not only by the high specialization but also by the severe lack of high-quality training data. To address these challenges, we construct CuKe, an augmented supervised fine-tuning dataset optimized for high-performance CUDA kernels. On top of it, we propose a bi-phase curated reinforcement learning (BiC-RL) framework consisting of a CUDA kernel infilling stage and an end-to-end CUDA kernel generation stage. Leveraging this training framework, we introduce DICE, a series of diffusion large language models designed for CUDA kernel generation, spanning three parameter scales, 1.7B, 4B, and 8B. Extensive experiments on KernelBench demonstrate that DICE significantly outperforms both autoregressive and diffusion LLMs of comparable scale, establishing a new state-of-the-art for CUDA kernel generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑