QiMeng-Kernel:基于宏思维微编码的LLM高性能GPU内核生成范式
QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation
AI总结:
QiMeng-Kernel通过宏思维微编码范式,结合强化学习和通用LLM,实现高性能GPU内核生成,准确率和效率均优于现有方法。
摘要:
开发高性能GPU内核对于AI和科学计算至关重要,但因其依赖专家手工编写和较差的可移植性而仍然具有挑战性。尽管LLM在自动化方面具有潜力,但通用和微调的LLM都面临两个根本且相互冲突的限制:正确性和效率。关键原因是现有的基于LLM的方法直接生成整个优化的低级程序,需要探索一个包含优化策略和实现代码的极广空间。为了解决探索不可行空间的挑战,我们提出了宏思维微编码(MTMC),一种受人类专家分阶段优化策略启发的分层框架。它将优化策略与实现细节解耦,通过高层策略确保效率,通过低层实现确保正确性。具体而言,宏思维利用强化学习指导轻量级LLM高效探索和学习最大化硬件利用率的语义优化策略。微编码利用通用LLM逐步实现宏思维提出的分步优化方案,避免全内核生成错误。共同作用下,它们有效导航广阔的优化空间和复杂的实现细节,使LLM能够实现高性能GPU内核生成。在广泛采用的基准测试中,全面结果展示了MTMC在GPU内核生成的准确性和运行时间上的优越性能。在KernelBench上,MTMC在Level 1-2和Level 3上分别达到近100%和70%的准确率,比最先进的通用和领域微调的LLM高出50%以上,比LLM快7.3倍,比专家优化的PyTorch Eager内核快2.2倍。在更具挑战性的TritonBench上,MTMC达到最高59.64%的准确率和34倍的速度提升。
英文摘要:
Developing high-performance GPU kernels is critical for AI and scientific computing, but remains challenging due to its reliance on expert crafting and poor portability. While LLMs offer promise for automation, both general-purpose and finetuned LLMs suffer from two fundamental and conflicting limitations: correctness and efficiency. The key reason is that existing LLM-based approaches directly generate the entire optimized low-level programs, requiring exploration of an extremely vast space encompassing both optimization policies and implementation codes. To address the challenge of exploring an intractable space, we propose Macro Thinking Micro Coding (MTMC), a hierarchical framework inspired by the staged optimization strategy of human experts. It decouples optimization strategy from implementation details, ensuring efficiency through high-level strategy and correctness through low-level implementation. Specifically, Macro Thinking employs reinforcement learning to guide lightweight LLMs in efficiently exploring and learning semantic optimization strategies that maximize hardware utilization. Micro Coding leverages general-purpose LLMs to incrementally implement the stepwise optimization proposals from Macro Thinking, avoiding full-kernel generation errors. Together, they effectively navigate the vast optimization space and intricate implementation details, enabling LLMs for high-performance GPU kernel generation. Comprehensive results on widely adopted benchmarks demonstrate the superior performance of MTMC on GPU kernel generation in both accuracy and running time. On KernelBench, MTMC achieves near 100% and 70% accuracy at Levels 1-2 and 3, over 50% than SOTA general-purpose and domain-finetuned LLMs, with up to 7.3x speedup over LLMs, and 2.2x over expert-optimized PyTorch Eager kernels. On the more challenging TritonBench, MTMC attains up to 59.64% accuracy and 34x speedup.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。