K-Search: 通过共进化内在世界模型生成LLM内核
K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model
AI总结:
K-Search通过共进化内在世界模型生成LLM内核,显著优于现有进化搜索方法,实现高效内核优化。
摘要:
优化GPU内核对于高效现代机器学习系统至关重要,但因其设计因素的复杂相互作用和硬件的快速演变而仍然具有挑战性。现有的自动化方法通常将大型语言模型(LLMs)仅视为启发式引导进化循环中的随机代码生成器。这些方法在处理需要协调、多步骤结构变换的复杂内核时常常遇到困难,因为它们缺乏明确的规划能力,且经常由于不高效或错误的中间实现而丢弃有前途的策略。为此,我们提出了通过共进化世界模型的搜索方法,并基于此方法构建了K-Search。通过将静态搜索启发式替换为共进化世界模型,我们的框架利用LLMs的先验领域知识来引导搜索,主动探索优化空间。这种方法明确地将高层算法规划与低层程序实例化解耦,使系统能够在非单调优化路径中导航,同时对临时实现缺陷具有鲁棒性。我们评估了K-Search在FlashInfer中的多样化复杂内核上,包括GQA、MLA和MoE内核。我们的结果表明,K-Search在复杂MoE内核上实现了高达14.3倍的改进,平均改进2.10倍,优于最先进的进化搜索方法。在GPUMode TriMul任务中,K-Search在H100上达到最先进的性能,达到1030us,超过了先前的进化和人工设计的解决方案。
英文摘要:
Optimizing GPU kernels is critical for efficient modern machine learning systems yet remains challenging due to the complex interplay of design factors and rapid hardware evolution. Existing automated approaches typically treat Large Language Models (LLMs) merely as stochastic code generators within heuristic-guided evolutionary loops. These methods often struggle with complex kernels requiring coordinated, multi-step structural transformations, as they lack explicit planning capabilities and frequently discard promising strategies due to inefficient or incorrect intermediate implementations. To address this, we propose Search via Co-Evolving World Model and build K-Search based on this method. By replacing static search heuristics with a co-evolving world model, our framework leverages LLMs' prior domain knowledge to guide the search, actively exploring the optimization space. This approach explicitly decouples high-level algorithmic planning from low-level program instantiation, enabling the system to navigate non-monotonic optimization paths while remaining resilient to temporary implementation defects. We evaluate K-Search on diverse, complex kernels from FlashInfer, including GQA, MLA, and MoE kernels. Our results show that K-Search significantly outperforms state-of-the-art evolutionary search methods, achieving an average 2.10x improvement and up to a 14.3x gain on complex MoE kernels. On the GPUMode TriMul task, K-Search achieves state-of-the-art performance on H100, reaching 1030us and surpassing both prior evolution and human-designed solutions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。