AutoMegaKernel:用于自我重定目标超内核合成的静态检查代理框架
AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis
AI总结:
提出AutoMegaKernel系统,将Llama模型编译为单个持久CUDA内核,通过静态调度验证器确保无死锁和无竞争,自动生成10种模型正确超内核,并在NVIDIA推理卡上以W8A16精度超越cuBLAS bf16。
摘要:
AutoMegaKernel (AMK) 将HuggingFace Llama系列模型编译成一个持久的协作CUDA内核,该内核在一次启动中运行整个前向传播,无需为每个模型手写CUDA代码。其贡献在于系统本身,而非原始速度。一个冻结的调度IR验证器通过静态图检查(非机械化证明)静态地认证无死锁和无竞争,因此不安全的智能体提议调度在启动前被拒绝:在7,160个对抗性调度(6,091个不安全)中,它实现了零误接受,并接受了所有360个实际底层实现。同一源代码可重定目标至sm_80/sm_90/sm_120,从单一代码库自动为10个支持模型中的全部生成正确的超内核,并在真实的SmolLM2-135M检查点上重现HuggingFace贪婪解码逐token匹配(困惑度差异2.5e-7)。一个无人值守、智能体驱动的自动研究循环在其自身基线之上自我改进超内核(1.25-1.72倍)。一个搜索发现的int8 (W8A16) 超内核在NVIDIA数据中心推理集群的batch-1解码中击败了CUDA图化的cuBLAS bf16:L4最高1.33倍,当前一代L40S 1.25-1.27倍,A10G大规模最高1.08倍,以及消费级RTX 5090 1.19-1.23倍。排序并非带宽的简单函数(864 GB/s的L40S击败了600 GB/s的A10G);分界线是推理级与训练级。AMK在高带宽训练级A100/H100上落后于cuBLAS,其中框架定位了跨SM同步瓶颈;我们坦率地报告了这一差距。这是解码位置0处精度不对称(W8A16 vs bf16)的比较;最大的真实检查点是TinyLlama-1.1B。代码和框架:https://github.com/RightNow-AI/AutoMegaKernel
英文摘要:
AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The contribution is the system, not raw speed. A frozen schedule-IR validator statically certifies deadlock-freedom and race-freedom via static graph checks (not a mechanized proof), so an unsafe agent-proposed schedule is rejected before launch: across 7,160 adversarial schedules (6,091 unsafe) it had zero false-accepts and accepted all 360 real lowerings. The same source retargets sm_80/sm_90/sm_120 from one codebase, auto-generates correct megakernels for 10 of 10 supported models, and on a real SmolLM2-135M checkpoint reproduces HuggingFace greedy decode token-for-token (perplexity match 2.5e-7). An unattended, agent-drivable autoresearch loop self-improves the megakernel over its own baseline (1.25-1.72x). A search-found int8 (W8A16) megakernel beats CUDA-graphed cuBLAS bf16 at batch-1 decode across NVIDIA's datacenter inference fleet: L4 up to 1.33x, the current-gen L40S 1.25-1.27x, A10G up to 1.08x at scale, and the consumer RTX 5090 1.19-1.23x. The ordering is not a clean function of bandwidth (the 864 GB/s L40S beats the 600 GB/s A10G); the divide is inference-class vs training-class. AMK trails cuBLAS on the high-bandwidth training-class A100/H100, where the harness localizes the cross-SM-sync bottleneck; we report the gap plainly. This is a precision-asymmetric (W8A16 vs bf16) comparison at decode position 0; the largest real checkpoint is TinyLlama-1.1B. Code and the harness: https://github.com/RightNow-AI/AutoMegaKernel
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。