KernelSight-LM: 一种内核级LLM推理模拟器
KernelSight-LM: A Kernel-Level LLM Inference Simulator
AI总结:
提出KernelSight-LM,一种细粒度推理模拟器,通过分解服务步骤为屋顶线内核模型、通信模型和主机开销模型,并集成前缀缓存和连续批处理,实现跨GPU代际或少量目标数据下的精确延迟预测,支持硬件/软件协同设计。
摘要:
随着大语言模型(LLM)进入生产服务,从业者必须快速评估不同硬件、模型和服务参数下的推理性能,以满足成本和延迟目标。然而,LLM的端到端行为将服务层策略与底层GPU内核执行及快速演进的架构耦合,导致难以泛化的缓慢、特定部署的基准测试。我们提出KernelSight-LM,一种细粒度推理模拟器,建模令牌级执行并生成内核级延迟分解。它将每个服务步骤分解为带有学习效率项的屋顶线内核模型、通信模型和主机开销模型,通过离散事件调度器组合,同时捕获前缀缓存和连续批处理等机制。KernelSight-LM提供两种预测层级,以目标GPU数据换取精度。跨代层级不使用目标GPU测量,仅依赖硬件规格和先前分析GPU的内核微基准,在未见过的GPU代际上预测每内核延迟误差为12.1%,比屋顶线基线(22.0%)提升1.8倍。第二个目标测量层级在目标GPU上增加一次模型无关的内核微基准扫描,将每内核误差降至3.8%,比可比基线(27.7%)提升7.3倍。两个层级所需的目标GPU数据远少于它们扩展的先前系统。在我们的模拟器中,这些预测在跨代层级上,六个模型家族的端到端中位数(p50)误差分别为15.4%(TTFT)、12.8%(TPOT)和3.0%(吞吐量),在目标测量层级上分别为14.3%、6.2%和2.7%,匹配专用分析工具,同时收集更少的设备上数据。除了预测,其内核级瓶颈分解支持硬件/软件协同设计和容量规划。
英文摘要:
As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets. However, the end-to-end behavior of LLMs couples serving-layer policies with low-level GPU kernel execution and rapidly evolving architectures, forcing slow, deployment-specific benchmarking that is hard to generalize. We present KernelSight-LM, a fine-grained inference simulator that models token-level execution and produces kernel-level latency breakdowns. It decomposes each serving step into a roofline kernel model with a learned efficiency term, a communication model, and a host-overhead model, composed through a discrete-event scheduler that also captures mechanisms like prefix caching and continuous batching. KernelSight-LM offers two prediction tiers that trade target-GPU data for accuracy. The cross-generation tier uses no target-GPU measurements, only hardware specifications and kernel microbenchmarks from previously profiled GPUs, and predicts per-kernel latency on an unseen GPU generation to 12.1% error, a 1.8x improvement over the roofline baseline (22.0%). A second target-measured tier adds one model-agnostic kernel-microbenchmark sweep on the target GPU, sharpening per-kernel error to 3.8%, a 7.3x improvement over a comparable baseline (27.7%). Both tiers require far less target-GPU data than the prior systems they extend. In our simulator, these predictions yield end-to-end median (p50) errors across six model families of 15.4%, 12.8%, and 3.0% (TTFT, TPOT, throughput) in the cross-generation tier and 14.3%, 6.2%, and 2.7% in the target-measured tier, matching dedicated profiling tools while collecting far less on-device data. Beyond prediction, its kernel-level bottleneck breakdowns support hardware/software co-design and capacity planning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。