编译器辅助的推测采样用于加速异构边缘设备上的大语言模型推理
Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
AI总结：
本文提出编译器辅助的推测采样方法，通过分析成本模型优化异构边缘设备上的LLM推理性能，实现翻译任务加速1.68倍。
摘要：
在资源受限的边缘设备上部署LLM面临严重的延迟限制，特别是在实时应用中，延迟响应可能会影响安全或可用性。在许多缓解顺序逐词生成效率低下的方法中，推测解码（SD）已成为一种有前途的技术。然而，在边缘端实施SD面临两个主要挑战：（1）在不牺牲性能或可编程性的前提下，将SD集成到基于编译器的工作流中；（2）通过精心设计的分区策略利用现代SoC的异构计算资源。本文通过使用分析成本模型来探索异构硬件配置，并指导LLM子图的粗粒度分区，特别是在边缘典型短输入序列长度的情况下。成本模型预测推测采样和异构执行联合有益的时机，并在配备六核Cortex-A CPU和Mali GPU的边缘设备上得到验证，揭示了翻译任务的加速高达1.68倍，接近分析预期。
英文摘要：
LLM deployment on resource-constrained edge devices faces severe latency constraints, particularly in real-time applications where delayed responses can compromise safety or usability. Among many approaches to mitigate the inefficiencies of sequential token-by-token generation, Speculative Decoding (SD) has emerged as a promising technique. However, SD at the edge is hindered by two major challenges: (1) integrating SD into a compiler-based workflow without sacrificing performance or programmability, and (2) exploiting the heterogeneous compute resources of modern SoCs through carefully designed partitioning strategies. This work addresses these challenges by using an analytical cost model that explores heterogeneous hardware configurations and guides coarse-grained partitioning of LLM subgraphs, particularly with edge-typical short input sequence lengths. The cost model predicts when speculative sampling and heterogeneous execution are jointly beneficial and is validated on an edge device featuring a hexacore Cortex-A CPU and a Mali GPU, revealing up to 1.68$\times$ speedup for translation tasks, closely matching analytic expectations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
