ODMA:面向LLM服务的LPDDR类加速器按需内存分配策略
ODMA: On-Demand Memory Allocation Strategy for LLM Serving on LPDDR-Class Accelerators
AI总结:
ODMA针对LPDDR类加速器的随机访问带宽限制,提出按需内存分配策略,通过动态调整内存桶边界和安全池提升KV缓存利用率和吞吐量。
摘要:
现有内存管理技术严重阻碍了在带宽受限的加速器上高效的大语言模型服务。静态预分配保持内存连续性,但因最坏情况分配导致显著开销。相反,细粒度分页缓解了这一开销,但依赖HBM的高随机访问容忍度,使其不适合LPDDR系统,其中非顺序访问迅速降低带宽。此外,先前工作通常假设静态分布和HBM特性,因此无法解决LPDDR硬件固有的碎片化和带宽限制问题。我们提出了ODMA,一种专为随机访问受限的加速器(如Cambricon MLU系列)设计的按需内存分配策略。ODMA通过解决生产工作负载中的两个关键限制:(i)分布漂移导致静态桶边界失效,以及(ii)在重尾请求模式下的性能脆弱性。ODMA集成了轻量级长度预测器、自适应桶分区和回退安全池。桶边界通过在线直方图动态重新校准以最大化利用率,而安全池确保预测错误的鲁棒性。在Alpaca和Google-NQ基准测试中,ODMA将S3的预测准确率从98.60%提升至99.55%,从82.68%提升至93.36%。在Cambricon MLU370-X4加速器上部署DeepSeek-R1-Distill-Qwen-7B时,ODMA将KV缓存利用率提高高达19.25%(绝对值),并比静态基线提高23-27%的吞吐量(TPS),验证了预测驱动的连续分配在LPDDR类设备中的有效性。
英文摘要:
Existing memory management techniques severely hinder efficient Large Language Model serving on accelerators constrained by poor random-access bandwidth.While static pre-allocation preserves memory contiguity,it incurs significant overhead due to worst-case provisioning.Conversely,fine-grained paging mitigates this overhead but relies on HBM's high random-access tolerance, making it unsuitable for LPDDR systems where non-sequential access rapidly degrades bandwidth. Furthermore, prior works typically assume static distributions and HBM characteristics, thereby failing to resolve the critical fragmentation and bandwidth constraints inherent to LPDDR hardware. We present ODMA, an on-demand memory allocation strategy tailored for random-access-constrained accelerators, such as the Cambricon MLU series.ODMA advances generation-length prediction by addressing two critical limitations in production workloads: (i) distribution drift that invalidates static bucket boundaries, and (ii) performance fragility under heavy-tailed request patterns. ODMA integrates a lightweight length predictor with adaptive bucket partitioning and a fallback safety pool. Bucket boundaries are dynamically recalibrated via online histograms to maximize utilization, while the safety pool ensures robustness against prediction errors. On Alpaca and Google-NQ benchmarks, ODMA improves S3's prediction accuracy from 98.60% to 99.55% and 82.68% to 93.36%, respectively. Deployment with DeepSeek-R1-Distill-Qwen-7B on Cambricon MLU370-X4 accelerators demonstrates that ODMA increases KV-cache utilization by up to 19.25% (absolute) and throughput (TPS) by 23-27% over static baselines, validating the efficacy of predictor-driven contiguous allocation for LPDDR-class devices.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。