DynaKV：在智能手机上实现准确高效的长序列大语言模型解码
DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones
AI总结：
针对智能手机长序列LLM解码受KVCache内存限制的问题，提出自适应KVCache管理方法DynaKV，通过无迁移集群适配等三项技术，实现1.38倍准确性提升和1.47倍速度提升，且适用于其他长上下文场景。
摘要：
随着对类人推理、多轮对话和长文本响应的需求增长，大语言模型（LLMs）越来越需要支持高效的长序列解码。然而，由于DRAM容量有限，智能手机上的长序列LLM解码受限于键值缓存（KVCache），其内存占用随序列长度线性增长。基于检索的方法通过将KVCache卸载到闪存并通过基于集群的索引检索查询相关条目来缓解DRAM压力。但随着解码推进，KVCache分布变化导致静态或局部集群更新逐渐错位，排除必要条目或获取冗余条目。这些问题因智能手机在带宽、IOPS和内存容量上的特定限制而进一步加剧。我们提出DynaKV，首个针对智能手机长序列解码联合解决准确性和效率的自适应KVCache管理方法。DynaKV集成三项关键技术：（1）无迁移集群适配，在检索时自适应拆分集群而不产生额外传输；（2）以连续性为中心的闪存管理，将相关条目和集群共定位并采用双头布局实现高效更新；（3）内存高效缓存设计，跨DRAM和闪存虚拟化缓存空间并扩展替换策略以匹配集群级访问模式。评估表明，与现有最优方案相比，DynaKV提升了检索准确性并降低了端到端延迟，实现平均1.38倍的准确性提升和1.47倍的速度提升。此外，DynaKV的见解可自然扩展到其他长上下文工作负载和多层内存架构，凸显其更广泛的适用性。
英文摘要：
As the demand for human-like reasoning, multi-turn dialogues, and long-form responses grows, large language models (LLMs) are increasingly expected to support efficient and effective long-sequence decoding. However, due to limited DRAM capacity, long-seuqence LLM decoding on smartphones is constrained by the key-value cache (KVCache), whose memory footprint increases linearly with sequence length. Retrieval-based methods mitigate DRAM pressure by offloading KVCache to flash and retrieving query-relevant entries through cluster-based indexing. Unfortunately, as decoding progresses, KVCache distribution shifts render static or local cluster updates progressively misaligned, excluding essential entries or fetching redundant ones. These issues are further exacerbated by smartphone-specific limitations in bandwidth, IOPS, and memory capacity. We propose DynaKV, the first adaptive KVCache management approach that jointly addresses accuracy and efficiency for long-sequence decoding on smartphones. DynaKV integrates three key techniques: (1) Migration-Free Cluster Adaptation, which adaptively splits clusters during retrieval without incurring additional transfers; (2) Continuity-Centric Flash Management, which co-locates correlated entries and clusters and employs a dual-head layout for efficient updates; and (3) Memory-Efficient Cache Design, which virtualizes cache space across DRAM and flash and extends replacement policies to align with cluster-level access patterns. Evaluations demonstrate that DynaKV improves retrieval accuracy and reduces end-to-end latency compared to state-of-the-art solutions, achieving average gains of $1.38\times$ in accuracy and $1.47\times$ speedups. Furthermore, the insights of DynaKV naturally extend to other long-context workloads and multi-tier memory hierarchies, underscoring its broader applicability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
