EdgeFlow: 针对移动设备上LLM的快速冷启动
EdgeFlow: Fast Cold Starts for LLMs on Mobile Devices
AI总结：
本文提出EdgeFlow框架，通过自适应调整LLM参数精度，减少移动设备上LLM冷启动延迟，实验表明其比现有框架提升4.07倍。
摘要：
在移动设备上部署大型语言模型（LLMs）是一项新兴趋势，旨在实现数据隐私和离线访问。现代移动神经处理单元（NPUs）使这种部署变得越来越可行。然而，现有移动LLM推理框架由于不可避免的冷启动问题（即当模型不在设备内存中时启动LLM推理）导致启动延迟高。本文识别出移动LLM冷启动的关键瓶颈是闪存带宽在不重要的模型参数上的浪费。我们设计了EdgeFlow，一种移动LLM推理框架，通过自适应调整LLM参数的精度来缓解冷启动问题。具体而言，EdgeFlow利用1）一种NPU感知的自适应量化算法，根据其重要性和NPU约束，以更细的粒度分配不同精度给权重；2）一种SIMD友好的打包格式，加速将各种精度的权重转换为固定大小的NPU原生数据类型；3）一种协同的细粒度流水线，以细粒度和动态的方式协调CPU和NPU计算。实验结果表明，与三种最先进的移动LLM推理框架（llama.cpp、MNN和llm.npu）相比，EdgeFlow在可比的模型精度下，冷启动延迟减少了高达4.07倍。
英文摘要：
Deploying large language models (LLMs) on mobile devices is an emerging trend to enable data privacy and offline accessibility of LLM applications. Modern mobile neural processing units (NPUs) make such deployment increasingly feasible. However, existing mobile LLM inference frameworks suffer from high start-up latency due to their inevitable cold starts, i.e., launching LLM inferences when the model is not hosted in device memory. In this paper, we identify the key bottleneck of mobile LLM cold starts as the waste of flash bandwidth on unimportant model parameters. We design EdgeFlow, a mobile LLM inference framework that mitigates the cold start issue by adaptively adjusting the precisions of LLM parameters. Specifically, EdgeFlow leverages 1) an NPU-aware adaptive quantization algorithm that assigns different precisions to weights in a finer granularity according to their importance and NPU constraints, 2) an SIMD-friendly packing format that accelerates the transformation of various-precision weights into fixed-sized NPU-native data types, and 3) a synergistic granular pipeline that coordinates CPU and NPU computation in a fine-grained and dynamic manner. Experimental results show that EdgeFlow reduces cold-start latency by up to 4.07x compared with three state-of-the-art mobile LLM inference frameworks, i.e., llama.cpp, MNN, and llm.npu, under comparable model accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
