DyMoE:基于混合精度量化动态专家调度的高效边缘MoE推理
DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge
AI总结:
本文提出DyMoE框架,通过动态混合精度量化优化边缘设备上MoE模型的推理效率,显著降低TTFT和TPOT,实现高精度实时推理。
摘要:
尽管MoE模型具有计算效率,但多专家架构中固有的内存足迹和I/O开销对资源受限边缘平台的实时推理构成了严峻挑战。尽管现有静态方法在延迟-精度权衡上表现僵化,我们观察到专家重要性高度倾斜且深度依赖。受这些见解启发,我们提出DyMoE,一种动态混合精度量化框架,旨在实现高性能边缘推理。利用对专家重要性倾斜和深度依赖敏感性的洞察,DyMoE引入:(1)重要性感知优先级,用于在运行时动态量化专家;(2)深度自适应调度,以在关键层保留语义完整性;(3)前瞻预取,以重叠I/O停滞。在商业边缘硬件上的实验结果表明,DyMoE将Time-to-First-Token(TTFT)减少了3.44x-22.7x,相比最先进的卸载基线,Time-Per-Output-Token(TPOT)速度提升了高达14.58x,实现了在资源受限边缘设备上的实时、高精度MoE推理。
英文摘要:
Despite the computational efficiency of MoE models, the excessive memory footprint and I/O overhead inherent in multi-expert architectures pose formidable challenges for real-time inference on resource-constrained edge platforms. While existing static methods struggle with a rigid latency-accuracy trade-off, we observe that expert importance is highly skewed and depth-dependent. Motivated by these insights, we propose DyMoE, a dynamic mixed-precision quantization framework designed for high-performance edge inference. Leveraging insights into expert importance skewness and depth-dependent sensitivity, DyMoE introduces: (1) importance-aware prioritization to dynamically quantize experts at runtime; (2) depth-adaptive scheduling to preserve semantic integrity in critical layers; and (3) look-ahead prefetching to overlap I/O stalls. Experimental results on commercial edge hardware show that DyMoE reduces Time-to-First-Token (TTFT) by 3.44x-22.7x and up to a 14.58x speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines, enabling real-time, accuracy-preserving MoE inference on resource-constrained edge devices.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。