ExpertFlow：通过预测专家缓存和令牌调度实现高效的混合专家推理
ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
AI总结：
本文提出ExpertFlow，通过预测专家缓存和令牌调度实现高效的混合专家推理，减少GPU内存使用达93.72%，提升推理吞吐量达10倍。
摘要：
稀疏混合专家（MoE）模型可通过仅激活少量专家来在相似计算下超越密集大语言模型。然而，堆叠许多专家模块会引入大量参数内存，使MoE模型难以在内存受限的环境中部署，如单GPU设备。卸载通过将不活跃的专家存储在CPU内存中并在需要时加载，但现有方法仍有限：静态缓存忽视输入依赖的路由，而提前训练单独模型预测专家使用的方法往往不准确或需要显著训练成本。我们提出ExpertFlow，一个轻量级的MoE推理系统，通过三个协调组件解决此路由依赖：1）基于Transformer的路由路径预测器，估计所有MoE层中的专家使用情况；2）令牌调度器，将具有相似预测路由的令牌分组以提高专家利用率；3）预测专家缓存，仅加载所需专家并在运行时纠正误预测。这些组件共同实现了高效的专家加载和执行，将GPU内存使用减少高达93.72%，并在单GPU上将推理吞吐量提高达10倍。
英文摘要：
Sparse Mixture-of-Experts (MoE) models can outperform dense large language models at similar computation by activating only a small set of experts per token. However, stacking many expert modules introduces substantial parameter memory, which makes MoE models difficult to deploy in memory-constrained environments such as single-GPU devices. Offloading alleviates this issue by storing inactive experts in CPU memory and loading them on demand, but existing methods remain limited: static caches disregard input-dependent routing, and methods that train separate models to predict expert usage ahead of time are often inaccurate or require significant training cost. We propose ExpertFlow, a lightweight MoE inference system that addresses this routing dependency through three coordinated components: 1) a transformer-based routing path predictor that estimates expert usage across all MoE layers in a single forward pass, 2) a token scheduler that groups tokens with similar predicted routes to improve expert utilization, and 3) a predictive expert cache that loads only the required experts while correcting mispredictions at runtime. Together, these components enable efficient expert loading and execution, reducing GPU memory usage by up to 93.72% and improving inference throughput by up to 10x over strong offloading baselines on a single GPU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
