并非所有模型都适合专家卸载：混合专家模型的局部路由一致性
Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
AI总结：
本文提出两个指标衡量混合专家模型的局部路由一致性，发现局部路由一致性与局部负载平衡存在权衡，而全局负载平衡可共存，并揭示领域专用专家对路由一致性贡献更大。
摘要：
混合专家（MoE）通过在推理过程中稀疏激活专家，实现了大语言模型（LLMs）的高效扩展。为了在内存受限的设备上有效部署大型MoE模型，许多系统引入了*专家卸载*，将一部分专家缓存到高速内存中，而将其他专家留在慢速内存中，以在CPU上运行或按需加载。尽管一些研究已经利用了专家激活的局部性，即连续的token会激活相似的专家，但这种**局部路由一致性**的程度在不同模型中存在差异，仍缺乏深入研究。在本文中，我们提出了两个指标来衡量MoE模型的局部路由一致性：（1）**段路由最佳性能（SRP）**，评估一组固定专家对一段token需求的覆盖能力；（2）**段缓存最佳命中率（SCH）**，衡量在缓存限制下利用未来信息长度的缓存命中率。我们分析了20个具有不同大小和架构的MoE LLMs，并使用玩具模型验证了与局部路由一致性相关的关键因素。我们发现局部路由一致性与*局部*负载平衡之间存在显著的权衡，同时显示*全局*负载平衡可以与局部路由一致性共存。同时，像共享专家这样的设置会减少专家组合空间，可能导致局部路由一致性较低。我们进一步揭示，领域专门化的专家比词汇专门化的专家对路由一致性贡献更大，且大多数模型在缓存大小约为活跃专家两倍时在缓存有效性与效率之间取得平衡。这些发现为在不牺牲推理速度的情况下实现内存高效的MoE设计和部署铺平了道路。我们发布代码以复制实验，地址为https://github.com/ljcleo/moe-lrc。
英文摘要：
Mixture-of-Experts (MoE) enables efficient scaling of large language models (LLMs) with sparsely activated experts during inference. To effectively deploy large MoE models on memory-constrained devices, many systems introduce *expert offloading* that caches a subset of experts in fast memory, leaving others on slow memory to run on CPU or load on demand. While some research has exploited the locality of expert activations, where consecutive tokens activate similar experts, the degree of this **local routing consistency** varies across models and remains understudied. In this paper, we propose two metrics to measure local routing consistency of MoE models: (1) **Segment Routing Best Performance (SRP)**, which evaluates how well a fixed group of experts can cover the needs of a segment of tokens, and (2) **Segment Cache Best Hit Rate (SCH)**, which measures the hit rate of an expert cache utilizing a length of future information under a cache limit. We analyze 20 MoE LLMs with diverse sizes and architectures and use toy models to verify key factors related to local routing consistency. We find a strong trade-off between local routing consistency and *local* load balance, while showing that *global* load balance can coexist with local routing consistency. Meanwhile, settings like shared experts that decrease expert combination space can lead to low local routing consistency. We further reveal that domain-specialized experts contribute more to routing consistency than vocabulary-specialized ones, and that most models balance between cache effectiveness and efficiency with cache sizes approximately twice the active experts. These findings pave the way for memory-efficient MoE design and deployment without compromising inference speed. We publish the code for replicating experiments at https://github.com/ljcleo/moe-lrc .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
