arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.15417cs.AI
收藏

SpecFit:卸载型MoE推理中并非所有草稿令牌的成本都相同

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Beihang University(北京航空航天大学);Monash University(莫纳什大学);The Hong Kong University of Science and Technology(香港科技大学)

Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu, Hailong Yang

AI总结:

针对卸载型MoE推理中推测解码的H2D专家传输成本问题,提出SpecFit通过选择最大化预测吞吐量的草稿子树提升解码速度,在CNN/DailyMail和GSM8K数据集上较EAGLE系列模型实现显著加速。

摘要:

混合专家(MoE)模型每个令牌仅激活少数专家,但当它们的专家权重被卸载到主机内存时,主机到设备(H2D)的专家传输会进入解码的关键路径。推测解码在一次前向传播中验证多个草稿令牌,但在MoE模型中,每个验证的令牌会激活自身的专家,因此更大的草稿树会传输更多专家。由于这种H2D成本,推测解码每个验证周期会提交多个令牌,但有时甚至比自回归解码更慢。基于此观察,我们提出SpecFit,它选择要验证的草稿子树以最大化预测吞吐量,而非接受长度。SpecFit从草稿路径概率估计预期提交令牌数,并从节点数和子树激活的专家的每层并集预测周期时间;该并集是单调次模覆盖函数,因此每个节点仅为其添加的专家付费。对祖先闭合子树的精确搜索返回预测吞吐量最高的子树;在每个草稿层级运行后,相同搜索还会决定何时停止草稿,未修改的目标模型验证结果。在CNN/DailyMail摘要任务和带有Qwen3-30B-A3B的GSM8K子集上,SpecFit在相同专家缓存下比EAGLE-3解码速度快21.6%和7.0%,是普通EAGLE-3解码速度的1.97倍和1.72倍。代码可在该https URL获取。

英文摘要:

Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at https://github.com/Chlience/SpecFit.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑