无需重新训练是否足够？高效MoE压缩中路由校准的必要性
Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
AI总结：
本文提出路由知识蒸馏方法，通过轻量级路由校准实现无需重新训练的MoE压缩，有效解决路由-专家不匹配问题，提升细粒度MoE的性能表现。
摘要：
混合专家（MoE）模型能够高效地扩展容量，但其庞大的参数足迹在部署时造成了内存瓶颈。我们将无需重新训练的MoE压缩分为三种范式——专家剪枝、专家编辑和专家合并，并表明持续压缩后的退化问题主要源于一个被忽视的因素：当专家被更改但路由器未被调整时的路由-专家不匹配。我们认为有效的无需重新训练压缩应避免更新专家参数，同时允许轻量级的路由校准。为此，我们提出了路由知识蒸馏（Router KD），仅更新极小比例的参数（路由器）通过在未标记的校准数据上蒸馏原始模型的下一个令牌分布。在所有三种范式中代表性方法的实验中，展示了持续的性能恢复，由于其更复杂的路由决策边界，在细粒度MoE（许多小专家）中获得了显著更大的收益。
英文摘要：
Mixture-of-Experts (MoE) models scale capacity efficiently, but their massive parameter footprint creates a deployment-time memory bottleneck. We organize retraining-free MoE compression into three paradigms - Expert Pruning, Expert Editing, and Expert Merging - and show that persistent post-compression degradation largely stems from a neglected factor: router-expert mismatch when experts are changed but the router is left untouched. We argue that effective retraining-free compression should avoid updating expert parameters while allowing lightweight router calibration. To this end, we propose Router Knowledge Distillation (Router KD), which updates only a tiny fraction of parameters (the router) by distilling the original model's next-token distribution on unlabeled calibration data. Experiments across representative methods in all three paradigms demonstrate consistent performance recovery, with substantially larger gains in fine-grained MoEs (many small experts) than in coarse-grained MoEs due to their more complex routing decision boundaries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
