面向路由一致性的混合专家模型量化的值与结构对齐
Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models
AI总结：
提出VSRAQ方法，通过值对齐和结构对齐两个互补目标保持量化前后的专家选择行为一致性，减少量化引起的性能下降，无需推理开销。
摘要：
混合专家（MoE）模型通过仅为每个token激活一部分专家来高效扩展基础模型，但大量的专家参数使得量化对于实际部署至关重要。然而，与密集模型不同，MoE模型对路由不稳定性敏感：小的量化引起的扰动可能改变top-$k$专家选择，改变计算路径并降低模型质量。我们提出了面向量化的值与结构路由对齐（VSRAQ），这是一种针对MoE的后训练量化目标，旨在量化下保持量化前的专家选择行为。VSRAQ结合了两个互补目标，共同保持专家选择行为：值对齐，匹配与路由相关的logits或分数；结构对齐，保持专家排序和top-$k$决策边界。通过维持路由一致性，VSRAQ减少了量化引起的性能下降，且不引入任何推理时开销，并可集成到现有量化框架中。在近期MoE基础模型上的实验表明，VSRAQ提高了专家选择一致性，并始终优于仅重建和考虑路由器的基线方法。
英文摘要：
Mixture-of-Experts (MoE) models scale foundation models efficiently by activating only a subset of experts for each token, but their large number of expert parameters still makes quantization essential for practical deployment. Unlike dense models, however, MoE models are sensitive to routing instability: small quantization-induced perturbations can change the top-$k$ expert selection, altering the computation path and degrading model quality. We propose Value-and-Structure Routing Alignment for Quantization (VSRAQ), a MoE-specific post-training quantization objective that preserves pre-quantization expert-selection behavior under quantization. VSRAQ combines two complementary objectives that jointly preserve expert-selection behavior: value alignment, which matches routing-relevant logits or scores, and structure alignment, which preserves expert ordering and top-$k$ decision boundaries. By maintaining routing consistency, VSRAQ reduces quantization-induced degradation without introducing any inference-time overhead and can be integrated into existing quantization frameworks. Experiments on recent MoE foundation models show that VSRAQ improves expert-selection consistency and consistently outperforms reconstruction-only and router-aware baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
