arXiv 2510.109622025-10-13cs.LGcs.AI

MC#：面向混合专家大模型的混合压缩器

MC#: Mixture Compressor for Mixture-of-Experts Large Models

Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；NVIDIA Research（NVIDIA研究）

Wei Huang, Yue Liao, Yukang Chen, Jianhui Liu, Haoru Tan, Si Liu, Shiming Zhang, Shuicheng Yan, Xiaojuan Qi

备注：: 15 pages, 13 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出MC#框架，结合静态混合精度量化（PMQ）与动态专家剪枝（OTP），在DeepSeek-VL2上实现6.2倍权重压缩且精度仅降1.7%，有效降低MoE大模型的存储与计算开销。

摘要：

混合专家通过稀疏激活增加容量，有效扩展了大型语言模型和视觉语言模型。然而，将所有专家预加载到内存中并为每个输入激活多个专家会引入显著的计算和内存开销，使专家模块成为模型大小和推理成本的主要因素。为解决此问题，我们提出MC#（Mixture-Compressor-sharp），该框架结合静态量化和动态专家剪枝，利用专家和Token的重要性对MoE-LLMs/VLMs进行激进压缩。为降低存储和加载成本，我们引入预加载混合精度量化（PMQ），通过线性规划优化比特分配，平衡专家重要性和量化误差，实现大小与性能的帕累托最优权衡。为减少运行时计算，在线Top-any剪枝（OTP）使用Gumbel-Softmax采样为每个Token动态选择专家子集，实现对激活的细粒度控制。结合PMQ的静态位宽优化与OTP的动态路由，MC#以最小的精度损失实现极致压缩。在DeepSeek-VL2上，MC#在2.57平均比特下实现6.2倍权重缩减，在五个多模态基准上精度仅下降1.7%。此外，OTP将专家激活减少超20%，性能下降不到1%，展示了高效部署MoE模型的强大潜力。

英文摘要：

Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation. However, preloading all experts into memory and activating multiple experts per input introduces significant computational and memory overhead, making the expert module a major contributor to model size and inference cost. To address this, we propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning by leveraging the significance of experts and tokens for aggressive compression of MoE-LLMs/VLMs. To reduce storage and loading costs, we introduce Pre-Loading Mixed-Precision Quantization (PMQ), which optimizes bit allocation via linear programming, balancing expert importance and quantization error for a Pareto-optimal trade-off between size and performance. To reduce runtime computation, Online Top-any Pruning (OTP) uses Gumbel-Softmax sampling to dynamically select a subset of experts per token, enabling fine-grained control over activation. By combining PMQ's static bit-width optimization with OTP's dynamic routing, MC# achieves extreme compression with minimal accuracy loss. On DeepSeek-VL2, MC# achieves a 6.2 times weight reduction at 2.57 average bits with only a 1.7% accuracy drop across five multimodal benchmarks. Additionally, OTP reduces expert activation over 20% with less than 1% performance degradation, demonstrating strong potential for efficient MoE-based model deployment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。