量化专家：基于专家混合的令牌感知自适应误差重建用于大视觉-语言模型量化
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
AI总结：
本文提出Quant Experts，通过令牌感知的专家混合方法，提升大视觉-语言模型在量化过程中的任务准确性，保持与全精度模型相当的性能。
摘要：
后训练量化（PTQ）作为一种有效的技术，通过压缩权重和激活以减轻视觉-语言模型（VLMs）显著的计算和内存开销。现有的PTQ方法主要依赖于静态识别和全局补偿敏感或异常通道，但往往忽略了这些重要通道在不同输入中的分布差异，导致量化效果不理想。在本工作中，我们发现重要通道的分布和出现频率在模态间以及在相同模态内，甚至在相同令牌中也显著不同。因此，我们提出了Quant Experts（QE），一种用于VLMs量化的令牌感知自适应误差补偿的专家混合方法。QE将重要通道分为令牌无关和令牌依赖组。对于前者，设计了一个共享专家，用于大多数令牌补偿全局量化误差使用低秩适配器。对于后者，路由专家包括多个路由低秩适配器，以补偿与特定令牌相关的局部量化误差。广泛的实验表明，QE在各种量化设置和模型规模（从2B到70B参数）中一致提高了任务准确性，同时保持与全精度模型相当的性能。
英文摘要：
Post-Training Quantization (PTQ) has emerged as an effective technique for alleviating the substantial computational and memory overheads of Vision-Language Models (VLMs) by compressing both weights and activations without retraining the full model. Existing PTQ methods primarily rely on static identification and global compensation of sensitive or outlier channels, yet they often overlook the distributional differences of these important channels across inputs, leading to unsatisfactory quantization. In this work, we observe that the distributions and occurrence frequencies of important channels vary significantly both across modalities and among tokens, even within the same modality. Accordingly, we propose \textbf{Quant Experts (QE)}, a token-aware adaptive error compensation with mixture-of-experts for VLMs quantization. QE divides the important channels into token-independent and token-dependent groups. For the former, a shared expert is designed for most tokens to compensate for global quantization error using a low-rank adapter. For the latter, routed experts including multiple routed low-rank adapters are elaborated to compensate for local quantization error related to specific tokens. Extensive experiments demonstrate that QE consistently enhances task accuracy across various quantization settings and model scales, ranging from 2B to 70B parameters, while maintaining performance comparable to full-precision models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
