SPEED-Q：面向高效低比特端侧视觉语言模型量化的增强蒸馏分阶段处理
SPEED-Q: Staged Processing with Enhanced Distillation towards Efficient Low-bit On-device VLM Quantization
AI总结：
针对10亿至20亿参数VLMs的激进量化难题，提出SPEED-Q增强蒸馏分阶段处理框架，通过分阶段敏感性自适应机制和蒸馏增强策略解决模态量化差异与训练不稳定问题，实现低比特量化且性能优于现有方法。
摘要：
将视觉语言模型（VLMs）部署在边缘设备（如智能手机和机器人）上，对于实现低延迟、隐私保护的智能应用至关重要。鉴于这些设备的资源限制，量化通过提高内存效率和降低带宽需求提供了一种有前景的解决方案，从而促进VLMs的部署。然而，现有研究很少探索VLMs的激进量化，尤其是针对10亿至20亿参数的模型——这类模型更适合资源受限的边缘设备。本文提出SPEED-Q，一种新颖的增强蒸馏分阶段处理框架，用于VLMs的低比特仅权重量化，系统解决以下两个关键障碍：（1）VLMs中视觉（ViT）和语言（LLM）组件之间量化敏感性的显著差异；（2）低比特量化固有的数值精度降低导致的训练不稳定性。在SPEED-Q中，引入分阶段敏感性自适应机制以有效协调不同模态的性能；进一步提出蒸馏增强量化策略，以稳定训练过程并减少数据依赖性。两者结合，SPEED-Q实现了复杂VLMs的准确、稳定且数据高效的量化。SPEED-Q是首个专为将整个小规模十亿参数级VLMs量化至低比特而设计的框架。在多个基准上的广泛实验表明，在2比特设置下，SPEED-Q的精度比现有量化方法高6倍；在2比特和4比特设置下，均持续优于之前的端侧VLMs。我们的代码和模型可在https://github.com/antgroup/SPEED-Q获取。
英文摘要：
Deploying Vision-Language Models (VLMs) on edge devices (e.g., smartphones and robots) is crucial for enabling low-latency and privacy-preserving intelligent applications. Given the resource constraints of these devices, quantization offers a promising solution by improving memory efficiency and reducing bandwidth requirements, thereby facilitating the deployment of VLMs. However, existing research has rarely explored aggressive quantization on VLMs, particularly for the models ranging from 1B to 2B parameters, which are more suitable for resource-constrained edge devices. In this paper, we propose SPEED-Q, a novel Staged Processing with Enhanced Distillation framework for VLM low-bit weight-only quantization that systematically addresses the following two critical obstacles: (1) significant discrepancies in quantization sensitivity between vision (ViT) and language (LLM) components in VLMs; (2) training instability arising from the reduced numerical precision inherent in low-bit quantization. In SPEED-Q, a staged sensitivity adaptive mechanism is introduced to effectively harmonize performance across different modalities. We further propose a distillation-enhanced quantization strategy to stabilize the training process and reduce data dependence. Together, SPEED-Q enables accurate, stable, and data-efficient quantization of complex VLMs. SPEED-Q is the first framework tailored for quantizing entire small-scale billion-parameter VLMs to low bits. Extensive experiments across multiple benchmarks demonstrate that SPEED-Q achieves up to 6x higher accuracy than existing quantization methods under 2-bit settings and consistently outperforms prior on-device VLMs under both 2-bit and 4-bit settings. Our code and models are available at https://github.com/antgroup/SPEED-Q.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
