视觉编码器的激活量化需要前缀寄存器
Activation Quantization of Vision Encoders Needs Prefixing Registers
AI总结：
本文提出RegCache算法，通过引入前缀标记减少视觉编码器中的异常值，提升低比特量化模型性能，尤其在极低比特情况下效果显著。
摘要：
大规模预训练视觉编码器是多模态智能的核心，驱动着从设备端视觉处理到视觉-语言模型的各种应用。由于这些应用往往需要实时处理大量视觉数据，降低视觉编码器的推理成本至关重要。量化提供了一条可行的路径，但即使在8位精度下，由于所谓的异常值，仍具挑战性。在本文中，我们提出了RegCache，一种无需训练的算法，旨在减轻大规模预训练视觉编码器中的异常值，并作为可插拔模块应用于其他量化方法之上。RegCache向视觉编码器引入易产生异常值但语义上无意义的前缀标记，从而防止其他标记产生异常值。值得注意的是，我们观察到视觉编码器中的异常值行为与语言模型中的不同，这促使我们提出两个技术创新：中间层前缀和标记删除。实验结果表明，我们的方法在各种视觉编码器上一致提升了量化模型的性能，尤其是在极低比特情况下（例如4位）。
英文摘要：
Large pretrained vision encoders are central to multimodal intelligence, powering applications from on-device vision processing to vision-language models. Since these applications often demand real-time processing of massive visual data, reducing the inference cost of vision encoders is critical. Quantization offers a practical path, but it remains challenging even at 8-bit precision due to so-called outliers. In this work, we propose $\textit{RegCache}$, a training-free algorithm that mitigates outliers in large-scale pretrained vision encoders and serves as a plug-in module that can be applied on top of other quantization methods. RegCache introduces outlier-prone yet semantically meaningless prefix tokens to the vision encoder, which prevent other tokens from having outliers. Notably, we observe that outliers in vision encoders behave differently from those in language models, motivating two technical innovations: middle-layer prefixing and token deletion. Experimental results show that our method consistently improves quantized model performance across various vision encoders, particularly in extremely low-bit regimes (e.g., 4-bit).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
