P4Q：学习为视觉语言模型的量化进行提示
P4Q: Learning to Prompt for Quantization in Visual-language Models
AI总结：
提出P4Q方法，通过可学习提示重组文本表示与低比特适配器对齐特征分布，结合对比损失和蒸馏损失，在极低额外参数下显著提升低比特PTQ视觉语言模型的性能。
摘要：
大规模预训练视觉语言模型（VLMs）在各种视觉和多模态任务中获得了重要地位，然而由于其对训练样本和计算资源的极高要求，将VLMs部署到下游应用平台上仍然具有挑战性。VLMs的微调和量化可以大幅降低样本和计算成本，这是当前迫切需要的。量化领域存在两种主流范式：量化感知训练（QAT）能够有效量化大规模VLMs，但会带来巨大的训练成本；而低比特训练后量化（PTQ）则存在显著的性能下降问题。我们提出了一种平衡微调和量化的方法，名为“为量化进行提示”（P4Q），其中我们设计了一个轻量级架构，利用对比损失监督来增强PTQ模型的识别性能。我们的方法能够有效缩小由低比特量化引起的图像特征与文本特征之间的差距，其基于可学习提示来重组文本表示，并利用低比特适配器重新对齐图像和文本特征的分布。我们还引入了一种基于余弦相似度预测的蒸馏损失，使用全精度教师模型来蒸馏量化模型。大量实验结果表明，我们的P4Q方法优于先前的方法，甚至取得了与其全精度对应模型相当的结果。例如，我们的8位P4Q在理论上可以将CLIP-ViT/B-32压缩4倍，同时在ImageNet数据集上实现66.94%的Top-1准确率，以可忽略的额外参数超越了经可学习提示微调的全精度模型2.24%。
英文摘要：
Large-scale pre-trained Vision-Language Models (VLMs) have gained prominence in various visual and multimodal tasks, yet the deployment of VLMs on downstream application platforms remains challenging due to their prohibitive requirements of training samples and computing resources. Fine-tuning and quantization of VLMs can substantially reduce the sample and computation costs, which are in urgent need. There are two prevailing paradigms in quantization, Quantization-Aware Training (QAT) can effectively quantize large-scale VLMs but incur a huge training cost, while low-bit Post-Training Quantization (PTQ) suffers from a notable performance drop. We propose a method that balances fine-tuning and quantization named ``Prompt for Quantization'' (P4Q), in which we design a lightweight architecture to leverage contrastive loss supervision to enhance the recognition performance of a PTQ model. Our method can effectively reduce the gap between image features and text features caused by low-bit quantization, based on learnable prompts to reorganize textual representations and a low-bit adapter to realign the distributions of image and text features. We also introduce a distillation loss based on cosine similarity predictions to distill the quantized model using a full-precision teacher. Extensive experimental results demonstrate that our P4Q method outperforms prior arts, even achieving comparable results to its full-precision counterparts. For instance, our 8-bit P4Q can theoretically compress the CLIP-ViT/B-32 by 4 $\times$ while achieving 66.94\% Top-1 accuracy, outperforming the learnable prompt fine-tuned full-precision model by 2.24\% with negligible additional parameters on the ImageNet dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
