面向视觉-语言模型的联合量化与令牌剪枝
Towards Joint Quantization and Token Pruning of Vision-Language Models
AI总结：
本文提出联合量化与令牌剪枝框架，通过统一低比特推理和确定性视觉令牌剪枝，提升视觉-语言模型在低比特下的鲁棒性与效率。
摘要：
在低比特推理下部署视觉-语言模型（VLMs）仍具挑战性，因为推理成本主要由预填充阶段的长视觉令牌前缀和自回归解码阶段增长的KV缓存主导。令牌剪枝和低比特量化可互补降低这些成本，但朴素的分阶段组合常因量化校准与剪枝执行不匹配而脆弱。本文提出联合量化与剪枝框架，引入QUOTA机制，将低比特校准信号转化为层间令牌分配计划，并将其实体化为剪枝配方。通过结合激活幅度、注意力线索和显式低比特风险信号评估令牌重要性，实现一致的预算化top-k选择。在标准VLM基准上的实验表明，与分阶段基线相比，相同低比特条件下，该方法在保持30%视觉令牌的同时实现了95.65%的平均保留率，比代表性分阶段组合的94.3%保留率有所提升。代码将被释放。
英文摘要：
Deploying Vision-Language Models (VLMs) under aggressive low-bit inference remains challenging because inference cost is dominated by the long visual-token prefix during prefill and the growing KV cache during autoregressive decoding. Token pruning and low-bit quantization are complementary for reducing these costs, yet naive stage-wise combinations are often brittle due to a mismatch between quantization calibration and pruning execution. We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning in a single deployable pipeline. The framework introduces the \textbf{Q}uantization \textbf{U}nified \textbf{O}ffline \textbf{T}oken \textbf{A}llocator (\textbf{QUOTA}), which converts low-bit calibration signals into a layer-wise token allocation schedule and materializes it as a pruning recipe. Token importance is evaluated under deployed W4A4 operators with a quantized KV cache by combining activation magnitude, attention cues, and an explicit low-bit risk signal, enabling consistent budgeted top-$k$ selection. Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime, achieving 95.65\% average retention while retaining only 30\% of visual tokens, compared with about 94.3\% retention for representative stage-wise combinations. The code will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
