视觉语言模型中用于保守令牌压缩的谱热流
Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models
浏览论文内容
中文总结 AI 辅助
研究针对视觉语言模型推理成本高及现有令牌剪枝方法的问题，提出无需训练的SpecFlow框架，通过谱热流等操作实现保守压缩，在多方面优于现有方法，平衡了效率与准确性。
中文摘要 AI 辅助
视觉语言模型在推理时成本高昂，因为它们必须处理长序列视觉令牌。现有令牌剪枝方法在高压缩率下常因盲目丢弃信息、破坏空间结构或降低多样性而性能下降。我们提出了SpecFlow，一个无需训练的框架，将范式从破坏性剪枝转变为保守压缩，严格执行空间覆盖和统计守恒以确保稳定性。将视觉令牌视为k近邻图中的节点，SpecFlow通过谱热流计算稳定重要性场以保持结构连贯性，通过自适应空间分区分配预算以保证覆盖，并将丢弃的信息聚合到协同集汇中以维持统计守恒。该方法即插即用，无需微调，且与FlashAttention兼容。实验表明，SpecFlow在各项任务、VLM架构和剪枝率上均优于现有方法。例如，使用SpecFlow的LLaVA - 1.5在剪枝88.9%的视觉令牌后仍保留95.6%的原始性能，实现了出色的效率 - 准确性平衡。
英文摘要
Vision-Language Models (VLMs) are costly at inference time because they must process long sequences of visual tokens. Existing token pruning methods often degrade under high compression by blindly discarding information, breaking spatial structure or collapsing diversity. We propose SpecFlow, a training-free framework that shifts the paradigm from destructive pruning to conservative condensation, strictly enforcing spatial coverage and statistical conservation to ensure stability. Treating visual tokens as nodes in a $k$NN graph, SpecFlow (i) computes a stable importance field via spectral heat flow to preserve structural coherence, (ii) allocates budgets via adaptive spatial partitioning to guarantee coverage, and (iii) aggregates discarded information into coreset sinks to maintain statistical conservation. The method is plug-and-play, requires no fine-tuning, and is compatible with FlashAttention. Experiments confirm that our SpecFlow outperforms SOTA methods across tasks, VLM architectures, and pruning ratios. Notably, LLaVA-1.5 with SpecFlow retains 95.6% of original performance despite pruning 88.9% of visual tokens, offering an exceptional efficiency-accuracy balance. Code is available at https://github.com/Lzy-dot/SpecFlow
发表机构
- School of Information Science/National Key Laboratory of Deep Space Exploration University of Science and Technology of China（中国科学技术大学信息科学学院/深空探测国家实验室）
机构由 AI 辅助整理，请以论文原文为准。
