TORINO：通过视觉语言模型中可解释的概念重叠进行令牌减少
TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models
AI总结：
研究视觉语言模型计算成本高问题，提出TORINO框架，利用稀疏自动编码器将视觉令牌投影到可解释潜在空间，通过概念重叠分组并减少令牌，兼顾效率与准确性。
摘要：
视觉语言模型在不同任务中展现出强大能力，但其计算成本受大量视觉令牌主导。现有令牌减少方法缺乏明确语义表示。我们引入TORINO，一个用于视觉语言模型中自适应视觉令牌减少的即插即用框架，无需微调基础模型。TORINO利用稀疏自动编码器将视觉令牌投影到可解释的潜在空间，通过共享概念激活分析令牌关系，定义概念重叠并分组，通过修剪或合并减少冗余，动态适应输入复杂性，在多个视觉语言基准测试中实现了良好的效率-准确性权衡。
英文摘要：
Vision-Language Models (VLMs) have demonstrated impressive capabilities across different tasks, but their computational cost is dominated by the large number of visual tokens fed to the language model. Existing token reduction methods rely on attention-based scores or pairwise similarity, without an explicit semantic representation of each token. We introduce TORINO (TOken Reduction via Interpretable coNcept Overlap), a plug-and-play framework for adaptive visual token reduction in VLMs that requires no fine-tuning of the underlying model. TORINO leverages Sparse Autoencoders (SAEs) to project visual tokens into an interpretable latent space where token relationships can be analyzed through shared concept activations. Specifically, we define concept overlap as the degree of agreement between active SAE latents and use it to group tokens that share semantic content. Reduction within each group is then performed by either pruning or merging, providing a unified framework that preserves semantically important visual information while removing redundancy. Unlike fixed-budget approaches, TORINO dynamically adapts the reduction rate to input complexity, allowing different images to retain different numbers of tokens. Experiments across multiple vision-language benchmarks show that TORINO achieves favorable efficiency-accuracy trade-offs, reducing the number of visual tokens with minimal performance loss. Source code released at https://eidoslab.github.io/TORINO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
