用于视觉令牌剪枝的AI4AI框架
An AI4AI Framework for Visual Token Pruning
AI总结：
本研究提出AutoPrune框架，通过TPDSL让LLM自动设计视觉令牌剪枝策略，在14个多模态基准和3个MLLM主干上，剪枝94.4%视觉令牌仍保留超99%性能，大幅降低推理成本。
AI中文摘要：
视觉令牌剪枝可大幅降低多模态大语言模型（MLLM）的推理成本，但现有方法大多依赖固定的手工启发式规则和昂贵的专家试错。随着剪枝目标、预算和模型架构的多样化，手动探索不断扩大的设计空间变得愈发困难。本文旨在构建一个用于视觉令牌剪枝的AI4AI框架，核心问题是：大语言模型（LLM）能否自动设计出有效的视觉令牌缩减算法？尽管LLM拥有广泛的算法知识和强大的推理能力，但将这类通用知识转化为特定任务的有效解决方案仍非易事。我们认为关键在于设计合适的搜索状态表示，以连接LLM的内部知识与视觉令牌剪枝的结构要求和约束。基于这一见解，我们提出了AutoPrune，一种无需训练的LLM驱动视觉令牌剪枝策略设计框架。AutoPrune的核心是引入了令牌剪枝领域特定语言（TPDSL），包含131个可复用原子，用于预算控制、令牌评分、选择约束和令牌重组。TPDSL的一个关键特性是将每个搜索状态表示为强基础策略的残差修改，这种残差公式缩小了搜索空间，并引导LLM关注对性能最关键的策略组件。在14个多模态基准和3个MLLM主干上的实验表明，AutoPrune具有有效性、效率和可迁移性。即使在移除94.4%的视觉令牌时，AutoPrune仍能保留超过99%的全令牌性能，同时将浮点运算量（FLOPs）降低9.9倍，预填充延迟降低6.4倍。
英文摘要：
Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error. As pruning objectives, budgets, and model architectures diversify, manually navigating the expanding design space becomes increasingly difficult. This paper aims to build an AI4AI framework for visual-token pruning by addressing a natural question: Can large language models automatically design effective visual-token reduction algorithms? Although LLMs possess broad algorithmic knowledge and strong reasoning capabilities, translating such general knowledge into effective solutions for a specialized task remains nontrivial. We argue that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning. Based on this insight, we propose AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design. At its core, AutoPrune introduces a Token Pruning Domain-Specific Language (TPDSL) comprising 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly. A key property of TPDSL is that it represents each search state as a residual modification of a strong base policy. This residual formulation narrows the search space and directs the LLM's attention toward the policy components that are most consequential for performance. Experiments on 14 multimodal benchmarks and three MLLM backbones demonstrate the effectiveness, efficiency, and transferability of AutoPrune. Even when removing 94.4% of visual tokens, AutoPrune preserves more than 99% of full-token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.
