arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.20035cs.CV
收藏

面向高效多模态大语言模型的阶段自适应令牌选择

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

Renmin University of China(中国人民大学);WeChat Vision, Tencent Inc.(腾讯微信视觉实验室)

Zijie Xin, Jie Yang, Ruixiang Zhao, Tianyi Wang, Fengyun Rao, Jing Lyu, Xirong Li

AI总结:

本文提出SEATS方法,通过阶段自适应的令牌选择技术,有效提升多模态大语言模型的推理效率,在保留96.3%原始性能的同时,实现9.3倍的FLOPs减少和4.8倍的prefill加速。

摘要:

多模态大语言模型(om-LLMs)通过将视频和音频编码为时间对齐的令牌序列,在窗口级别交错处理以实现统一的音频-视觉理解。然而,处理这些密集的非文本令牌会带来显著的计算开销。尽管训练无关的令牌选择可以减少这种成本,但现有方法要么专注于视觉输入,要么在LLM之前以固定的每模态比例修剪om-LLM令牌,无法捕捉跨模态令牌重要性在层间的变化。为了解决这一限制,我们首先分析om-LLMs的层间令牌依赖性。我们发现视觉和音频依赖性遵循块状模式,并随着深度逐渐减弱,表明许多后期层的非文本令牌在跨模态融合后变得冗余。受此启发,我们提出SEATS,一种训练无关的、阶段自适应的令牌选择方法,用于高效的om-LLM推理。在LLM之前,SEATS通过注意力加权多样性选择去除时空冗余。在LLM内部,它逐步在块间修剪令牌,并利用查询相关性分数动态分配从时间窗口到模态的保留预算。在后期层中,一旦完成跨模态融合,它会移除所有剩余的非文本令牌。在Qwen2.5-Omni和Qwen3-Omni上的实验表明,SEATS有效提高了推理效率。仅保留10%的视觉和音频令牌,实现了9.3倍的FLOPs减少和4.8倍的prefill加速,同时保持96.3%的原始性能。

英文摘要:

The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens throughout the LLM. To reduce the cost, training-free methods for om-LLM token selection are being actively developed, operating either pre-LLM or inner-LLM. Recognizing that the non-textual token redundancy varies across stages and LLM layers, we propose in this paper a novel stage-adaptive token selection (SEATS) method that operates in both pre-LLM and inner-LLM stages. Given an overall token budget measured by the token retention rate (TRR), SEATS performs top-down token budget allocation in an input-dependent manner. Specifically, a larger TRR is used for pre-LLM token selection, whilst within the LLM, all tokens are retained in shallow layers, increasingly decreased TRRs are applied in middle layers, and the non-textual tokens are fully removed in late layers. Extensive experiments on three representative om-LLMs and six benchmarks verify the viability of the proposed method. By retaining only 10\% of visual and audio tokens, SEATS preserves 94.9\% of the full-token performance on Qwen2.5-Omni while achieving a 9.3x FLOPs reduction and a 4.9x prefill speedup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑