arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04804cs.CL
收藏

OmniSIFT: 多模态非对称令牌压缩用于高效的多模态大语言模型

OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models

New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA)(模式识别新实验室(NLPR)、自动化研究所、中国科学院(CASIA));Nanjing University(南京大学);The Hong Kong University of Science(香港科学大学);Sichuan University(四川大学);Peking University(北京大学)

Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, Yuanxing Zhang, Jiaheng Liu, Qiang Liu, Pengfei Wan, Liang Wang

AI总结:

OmniSIFT通过非对称令牌压缩框架提升多模态大语言模型效率,采用时空视频修剪和视觉引导音频选择模块,实现低参数高鲁棒性。

摘要:

多模态大语言模型(Omni-LLMs)在音频视频理解任务中表现出色,但依赖长多模态令牌序列导致计算开销大。为解决此问题,本文提出OmniSIFT框架,采用两阶段压缩策略:(i)时空视频修剪模块去除帧内结构和帧间重叠冗余;(ii)视觉引导音频选择模块过滤音频令牌。整个框架通过可微直线通过估计器端到端优化。在五个基准测试中验证了OmniSIFT的有效性和鲁棒性。值得注意的是,对于Qwen2.5-Omni-7B,OmniSIFT仅引入4.85M参数,其延迟低于训练自由基线如OmniZip。在仅25%原始令牌上下文下,OmniSIFT在多个任务中超越所有压缩基线,甚至在某些任务中超越全令牌模型性能。

英文摘要:

Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑