音频-视觉交换感知令牌剪枝用于高效音频-视觉字幕生成
Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning
AI总结：
提出基于强化学习的AVEX-Prune方法，通过跨模态令牌交换策略选择高置信度令牌，在40%保留率下保持全令牌质量。
摘要：
音频-视觉字幕生成从视频和音频内容生成自然语言描述。多模态大语言模型推进了这一任务，但两种模态都为LLM输入贡献了大量令牌，其中预填充自注意力呈二次方扩展。现有的令牌剪枝方法通常通过注意力、显著性或交叉熵损失保留令牌，但硬阈值选择使得难以保留真正有价值的令牌，尤其是决策边界附近的高混淆令牌。为此，我们提出AVEX-Prune，一种基于强化学习的音频-视觉动态令牌剪枝方法。在我们的AVEX-Prune中，提出了一种音频-视觉令牌交换策略，通过用来自同一或另一模态的高置信度候选令牌替换低置信度保留令牌，并测量令牌交换带来的字幕生成差异，来选择真正有价值的令牌。AVEX-Prune在VILA 1.5-8B（54.5 vs. 54.6）和VideoLLaMA 2（57.0 vs. 56.8）上以40%保留率保持了全令牌质量。
英文摘要：
Audio-visual captioning generates natural language descriptions from video and audio content. Multimodal LLMs have advanced this task, but both modalities contribute many tokens to the LLM input, where prefill self-attention scales quadratically. Existing token-pruning methods usually retain tokens by attention, saliency, or cross-entropy loss, yet the hard threshold selection makes it difficult to retain tokens that are truly valuable, especially for high-confusing tokens near the decision boundary. To this end, we propose a AVEX-Prune, an RL-based audio-visual dynamic token pruning method in this work. In our AVEX-Prune, an audio-visual token exchange strategy is proposed to select truly valuable tokens by replacing low-confidence retained tokens with high-confidence candidate tokens from the same or the other modality, and measuring the differences in caption generation from token swaps. AVEX-Prune preserves full-token quality at a 40% retention ratio on both VILA 1.5-8B (54.5 vs. 54.6) and VideoLLaMA 2 (57.0 vs. 56.8).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
