GIFT:全局不可替代性帧目标用于高效视频理解
GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding
AI总结:
GIFT通过评估帧的内在不可替代性选择关键帧,解决视频理解中处理密集帧的高计算成本问题,提升视频分析效率。
摘要:
视频大语言模型(VLMs)在视频理解中取得了显著成功,但处理密集帧的高计算成本限制了其实际应用。现有方法通过选择关键帧缓解这一问题,但其贪心决策结合相关性和多样性分离评估常陷入局部最优,导致错误选择无关噪声帧。为此,我们提出GIFT:全局不可替代性帧目标,一种无需训练的框架,通过评估帧的内在不可替代性选择帧。具体而言,我们首先引入定向多样性,量化帧的唯一性,基于相关性条件,从而制定统一的不可替代性分数。随后,我们的预算感知细化策略采用自适应迭代过程,首先确保最高不可替代性的核心帧集,然后随着预算扩展,将其优先级转向构建这些选择周围的关键时间上下文。大量实验表明,GIFT在LLaVA-Video-7B长视频基准上,相比均匀采样,最大平均改进达12.5%。
英文摘要:
Video Large Language Models (VLMs) have achieved remarkable success in video understanding, but the significant computational cost from processing dense frames severely limits their practical application. Existing methods alleviate this by selecting keyframes, but their greedy decision-making, combined with a decoupled evaluation of relevance and diversity, often falls into local optima and results in erroneously selecting irrelevant noise frames. To address these challenges, we propose GIFT: Global Irreplaceability Frame Targeting, a novel training-free framework that selects frames by assessing their intrinsic irreplaceability. Specifically, we first introduce Directed Diversity to quantify a frame's uniqueness conditioned on relevance, which allows us to formulate a unified irreplaceability score. Subsequently, our Budget-Aware Refinement strategy employs a adaptive iterative process that first secures a core set of frames with the highest irreplaceability, and then shifts its priority to building crucial temporal context around these selections as the budget expands. Extensive experiments demonstrate that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling. Code is available at https://github.com/ylnxxts/GIFT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。