arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.04428cs.CV
收藏

A.I.R.: 为视频问答实现适应性、迭代性和基于推理的帧选择

A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering

University of Central Florida(中佛罗里达大学);Weill Cornell Medicine(威尔康奈尔医学院)

Yuanhao Zou, Shengji Jin, Andong Deng, Youpeng Zhao, Jun Wang, Chen Chen

AI总结:

本文提出A.I.R方法,通过深度语义分析和高效迭代循环提升视频问答的帧选择效果,实验表明其在性能和计算效率上均优于现有方法。

摘要:

有效应用视觉语言模型(VLMs)到视频问答(VideoQA)的关键在于选择简洁而全面的帧集,因为处理整个视频在计算上不可行。然而,当前的帧选择方法面临关键权衡:依赖轻量级相似性模型(如CLIP)的方法往往无法捕捉复杂查询的细微差别,导致相似性评分不准确,无法反映真实查询与帧的相关性,从而削弱帧选择。同时,利用VLM进行深入分析的方法虽然更准确,但计算成本过高。为解决这些限制,我们提出A.I.R,一种无需训练的适应性、迭代性和基于推理的帧选择方法。我们利用强大的VLM对复杂查询进行深度语义分析,并将此分析部署在成本效益高的迭代循环中,每次仅处理一小批最有潜力的帧。在各种VideoQA基准上的大量实验表明,我们的方法在性能和计算效率上均优于现有帧选择方法,并显著提升了基础VLM的性能。

英文摘要:

Effectively applying Vision-Language Models (VLMs) to Video Question Answering (VideoQA) hinges on selecting a concise yet comprehensive set of frames, as processing entire videos is computationally infeasible. However, current frame selection methods face a critical trade-off: approaches relying on lightweight similarity models, such as CLIP, often fail to capture the nuances of complex queries, resulting in inaccurate similarity scores that cannot reflect the authentic query-frame relevance, which further undermines frame selection. Meanwhile, methods that leverage a VLM for deeper analysis achieve higher accuracy but incur prohibitive computational costs. To address these limitations, we propose A.I.R., a training-free approach for Adaptive, Iterative, and Reasoning-based frame selection. We leverage a powerful VLM to perform deep, semantic analysis on complex queries, and this analysis is deployed within a cost-effective iterative loop that processes only a small batch of the most high-potential frames at a time. Extensive experiments on various VideoQA benchmarks demonstrate that our approach outperforms existing frame selection methods, significantly boosts the performance of the foundation VLM, and achieves substantial gains in computational efficiency over other VLM-based techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑