面向图像-文本模型的高效视频问答的自适应采样
Self-Adaptive Sampling for Efficient Video Question-Answering on Image--Text Models
- Singapore University of Technology and Design（新加坡科技设计大学）
- National University of Singapore（新加坡国立大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对视频问答中随机采样遗漏关键帧的问题，提出MDF和MIF两种自适应帧采样策略，在三个数据集和三种先进VLM上验证可有效提升图像-文本预训练模型的性能。
AI中文摘要：
视频问答是视频理解领域的一项基础任务。尽管当前配备视频Transformer的视觉-语言模型（VLMs）已经能够实现时序建模并取得优越的结果，但它们以巨大的计算能力为代价，因此过于昂贵，难以部署在实时应用场景中。一种经济的变通方法是仅采样一小部分帧来表示该视频的主要内容，并在这些采样帧上微调图像-文本模型。近期的视频理解模型通常随机采样一组帧或片段，而不考虑其视觉内容之间的内部相关性，也不考虑它们与问题之间的相关性。我们认为，这种漫无目的的采样可能会遗漏能够推导出正确答案的关键帧，并且当采样稀疏度增加时情况会变得更糟，而随着视频长度增加，采样稀疏度增加总是会发生。为了缓解这一问题，我们提出了两种帧采样策略，即最具领域代表性帧（MDF）和最具隐含信息帧（MIF），以最大限度地保留那些最有可能对给定问题至关重要的帧。MDF以自举方式被动地最小化关键帧遗漏的风险，而MIF在辅助模型的协助下主动搜索为每个视频-问题对定制的最具隐含信息帧。在三个公共数据集上基于三种先进VLM（CLIP、GIT和All-in-one）的实验结果表明，我们提出的策略能够提升图像-文本预训练模型的性能。与本文所提方法相关的源代码已在https://github.com/declare-lab/sas-vqa上公开提供。
英文摘要：
Video question-answering is a fundamental task in the field of video understanding. Although current vision--language models (VLMs) equipped with Video Transformers have enabled temporal modeling and yielded superior results, they are at the cost of huge computational power and thus too expensive to deploy in real-time application scenarios. An economical workaround only samples a small portion of frames to represent the main content of that video and tune an image--text model on these sampled frames. Recent video understanding models usually randomly sample a set of frames or clips, regardless of internal correlations between their visual contents, nor their relevance to the problem. We argue that such kinds of aimless sampling may omit the key frames from which the correct answer can be deduced, and the situation gets worse when the sampling sparsity increases, which always happens as the video lengths increase. To mitigate this issue, we propose two frame sampling strategies, namely the most domain frames (MDF) and most implied frames (MIF), to maximally preserve those frames that are most likely vital to the given questions. MDF passively minimizes the risk of key frame omission in a bootstrap manner, while MIS actively searches key frames customized for each video--question pair with the assistance of auxiliary models. The experimental results on three public datasets from three advanced VLMs (CLIP, GIT and All-in-one) demonstrate that our proposed strategies can boost the performance for image-text pretrained models. The source codes pertaining to the method proposed in this paper are publicly available at https://github.com/declare-lab/sas-vqa.
