Effectively applying Vision-Language Models (VLMs) to Video Question Answering (VideoQA) hinges on selecting a concise yet comprehensive set of frames, as processing entire videos is computationally infeasible. However, current frame selection methods face a critical trade-off: approaches relying on lightweight similarity models, such as CLIP, often fail to capture the nuances of complex queries, resulting in inaccurate similarity scores that cannot reflect the authentic query-frame relevance, which further undermines frame selection. Meanwhile, methods that leverage a VLM for deeper analysis achieve higher accuracy but incur prohibitive computational costs. To address these limitations, we propose A.I.R., a training-free approach for Adaptive, Iterative, and Reasoning-based frame selection. We leverage a powerful VLM to perform deep, semantic analysis on complex queries, and this analysis is deployed within a cost-effective iterative loop that processes only a small batch of the most high-potential frames at a time. Extensive experiments on various VideoQA benchmarks demonstrate that our approach outperforms existing frame selection methods, significantly boosts the performance of the foundation VLM, and achieves substantial gains in computational efficiency over other VLM-based techniques.
Leveraging Video Descriptions to Learn Video Question Answering
National Tsing Hua University(台湾清华大学);Stanford University(斯坦福大学)
我们提出了一种可扩展的方法来学习基于视频的问答(QA):回答一个“自由形式的自然语言问题”的视频内容。我们的方法自动收集了大量在线免费提供的视频和描述。然后,大量的候选QA对自动生成的描述,而不是手动注释。接下来,我们使用这些候选QA对来训练许多基于视频的QA方法,这些方法是从MN(Sukhbaatar et al. 2015)、VQA(Antol et al. 2015)、SA(Yao et al. 2015)、SS(Venugopalan et al. 2015)扩展而来的。为了处理非完美的候选QA对,我们提出了一个自定进度的学习过程来迭代地识别它们并减轻它们在训练中的影响。最后,我们评估手动生成的基于视频的QA对的性能。结果表明,我们的自定进度的学习过程是有效的,和扩展的SS模型优于各种基线。
ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering
Digital Research Center of Sfax, SMARTS Laboratory(突尼斯斯法克斯数字研究中心,SMARTS实验室);École Polytechnique de Tunisie, University of Carthage(突尼斯理工学院,卡塔赫纳大学);Computer Vision Center, Universitat Autònoma de Barcelona(巴塞罗那自治大学计算机视觉中心)
MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别新实验室)
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence(人工智能交叉学科项目);Department of Intelligence and Information(智能与信息系);Samsung Advanced Institute of Technology(三星先进技术研究院)
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
OPPO AI Center, OPPO Inc. China(OPPO AI中心,OPPO公司);The Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Nanyang Technological University, Singapore(新加坡南洋理工大学)
Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
The University of Sydney, School of Computer Science(悉尼大学计算机科学学院);University of Technology Sydney, School of Computer Science(新南威尔士大学技术学院);University of Technology Sydney, TD School(新南威尔士大学TD学院);Australian Catholic University, Peter Faber Business School(澳大利亚天主教大学彼得·法伯商学院)
A Knowledge Noise Mitigation Framework for Knowledge-based Visual Question Answering
School of Computer, Electronics and Information(计算机、电子与信息学院);Guangxi University(广西大学);Guangxi Key Laboratory of Multimedia Communications and Network Technology(广西多媒体通信与网络技术重点实验室)
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline
Buzzni AI Lab(Buzzni人工智能实验室);KAIST(韩国科学技术院);Mathpresso Inc.(Mathpresso公司);UC Berkeley(加州大学伯克利分校);Fainders AI(Fainders人工智能公司);Seoul National University(首尔大学)
Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Microsoft Research Asia(微软亚洲研究院);University of Adelaide(阿德莱德大学)
Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering