弱监督高斯对比定位与大型多模态模型用于视频问答
Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering
AI总结：
针对视频问答中均匀采样忽略问题相关线索且缺乏关键时间戳标注的问题，提出弱监督高斯对比定位框架，利用CLIP生成伪标签并学习高斯函数采样关键帧作为LMMs输入，显著提升问答性能。
摘要：
视频问答（VideoQA）旨在根据视频中观察到的信息回答自然语言问题。尽管大型多模态模型（LMMs）近期在图像-语言理解和推理方面取得成功，但它们处理VideoQA的能力不足，仅将均匀采样的帧作为视觉输入，忽略了与问题相关的视觉线索。此外，现有VideoQA数据集中没有对问题关键时间戳的人工标注。鉴于此，我们提出一种新颖的弱监督框架，强制LMMs以问题关键时刻作为视觉输入来推理出答案。具体而言，我们首先利用CLIP模型的视觉-语言对齐能力，将问答对融合为事件描述，以找到多个关键帧作为目标时刻和伪标签。以这些伪标签关键帧作为额外的弱监督，我们设计了一个轻量级的基于高斯的对比定位（GCG）模块。GCG学习多个高斯函数来刻画视频的时间结构，并采样问题关键帧作为正样本时刻，作为LMMs的视觉输入。在多个基准上的大量实验验证了我们框架的有效性，与之前的最先进方法相比取得了显著改进。
英文摘要：
Video Question Answering (VideoQA) aims to answer natural language questions based on the information observed in videos. Despite the recent success of Large Multimodal Models (LMMs) in image-language understanding and reasoning, they deal with VideoQA insufficiently, by simply taking uniformly sampled frames as visual inputs, which ignores question-relevant visual clues. Moreover, there are no human annotations for question-critical timestamps in existing VideoQA datasets. In light of this, we propose a novel weakly supervised framework to enforce the LMMs to reason out the answers with question-critical moments as visual inputs. Specifically, we first fuse the question and answer pairs as event descriptions to find multiple keyframes as target moments and pseudo-labels, with the visual-language alignment capability of the CLIP models. With these pseudo-labeled keyframes as additionally weak supervision, we devise a lightweight Gaussian-based Contrastive Grounding (GCG) module. GCG learns multiple Gaussian functions to characterize the temporal structure of the video, and sample question-critical frames as positive moments to be the visual inputs of LMMs. Extensive experiments on several benchmarks verify the effectiveness of our framework, and we achieve substantial improvements compared to previous state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
