迈向细粒度视频问答
Towards Fine-Grained Video Question Answering
AI总结：
本文提出MOMA-QA数据集和SGVLM模型，通过场景图预测、帧检索与大语言模型结合，提升细粒度视频问答中的时间定位和空间关系推理能力。
摘要：
在快速发展的视频理解领域，视频问答（VideoQA）仍然是研究焦点。然而，现有数据集在时间和空间粒度上存在不足，从而限制了现有VideoQA方法的能力。本文提出了多对象多主体问答（MOMA-QA）数据集，旨在通过强调时间定位、空间关系推理和以实体为中心的查询来解决这些不足。凭借真实场景图和时序区间标注，MOMA-QA非常适合用于开发细粒度视频理解模型。此外，我们提出了一个新颖的视频-语言模型SGVLM，该模型结合了场景图预测器、高效帧检索器和预训练大语言模型，用于时间定位和细粒度关系理解。在MOMA-QA和其他公开数据集上的评估表明，我们的模型性能优越，为VideoQA设立了新的基准。
英文摘要：
In the rapidly evolving domain of video understanding, Video Question Answering (VideoQA) remains a focal point. However, existing datasets exhibit gaps in temporal and spatial granularity, which consequently limits the capabilities of existing VideoQA methods. This paper introduces the Multi-Object Multi-Actor Question Answering (MOMA-QA) dataset, which is designed to address these shortcomings by emphasizing temporal localization, spatial relationship reasoning, and entity-centric queries. With ground truth scene graphs and temporal interval annotations, MOMA-QA is ideal for developing models for fine-grained video understanding. Furthermore, we present a novel video-language model, SGVLM, which incorporates a scene graph predictor, an efficient frame retriever, and a pre-trained large language model for temporal localization and fine-grained relationship understanding. Evaluations on MOMA-QA and other public datasets demonstrate the superior performance of our model, setting new benchmarks for VideoQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
