StoryVideoQA: 通过大规模、多类型和自动生成的数据集扩展深度视频理解
StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Dataset
AI总结：
提出StoryVideoQA数据集和PlotTree方法，通过多智能体协作框架自动生成大规模深度视频理解问答对，并利用层次化情节结构提升复杂故事线推理能力。
摘要：
视频问答（VideoQA）旨在回答关于给定视频的问题。现有方法在事实型VideoQA上表现出色，但在深度视频理解（DVU）上存在困难，后者需要理解复杂的故事线。这一挑战源于固有的长程视频内容、多类型问题以及实例级故事元素，这些都限制了人工构建DVU数据集的规模和多样性。为了解决这些问题，我们之前引入了StoryMind来自动构建具有平衡细粒度主题的DVU数据集。尽管它能为电视剧生成高质量问答对，但在处理更长更复杂的电影时性能显著下降。本文进一步设计了StoryMindv2，一个增强的多智能体协作框架，用于为电视剧和电影生成高质量的DVU数据集。通过集成新颖的监督引导生成机制和精细的多审阅者投票策略，该框架用于构建StoryVideoQA，这是迄今为止最大的DVU数据集，包含超过363K个问答对，覆盖393.2小时多样化的故事视频，包括电视剧（平均1635秒）和电影（平均7878秒）。在此大规模基准上对20种最先进的VideoQA方法进行全面评估，发现它们无法完全维持长程角色关联或构建对复杂故事线的连贯理解。为弥补这一差距，我们提出PlotTree，一种新颖的视频理解智能体，将长程视频内容重新组织为层次化情节结构，从而在StoryVideoQA上实现高效的故事线推理。项目页面：https://github.com/nercms-mmap/StoryVideoQA/
英文摘要：
Video question answering (VideoQA) aims to answer questions about given videos. While existing approaches excel on factoid VideoQA, they struggle with deep video understanding (DVU), which requires the comprehension of complex storylines. This challenge arises from the inherent long-range video content, multi-faceted question types, and instance-level story elements, all of which constrain the scale and diversity of manually constructed DVU datasets. These difficulties constrain the scale and diversity of manually-constructed DVU dataset. To address these, we previously introduced StoryMind to automatically construct DVU datasets with balanced fine-grained topics. Though it can generate high-quality question-answer pairs (QAs) for TV series, it suffers significant performance degradation when handling longer and more complex movies. In this paper, we further design StoryMindv2, an enhanced multi-agent collaboration framework to generate high-quality DVU datasets for both TV series and movies. By integrating a novel supervisor-guided generation mechanism and a refined multi-reviewer voting strategy, the framework is utilized to construct StoryVideoQA, the largest DVU dataset to date, featuring over 363K QAs on 393.2 hours diverse story videos including TV series (avg. 1,635 seconds) and movies (avg. 7,878 seconds). Comprehensive evaluations of 20 state-of-the-art VideoQA methods on this large-scale benchmark reveal that they cannot fully maintain long-range character associations or construct a coherent understanding of complex storylines. To bridge this gap, we propose PlotTree, a novel video understanding agent, re-organizing long-range video content into a hierarchical plot structure, enabling efficient storyline reasoning on StoryVideoQA. Project page: https://github.com/nercms-mmap/StoryVideoQA/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
