arXiv 2312.048172025-09-01cs.CV

LvBench：一个面向长视频理解的多模态多功能问答基准

LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering

OpenGVLab, Shanghai AI Laboratory, China（OpenGVLab，上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Honor Device Co.,Ltd（荣耀设备有限公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；Nanjing University（南京大学）

Hongjie Zhang, Lu Dong, Yi Liu, Yifei Huang, Yali Wang, Limin Wang, Yu Qiao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有长视频问答数据集存在的视频时长短、问题类型与模态单一等问题，研究团队提出具备长时长、多模态多问题类型、高质量标注特点的LvBench基准，实验发现现有方法性能随视频与线索长度增加显著下降，该基准可为长视频理解研究提供支撑。

摘要：

尽管近期取得了显著进展，现有长视频问答数据集仍无法满足真正的长视频理解标准，这主要是因为其问题构建基于短视频，且回答问题时依赖长度有限的子片段作为线索。同时，以往数据集对问题类型和模态的关注较为局限。为弥补这一缺陷，我们提出了LvBench——一个面向多功能多模态问答的长视频理解基准。LvBench与现有长视频问答数据集相比，具备三大核心特征：1）更长的时间跨度：我们采用的视频时长从70秒到4小时不等，覆盖单场景、多场景及全场景上下文，该设计同时兼顾视频长度与线索长度，能够捕捉多样的上下文动态。2）丰富的问题类型与模态：LvBench设置了六种不同的问题类型，结合视频帧与字幕，评估多种感知与认知能力。3）高质量标注：我们采用严格的人工标注流程。该数据集包含20061个问答对，源自100部精心挑选的不同类型电影，由多名标注人员协作完成标注。对多种基线模型的分析呈现出一致趋势：当视频长度与线索长度增加时，所有现有方法的性能均显著下降。我们期望LvBench能为未来长视频理解相关研究提供有价值的资源。

英文摘要：

Despite remarkable recent progress, existing long-form VideoQA datasets fall short of meeting the criteria for genuine long-form video understanding. This is primarily due to the use of short videos for question curation, and the reliance on limited-length sub-clips as clues to answer those questions. Meanwhile, previous datasets have limited focus on question type and modality. To remedy this, we introduce LvBench, a Long-form video understanding benchmark for versatile multi-modal question-answering. Our LvBench stands out from existing long-form VideoQA datasets through three key characteristics: 1) Extended temporal durations: We consider videos ranging from 70 seconds to 4 hours, covering single-scene, multi-scene, and full-scene contexts. This design accounts for both video and clue lengths, capturing diverse contextual dynamics. 2) Diverse question types and modalities: LvBench introduces six distinct question types that evaluate various perceptual and cognitive capabilities, utilizing both video frames and subtitles. 3) High-quality annotations: We employ rigorous manual labeling by human annotators. Our dataset comprises 20,061 question-answer pairs sourced from 100 carefully selected movies across diverse genres, annotated collaboratively by multiple individuals. Analysis involving various baselines reveals a consistent trend: the performance of all existing methods significantly deteriorates when video and clue length increases. We expect LvBench to serve as a valuable resource for future works on long-form video understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。