arXiv 2305.033472023-05-05cs.CV

一个具备阅读理解能力的大规模跨模态视频检索数据集

A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension

University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Kuaishou Technology（快手科技）；Huazhong University of Science and Technology（华中科技大学）

Weijia Wu, Yuzhong Zhao, Zhuang Li, Jiahong Li, Hong Zhou, Mike Zheng Shou, Xiang Bai

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出了具备文本阅读理解能力的大规模跨模态视频检索数据集TextVR，设计了对应的多模态检索基线，实验表明该基准为视频与语言领域带来了新的技术挑战与研究洞见。

摘要：

现有的大多数跨模态文本到视频检索（VR）研究都聚焦于视频的单模态输入，即视觉表征，而文本在人类环境中无处不在，且通常对理解视频至关重要。为研究如何利用视觉和文本语义表征这两种模态输入进行视频检索，我们首先提出了一个具备文本阅读理解能力的大规模跨模态视频检索数据集TextVR，该数据集包含针对8个场景领域（室内街景、室外街景、游戏、体育、驾驶、活动、电视节目、烹饪）的10.5k个视频的42.2k个句子查询。所提出的TextVR要求一个统一的跨模态模型能够识别并理解文本，将其与视觉上下文相关联，并判断哪些文本语义信息对视频检索任务至关重要。此外，我们将TextVR与现有数据集进行了详细对比分析，并为基于文本的视频检索任务设计了一种新颖的多模态视频检索基线方法。数据集分析和大量实验表明，我们的TextVR基准为视频与语言领域带来了以往数据集所不具备的诸多新技术挑战与研究洞见。项目网站和GitHub代码库分别位于https://sites.google.com/view/loveucvpr23/guest-track和https://github.com/callsys/TextVR。

英文摘要：

Most existing cross-modal language-to-video retrieval (VR) research focuses on single-modal input from video, i.e., visual representation, while the text is omnipresent in human environments and frequently critical to understand video. To study how to retrieve video with both modal inputs, i.e., visual and text semantic representations, we first introduce a large-scale and cross-modal Video Retrieval dataset with text reading comprehension, TextVR, which contains 42.2k sentence queries for 10.5k videos of 8 scenario domains, i.e., Street View (indoor), Street View (outdoor), Games, Sports, Driving, Activity, TV Show, and Cooking. The proposed TextVR requires one unified cross-modal model to recognize and comprehend texts, relate them to the visual context, and decide what text semantic information is vital for the video retrieval task. Besides, we present a detailed analysis of TextVR compared to the existing datasets and design a novel multimodal video retrieval baseline for the text-based video retrieval task. The dataset analysis and extensive experiments show that our TextVR benchmark provides many new technical challenges and insights from previous datasets for the video-and-language community. The project website and GitHub repo can be found at https://sites.google.com/view/loveucvpr23/guest-track and https://github.com/callsys/TextVR, respectively.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。