arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2001.09099cs.CVcs.CLcs.IR
收藏

TVR:一个用于视频字幕矩检索的大规模数据集

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)

Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal

摘要:

我们介绍了电视节目检索(TVR),一个新的多模态检索数据集。TVR要求系统理解视频及其相关的字幕(对话)文本,使其更加逼真。该数据集包含从6个不同类型的电视节目中收集的21.8K视频上的109 K查询,其中每个查询都与紧密的时间窗口相关联。这些查询还标记了查询类型,表明它们中的每一个都与视频或字幕或两者更相关,从而允许对数据集和构建在其上的方法进行深入分析。应用严格的资格和注释后验证测试来确保收集的数据的质量。此外,我们提出了几个基线和一个新的跨模态矩定位(XML)网络的多模态矩检索任务。所提出的XML模型使用了一种新的卷积开始-结束检测器(ConvSE)的后期融合设计,大大超过了基线,效率更高,为未来的工作提供了一个强有力的起点。我们还为TVR中的每个注释时刻收集了额外的描述,以形成一个新的具有262 K字幕的多模态字幕数据集,称为电视节目字幕(TVC)。这两个数据集都是公开的。TVR:https://tvr.cs.unc.edu,TVC:https://tvr.cs.unc.edu/tvc.html。

英文摘要:

We introduce TV show Retrieval (TVR), a new multimodal retrieval dataset. TVR requires systems to understand both videos and their associated subtitle (dialogue) texts, making it more realistic. The dataset contains 109K queries collected on 21.8K videos from 6 TV shows of diverse genres, where each query is associated with a tight temporal window. The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it. Strict qualification and post-annotation verification tests are applied to ensure the quality of the collected data. Further, we present several baselines and a novel Cross-modal Moment Localization (XML ) network for multimodal moment retrieval tasks. The proposed XML model uses a late fusion design with a novel Convolutional Start-End detector (ConvSE), surpassing baselines by a large margin and with better efficiency, providing a strong starting point for future work. We have also collected additional descriptions for each annotated moment in TVR to form a new multimodal captioning dataset with 262K captions, named TV show Caption (TVC). Both datasets are publicly available. TVR: https://tvr.cs.unc.edu, TVC: https://tvr.cs.unc.edu/tvc.html.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑