arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.04632cs.CVcs.CL
收藏

VALUE:一个多任务的视频和语言理解评估基准

VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

Microsoft Corporation(微软公司);UNC Chapel Hill(北卡罗来纳大学教堂山分校);UC Santa Cruz(加州大学圣克鲁兹分校);UC Santa Barbara(加州大学圣塔芭芭拉分校);Tsinghua University(清华大学)
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara Lee Berg, Mohit Bansal, Ji… 展开作者

Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara Lee Berg, Mohit Bansal, Jingjing Liu, Lijuan Wang, Zicheng Liu

摘要:

大多数现有的视频和语言(VidL)的研究集中在一个单一的数据集,或一个单一的任务的多个数据集。实际上,一个真正有用的VidL系统应该很容易推广到不同的任务、领域和数据集。为了便于评估这样的系统,我们引入视频和语言理解评估(VALUE)基准,集合11 VidL数据集超过3个流行的任务:(i)文本到视频检索;(ii)视频问答;和(iii)视频描述。VALUE基准旨在涵盖广泛的视频类型、视频长度、数据量和任务难度级别。与其专注于仅包含视觉信息的单通道视频,VALUE还推广了利用视频帧及其相关字幕信息的模型,以及跨多个任务共享知识的模型。我们评估了有和没有大规模VidL预训练的各种基线方法,并系统地研究了视频输入通道,融合方法和不同视频表示的影响。我们还研究了任务之间的可转移性,并在不同的设置下进行多任务学习。我们最好的模型和人类的表现之间的显着差距,需要进一步研究先进的VidL模型。VALUE可在https://value-benchmark.github.io/上获得。

英文摘要:

Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑