arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.07086cs.CV
收藏

MDMMT-2:多域多模态视频检索Transformer,向通用化又迈进了一步

MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalization

Huawei(华为)

Alexander Kunitsyn, Maksim Kalashnikov, Maksim Dzabraev, Andrei Ivaniuta

摘要:

在这项工作中,我们提出了一个新的国家的最先进的文本到视频检索任务的MSR-VTT,LSMDC,MSVD,YouCook 2和TGIF获得一个单一的模型。三种不同的数据源相结合:弱监督的视频,人群标记的文本图像对和文本视频对。仔细分析可用的预训练网络有助于选择最佳的先验知识网络。我们引入了三阶段的训练过程,提供了高的知识转移效率,并允许在训练过程中使用噪声数据集,而不会降低先验知识。此外,双位置编码用于更好地融合不同的模态和非方输入处理的一个简单的方法提出。

英文摘要:

In this work we present a new State-of-The-Art on the text-to-video retrieval task on MSR-VTT, LSMDC, MSVD, YouCook2 and TGIF obtained by a single model. Three different data sources are combined: weakly-supervised videos, crowd-labeled text-image pairs and text-video pairs. A careful analysis of available pre-trained networks helps to choose the best prior-knowledge ones. We introduce three-stage training procedure that provides high transfer knowledge efficiency and allows to use noisy datasets during training without prior knowledge degradation. Additionally, double positional encoding is used for better fusion of different modalities and a simple method for non-square inputs processing is suggested.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑