振兴地域特色实现检索课视频语言预训练民主化
Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval
摘要:
视频语言预训练(VLP)的最新主流方法以端到端的方式从原始像素中学习可转移的表示,以实现下游视频语言检索的高级性能。尽管取得了令人印象深刻的结果,但VLP研究由于需要大量数据和较长的训练时间而变得非常昂贵,从而阻止了进一步的探索。在这项工作中,我们振兴稀疏采样的视频剪辑的区域功能,以显着减少空间和时间的视觉冗余,实现民主化的VLP研究,同时达到国家的最先进的结果。具体来说,为了充分挖掘区域特征的潜力,我们引入了一种新的双向区域-单词对齐正则化,该正则化适当地优化了句子中区域和某些单词之间的细粒度关系,消除了预提取的区域特征和文本之间的域/模态断开。在四个数据集上的大量下游视频语言检索任务的结果证明了我们的方法在有效性和效率上的优越性,\textit{e.g.},我们的方法与迄今为止最有效的VLP方法\cite{lei2021less}相比,数据减少了80%,预训练时间减少了85%。该代码将在\url{https://github.com/showlab/DemoVLP}上提供。
英文摘要:
Recent dominant methods for video-language pre-training (VLP) learn transferable representations from the raw pixels in an end-to-end manner to achieve advanced performance on downstream video-language retrieval. Despite the impressive results, VLP research becomes extremely expensive with the need for massive data and a long training time, preventing further explorations. In this work, we revitalize region features of sparsely sampled video clips to significantly reduce both spatial and temporal visual redundancy towards democratizing VLP research at the same time achieving state-of-the-art results. Specifically, to fully explore the potential of region features, we introduce a novel bidirectional region-word alignment regularization that properly optimizes the fine-grained relations between regions and certain words in sentences, eliminating the domain/modality disconnections between pre-extracted region features and text. Extensive results of downstream video-language retrieval tasks on four datasets demonstrate the superiority of our method on both effectiveness and efficiency, \textit{e.g.}, our method achieves competing results with 80\% fewer data and 85\% less pre-training time compared to the most efficient VLP method so far \cite{lei2021less}. The code will be available at \url{https://github.com/showlab/DemoVLP}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。