SNP-S3: 面向各类视频-文本任务的共享网络预训练与显著语义增强
SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
AI总结:
针对现有像素级视频-文本预训练架构的缺陷,提出含SNP共享网络预训练与S3显著语义增强策略的框架,在多任务多数据集上实现SOTA,兼顾预训练效率与微调性能。
摘要:
我们提出一种通过直接在原始数据上预训练来学习跨模态视频表征的框架,以支撑各类下游视频-文本任务。我们的核心贡献在于预训练框架与代理任务两方面。首先,针对两种主流像素级预训练架构存在的应用范围有限或效率较低的缺陷,我们提出共享网络预训练(Shared Network Pre-training,SNP)。通过采用单个共享的BERT型网络同时优化文本特征与跨模态特征,SNP具备轻量性,可支持多种下游应用。其次,基于人类理解句子时总会关注若干“显著词”的直觉,我们提出显著语义增强(Significant Semantic Strengthening,S3)策略,该策略包含一种新颖的掩码与匹配代理任务,以提升预训练性能。在三项下游视频-文本任务、六个数据集上开展的实验表明,我们在像素级视频-文本预训练领域创下了新的state-of-the-art;同时在预训练效率与微调性能之间实现了令人满意的平衡。代码库可在https://github.com/alipay/Ant-Multi-Modal-Framework/tree/main/prj/snps3_vtp获取。
英文摘要:
We present a framework for learning cross-modal video representations by directly pre-training on raw data to facilitate various downstream video-text tasks. Our main contributions lie in the pre-training framework and proxy tasks. First, based on the shortcomings of two mainstream pixel-level pre-training architectures (limited applications or less efficient), we propose Shared Network Pre-training (SNP). By employing one shared BERT-type network to refine textual and cross-modal features simultaneously, SNP is lightweight and could support various downstream applications. Second, based on the intuition that people always pay attention to several "significant words" when understanding a sentence, we propose the Significant Semantic Strengthening (S3) strategy, which includes a novel masking and matching proxy task to promote the pre-training performance. Experiments conducted on three downstream video-text tasks and six datasets demonstrate that, we establish a new state-of-the-art in pixel-level video-text pre-training; we also achieve a satisfactory balance between the pre-training efficiency and the fine-tuning performance. The codebase are available at https://github.com/alipay/Ant-Multi-Modal-Framework/tree/main/prj/snps3_vtp.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。