中国人民大学在TRECVID 2022:通过特征融合和否定理解来改善视频搜索
Renmin University of China at TRECVID 2022: Improving Video Search by Feature Fusion and Negation Understanding
摘要:
我们总结了我们的TRECVID 2022 Ad-hoc视频搜索(AVS)实验。我们的解决方案采用了两种新技术,即用于结合不同视觉/文本特征的轻量级注意力特征融合(LAFF)和用于解决包含否定线索的查询的双向否定学习(BNL)。特别是,LAFF在早期和后期以及文本和视频端执行特征融合,以利用不同的(现成的)特征。与多头自我注意相比,LAFF更紧凑,但更有效。它的注意力权重也可以用于选择更少的特征,而检索性能基本保持不变。BNL通过最小化每个三元组的双向约束损失来训练否定感知视频检索模型,其中三元组由给定的训练视频、其原始描述和部分否定的描述组成。对于视频特征提取,我们使用预训练的CLIP,BLIP,BEiT,ResNeXt-101和irCSN。在文本特征方面,我们采用了词袋、词向量、CLIP和BLIP。我们的训练数据包括我们以前参与时使用的MSR-VTT、TGIF和VATEX。此外,我们自动为V3 C1集合添加标题以进行预训练。2022年版TRECVID基准测试再次成为RUCMM团队富有成效的参与。我们最好的成绩是0.262,排在第二位。
英文摘要:
We summarize our TRECVID 2022 Ad-hoc Video Search (AVS) experiments. Our solution is built with two new techniques, namely Lightweight Attentional Feature Fusion (LAFF) for combining diverse visual / textual features and Bidirectional Negation Learning (BNL) for addressing queries that contain negation cues. In particular, LAFF performs feature fusion at both early and late stages and at both text and video ends to exploit diverse (off-the-shelf) features. Compared to multi-head self attention, LAFF is much more compact yet more effective. Its attentional weights can also be used for selecting fewer features, with the retrieval performance mostly preserved. BNL trains a negation-aware video retrieval model by minimizing a bidirectionally constrained loss per triplet, where a triplet consists of a given training video, its original description and a partially negated description. For video feature extraction, we use pre-trained CLIP, BLIP, BEiT, ResNeXt-101 and irCSN. As for text features, we adopt bag-of-words, word2vec, CLIP and BLIP. Our training data consists of MSR-VTT, TGIF and VATEX that were used in our previous participation. In addition, we automatically caption the V3C1 collection for pre-training. The 2022 edition of the TRECVID benchmark has again been a fruitful participation for the RUCMM team. Our best run, with an infAP of 0.262, is ranked at the second place teamwise.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。