Hybrid-Tower：面向文本到视频检索的细粒度伪查询交互与生成
Hybrid-Tower: Fine-grained Pseudo-query Interaction and Generation for Text-to-Video Retrieval
AI总结：
针对文本到视频检索中单塔高效但低效、双塔高效但效果差的问题，提出融合二者优势的Hybrid-Tower框架及PIG方法，通过预生成伪查询实现细粒度交互，在保持双塔推理效率的同时显著提升检索效果。
摘要：
文本到视频检索（Text-to-Video Retrieval，T2VR）任务旨在通过语义匹配的文本查询检索未标注视频。近期基于CLIP的方法主要探索了两类框架：双塔（Two-Tower）框架与单塔（Single-Tower）框架，但前者存在效果不佳的问题，后者则存在效率低下的缺陷。本研究探索了一种全新的Hybrid-Tower（混合塔）框架，能够融合双塔与单塔框架的优势，同时实现高效果与高效率。我们提出了一种新颖的混合方法——面向T2VR的细粒度伪查询交互与生成，即PIG，该方法包含一个全新的伪查询生成器，用于为每个视频生成一条伪查询。这使得视频特征与伪查询的文本特征能够进行细粒度交互，类似单塔方法以保持高效果，且该过程甚至可在接收到真实文本查询之前完成。同时，在推理阶段，本方法相较于双塔框架不会引入额外的存储或计算开销，因此保持了高效率。在5个常用的文本-视频检索基准上开展的大量实验表明，本方法相较基线取得了显著提升，R@1指标提升了1.6%~3.9%。此外，本方法在达到接近当前最优性能的同时，保持了与双塔模型相当的效率，凸显了Hybrid-Tower框架的优势。
英文摘要：
The Text-to-Video Retrieval (T2VR) task aims to retrieve unlabeled videos by textual queries with the same semantic meanings. Recent CLIP-based approaches have explored two frameworks: Two-Tower versus Single-Tower framework, yet the former suffers from low effectiveness, while the latter suffers from low efficiency. In this study, we explore a new Hybrid-Tower framework that can hybridize the advantages of the Two-Tower and Single-Tower framework, achieving high effectiveness and efficiency simultaneously. We propose a novel hybrid method, Fine-grained Pseudo-query Interaction and Generation for T2VR, ie, PIG, which includes a new pseudo-query generator designed to generate a pseudo-query for each video. This enables the video feature and the textual features of pseudo-query to interact in a fine-grained manner, similar to the Single-Tower approaches to hold high effectiveness, even before the real textual query is received. Simultaneously, our method introduces no additional storage or computational overhead compared to the Two-Tower framework during the inference stage, thus maintaining high efficiency. Extensive experiments on five commonly used text-video retrieval benchmarks demonstrate that our method achieves a significant improvement over the baseline, with an increase of $1.6\% \sim 3.9\%$ in R@1. Furthermore, our method matches the efficiency of Two-Tower models while achieving near state-of-the-art performance, highlighting the advantages of the Hybrid-Tower framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
