arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19865cs.CV
收藏

TokenBinder:采用一对多对齐范式的文本-视频检索

TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigm

The University of Queensland(昆士兰大学);CSIRO Data61(联邦科学与工业研究组织Data61)

Bingqing Zhang, Zhuo Cao, Heming Du, Xin Yu, Xue Li, Jiajun Liu, Sen Wang

AI总结:

TokenBinder提出一对多由粗到细对齐范式,通过聚焦视图融合网络和交叉注意力机制动态比较多视频特征,在六个基准上显著超越现有文本-视频检索方法。

摘要:

文本-视频检索(TVR)方法通常通过粗粒度、细粒度或组合(由粗到细)的方式对齐文本和视频特征来匹配查询-候选对。然而,这些框架主要采用一(查询)对一(候选)的对齐范式,难以辨别候选之间的细微差异,导致频繁误匹配。受人类认知科学中的比较判断启发,即决策是通过直接比较项目而非独立评估它们来做出,我们提出了TokenBinder。这一创新的两阶段TVR框架引入了一种新颖的一对多由粗到细对齐范式,模仿人类在大型集合中识别特定项目的认知过程。我们的方法采用带有复杂交叉注意力机制的聚焦视图融合网络,动态对齐和比较多个视频之间的特征,以捕捉更细微的差异和上下文变化。在六个基准数据集上的大量实验证实,TokenBinder显著优于现有的最先进方法。这些结果证明了其鲁棒性以及其细粒度对齐在弥合TVR任务中模态内和模态间信息差距方面的有效性。

英文摘要:

Text-Video Retrieval (TVR) methods typically match query-candidate pairs by aligning text and video features in coarse-grained, fine-grained, or combined (coarse-to-fine) manners. However, these frameworks predominantly employ a one(query)-to-one(candidate) alignment paradigm, which struggles to discern nuanced differences among candidates, leading to frequent mismatches. Inspired by Comparative Judgement in human cognitive science, where decisions are made by directly comparing items rather than evaluating them independently, we propose TokenBinder. This innovative two-stage TVR framework introduces a novel one-to-many coarse-to-fine alignment paradigm, imitating the human cognitive process of identifying specific items within a large collection. Our method employs a Focused-view Fusion Network with a sophisticated cross-attention mechanism, dynamically aligning and comparing features across multiple videos to capture finer nuances and contextual variations. Extensive experiments on six benchmark datasets confirm that TokenBinder substantially outperforms existing state-of-the-art methods. These results demonstrate its robustness and the effectiveness of its fine-grained alignment in bridging intra- and inter-modality information gaps in TVR tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑