ViGT：基于Transformer中可学习令牌的无候选框视频定位
ViGT: Proposal-free Video Grounding with Learnable Token in Transformer
AI总结：
本文提出无候选框视频定位框架ViGT，通过Transformer中可学习的回归令牌[REG]聚合视频与查询全局上下文并预测时间边界，在三个公开数据集上表现良好。
摘要：
视频定位（VG）任务旨在根据丰富的语言描述，在未剪辑的视频中定位所查询的动作或事件。现有的无候选框方法受困于视频与查询之间复杂的交互，过度强调跨模态特征融合和特征相关性以完成视频定位。本文提出了一种新颖的边界回归范式，在Transformer中执行回归令牌学习。特别地，我们提出了一个简单但有效的无候选框框架，即视频定位Transformer（ViGT），它使用一个可学习的回归令牌而非多模态或跨模态特征来预测时间边界。在ViGT中，可学习令牌的优势体现在以下几个方面：（1）该令牌与视频或查询无关，避免了对原始视频和查询的数据偏差。（2）该令牌同时从视频和查询特征中进行全局上下文聚合。首先，我们采用共享特征编码器将视频和查询投影到联合特征空间中，然后执行跨模态协同注意力（即视频到查询的注意力和查询到视频的注意力），以突出每种模态中的判别性特征。此外，我们将一个可学习的回归令牌[REG]与视频和查询特征拼接起来，作为视觉-语言Transformer的输入。最后，我们利用令牌[REG]预测目标时刻，并利用视觉特征约束每个时间戳的前景和背景概率。所提出的ViGT在三个公开数据集上表现良好：ANet Captions、TACoS和YouCookII。大量的消融研究和定性分析进一步验证了ViGT的可解释性。
英文摘要：
The video grounding (VG) task aims to locate the queried action or event in an untrimmed video based on rich linguistic descriptions. Existing proposal-free methods are trapped in complex interaction between video and query, overemphasizing cross-modal feature fusion and feature correlation for VG. In this paper, we propose a novel boundary regression paradigm that performs regression token learning in a transformer. Particularly, we present a simple but effective proposal-free framework, namely Video Grounding Transformer (ViGT), which predicts the temporal boundary using a learnable regression token rather than multi-modal or cross-modal features. In ViGT, the benefits of a learnable token are manifested as follows. (1) The token is unrelated to the video or the query and avoids data bias toward the original video and query. (2) The token simultaneously performs global context aggregation from video and query features. First, we employed a sharing feature encoder to project both video and query into a joint feature space before performing cross-modal co-attention (i.e., video-to-query attention and query-to-video attention) to highlight discriminative features in each modality. Furthermore, we concatenated a learnable regression token [REG] with the video and query features as the input of a vision-language transformer. Finally, we utilized the token [REG] to predict the target moment and visual features to constrain the foreground and background probabilities at each timestamp. The proposed ViGT performed well on three public datasets: ANet Captions, TACoS and YouCookII. Extensive ablation studies and qualitative analysis further validated the interpretability of ViGT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
