面向高效2D时序视频定位的文本-视觉提示
Text-Visual Prompting for Efficient 2D Temporal Video Grounding
AI总结：
针对3D特征提取成本高的问题，提出TVP框架与TDIoU损失，使2D TVG仅用稀疏2D特征即可显著提升性能并实现5倍推理加速。
摘要：
本文研究时序视频定位（TVG）问题，其目标是在一段长的未裁剪视频中，预测由文本句子描述的片段的起始和结束时间点。得益于细粒度3D视觉特征，TVG技术近年来取得了显著进展。然而，3D卷积神经网络（CNNs）的高复杂度使密集3D视觉特征的提取非常耗时，并需要大量内存和计算资源。为实现高效TVG，我们提出一种新颖的文本-视觉提示（TVP）框架，将优化后的扰动模式（我们称之为“提示”）同时引入TVG模型的视觉输入和文本特征。与3D CNNs形成鲜明对比的是，我们表明TVP能够在2D TVG模型中有效联合训练视觉编码器和语言编码器，并且仅使用低复杂度的稀疏2D视觉特征即可改进跨模态特征融合性能。此外，我们提出时间距离IoU（TDIoU）损失，以高效学习TVG。在Charades-STA和ActivityNet Captions两个基准数据集上的实验实证表明，所提出的TVP显著提升了2D TVG的性能（例如，在Charades-STA上提升9.79%，在ActivityNet Captions上提升30.77%），并且相比使用3D视觉特征的TVG实现了5倍推理加速。代码可在Open.Intel获取。
英文摘要：
In this paper, we study the problem of temporal video grounding (TVG), which aims to predict the starting/ending time points of moments described by a text sentence within a long untrimmed video. Benefiting from fine-grained 3D visual features, the TVG techniques have achieved remarkable progress in recent years. However, the high complexity of 3D convolutional neural networks (CNNs) makes extracting dense 3D visual features time-consuming, which calls for intensive memory and computing resources. Towards efficient TVG, we propose a novel text-visual prompting (TVP) framework, which incorporates optimized perturbation patterns (that we call 'prompts') into both visual inputs and textual features of a TVG model. In sharp contrast to 3D CNNs, we show that TVP allows us to effectively co-train vision encoder and language encoder in a 2D TVG model and improves the performance of crossmodal feature fusion using only low-complexity sparse 2D visual features. Further, we propose a Temporal-Distance IoU (TDIoU) loss for efficient learning of TVG. Experiments on two benchmark datasets, Charades-STA and ActivityNet Captions datasets, empirically show that the proposed TVP significantly boosts the performance of 2D TVG (e.g., 9.79% improvement on Charades-STA and 30.77% improvement on ActivityNet Captions) and achieves 5x inference acceleration over TVG using 3D visual features. Codes are available at Open.Intel.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
