利用结构化时空对齐增强视频-语言表示
Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment
AI总结：
本文提出了一种即插即用的细粒度结构化时空对齐方法 Finsta，通过构建场景图并利用图 Transformer 进行时空特征传播与对齐，有效提升了现有视频-语言模型在多项下游任务中的性能。
摘要：
尽管预训练大规模视频-语言模型（VLMs）在各种下游视频-语言任务中展现出了显著潜力，但现有的 VLMs 仍可能受到一些常见局限性的困扰，例如粗粒度的跨模态对齐、对时间动态的建模不足以及视频-语言视角的分离。在本研究中，我们旨在通过一种细粒度的结构化时空对齐学习方法（即 Finsta）来增强 VLMs。首先，我们用细粒度的场景图（SG）结构表示输入文本和视频，并将两者进一步统一为一个整体场景图（HSG），以桥接两种模态。随后，构建了一个基于 SG 的框架，其中文本场景图（TSG）由 graph Transformer 进行编码，而视频动态场景图（DSG）和 HSG 则通过一种新颖的循环 graph Transformer 进行建模，以实现空间和时间特征传播。我们进一步设计了一种时空高斯微分 graph Transformer，以增强对物体在空间和时间维度上变化的感知。接着，基于 TSG 和 DSG 的细粒度结构特征，我们分别执行以物体为中心的空间对齐和以谓词为中心的时间对齐，从而在空间性和时间性两方面增强视频-语言的对应关系。我们将该方法设计为即插即用系统，可集成到现有的训练良好的 VLMs 中进行进一步的表示增强，而无需从头训练或在下游应用中依赖 SG 标注。在标准及长视频场景下的 12 个数据集上的 6 项代表性 VL 建模任务中，Finsta 持续提升了现有的 13 个性能优异的 VLMs，并在微调和零样本设置下显著刷新了当前最先进的端任务性能。
英文摘要：
While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning , under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
