用于时刻检索、分割和步骤描述的以查询为中心的音视频认知网络
Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning
AI总结：
针对HIREST任务中跨模态层次和关联关系被忽略的问题，提出以查询为中心的音视频认知网络QUAG，通过模态协同感知和深层查询过滤构建多模态表示，在多项任务上达到SOTA并具备良好泛化能力。
摘要：
视频已成为互联网上备受青睐的多媒体格式。为了更好地获取视频内容，一个名为HIREST的新课题被提出，包含视频检索、时刻检索、时刻分割和步骤描述。这项开创性工作选择预训练的基于CLIP的模型进行视频检索，并将其作为特征提取器，在多任务学习范式中解决其他三个具有挑战性的任务。然而，由于忽略了跨模态的层次和关联关系，该工作难以学习对用户偏好内容的全面认知。在本文中，受由浅入深原则指导，我们提出以查询为中心的音视频认知（QUAG）网络，为时刻检索、分割和步骤描述构建可靠的多模态表示。具体而言，我们首先设计模态协同感知，通过对视觉和音频模态之间的全局对比对齐和局部细粒度交互进行建模，获取丰富的音视频内容。然后，我们设计以查询为中心的认知，使用深层查询对浅层音视频表示执行时间通道过滤。这能认知用户偏好内容，从而为三项任务获得以查询为中心的音视频表示。大量实验表明，QUAG在HIREST上取得了SOTA结果。此外，我们在基于查询的视频摘要任务上测试QUAG，并验证了其良好的泛化能力。
英文摘要：
Video has emerged as a favored multimedia format on the internet. To better gain video contents, a new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. The pioneering work chooses the pre-trained CLIP-based model for video retrieval, and leverages it as a feature extractor for other three challenging tasks solved in a multi-task learning paradigm. Nevertheless, this work struggles to learn the comprehensive cognition of user-preferred content, due to disregarding the hierarchies and association relations across modalities. In this paper, guided by the shallow-to-deep principle, we propose a query-centric audio-visual cognition (QUAG) network to construct a reliable multi-modal representation for moment retrieval, segmentation and step-captioning. Specifically, we first design the modality-synergistic perception to obtain rich audio-visual content, by modeling global contrastive alignment and local fine-grained interaction between visual and audio modalities. Then, we devise the query-centric cognition that uses the deep-level query to perform the temporal-channel filtration on the shallow-level audio-visual representation. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks. Extensive experiments show QUAG achieves the SOTA results on HIREST. Further, we test QUAG on the query-based video summarization task and verify its good generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
