掩码重建：视频-文本检索的协同语义补全
Mask to reconstruct: Cooperative Semantics Completion for Video-text Retrieval
AI总结：
提出基于语义掩码建模的MASCOT方法，通过注意力掩码、知情语义补全和双掩码协同学习，在四个视频-文本检索基准上达到最先进性能。
摘要：
近年来，掩码视频建模被广泛探索，并在局部层面显著提升了模型对视觉区域的理解能力。然而，现有方法通常采用随机掩码并遵循相同的重建范式来补全掩码区域，未能利用跨模态内容之间的相关性。本文提出基于语义掩码建模的语义补全掩码（MASCOT）。具体而言，在应用基于注意力的视频掩码生成高信息量和低信息量掩码后，我们提出知情语义补全来恢复掩码语义信息。该恢复机制通过将掩码内容与未掩码视觉区域及相应文本上下文对齐来实现，使模型在补丁层面捕获更多与文本相关的细节。此外，我们将重建重点从无关背景转移到判别性部分，以忽略低信息量掩码区域。进一步地，我们设计双掩码协同学习，以融合不同掩码下的视频线索并学习更对齐的视频表示。我们的MASCOT在四个主要文本-视频检索基准（包括MSR-VTT、LSMDC、ActivityNet和DiDeMo）上取得了最先进的性能。大量消融研究证明了所提方案的有效性。
英文摘要：
Recently, masked video modeling has been widely explored and significantly improved the model's understanding ability of visual regions at a local level. However, existing methods usually adopt random masking and follow the same reconstruction paradigm to complete the masked regions, which do not leverage the correlations between cross-modal content. In this paper, we present Mask for Semantics Completion (MASCOT) based on semantic-based masked modeling. Specifically, after applying attention-based video masking to generate high-informed and low-informed masks, we propose Informed Semantics Completion to recover masked semantics information. The recovery mechanism is achieved by aligning the masked content with the unmasked visual regions and corresponding textual context, which makes the model capture more text-related details at a patch level. Additionally, we shift the emphasis of reconstruction from irrelevant backgrounds to discriminative parts to ignore regions with low-informed masks. Furthermore, we design dual-mask co-learning to incorporate video cues under different masks and learn more aligned video representation. Our MASCOT performs state-of-the-art performance on four major text-video retrieval benchmarks, including MSR-VTT, LSMDC, ActivityNet, and DiDeMo. Extensive ablation studies demonstrate the effectiveness of the proposed schemes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
