SyncMask：面向时尚领域的视觉-语言预训练同步注意力掩码
SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining
AI总结：
本文提出SyncMask方法，利用动量模型的跨注意力特征同步掩码图像块与词元中信息共现的位置，解决时尚领域视觉-语言数据中图文信息不匹配问题，并结合半困难负样本分组采样，在三个下游任务上超越现有方法。
摘要：
视觉-语言模型（VLMs）通过大规模成对数据集在跨模态理解方面取得了显著进展。然而，在时尚领域，数据集往往表现出图像与文本所传达信息之间的不一致性。这一问题源于数据集中包含单个时尚单品的多张图像，而所有这些图像仅与一段文本配对，导致某些文本细节在单张图像中不可见。这种不匹配——尤其是当非共现元素被掩码时——会削弱传统VLM目标（如掩码语言建模和掩码图像建模）的训练效果，从而阻碍模型准确对齐细粒度视觉与文本特征的能力。针对这一问题，我们提出了同步注意力掩码（SyncMask），该方法生成的掩码能够精确定位图像块和词元中信息在图像与文本中共同出现的位置。这种同步是通过利用从动量模型获得的跨注意力特征来实现的，确保两种模态之间的精确对齐。此外，我们通过半困难负样本增强了分组批次采样，有效缓解了时尚数据集中图像-文本匹配和图像-文本对比学习目标中的假阴性问题。我们的实验证明了所提方法的有效性，在三个下游任务中优于现有方法。
英文摘要：
Vision-language models (VLMs) have made significant strides in cross-modal understanding through large-scale paired datasets. However, in fashion domain, datasets often exhibit a disparity between the information conveyed in image and text. This issue stems from datasets containing multiple images of a single fashion item all paired with one text, leading to cases where some textual details are not visible in individual images. This mismatch, particularly when non-co-occurring elements are masked, undermines the training of conventional VLM objectives like Masked Language Modeling and Masked Image Modeling, thereby hindering the model's ability to accurately align fine-grained visual and textual features. Addressing this problem, we propose Synchronized attentional Masking (SyncMask), which generate masks that pinpoint the image patches and word tokens where the information co-occur in both image and text. This synchronization is accomplished by harnessing cross-attentional features obtained from a momentum model, ensuring a precise alignment between the two modalities. Additionally, we enhance grouped batch sampling with semi-hard negatives, effectively mitigating false negative issues in Image-Text Matching and Image-Text Contrastive learning objectives within fashion datasets. Our experiments demonstrate the effectiveness of the proposed approach, outperforming existing methods in three downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
