MLIP：基于穷尽数据利用的高效多视角语言-图像预训练
MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization
AI总结：
针对CLIP数据利用效率低及计算成本高的问题，提出MLIP框架，通过引入频率变换与token级对齐扩展监督信号，并采用语义引导的token合并方法加速模型，有效提升了表征学习效率。
摘要：
对比语言-图像预训练（CLIP）取得了显著成功，推动了多模态研究的快速发展。然而，CLIP在数据利用效率低下方面面临显著挑战。它在表征学习期间依赖每个图文对的单一对比监督，忽略了大量可提供更丰富监督的有价值信息。此外，保留非信息性token会导致计算需求和时间成本增加，特别是在CLIP的ViT图像编码器中。为了解决这些问题，我们提出了多视角语言-图像预训练（MLIP）。在MLIP中，我们利用频率变换对高频和低频变化的敏感性，这补充了空间域仅对低频变化的敏感性。通过结合频率变换和token级对齐，我们将CLIP的单一监督扩展为多域多级监督，从而能够更彻底地探索信息性图像特征。此外，我们引入了一种由频率和空间域综合语义引导的token合并方法。这使我们能够以可控的压缩率将token合并为多粒度token，从而加速CLIP。大量实验验证了我们设计的有效性。
英文摘要：
Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success, leading to rapid advancements in multimodal studies. However, CLIP faces a notable challenge in terms of inefficient data utilization. It relies on a single contrastive supervision for each image-text pair during representation learning, disregarding a substantial amount of valuable information that could offer richer supervision. Additionally, the retention of non-informative tokens leads to increased computational demands and time costs, particularly in CLIP's ViT image encoder. To address these issues, we propose Multi-Perspective Language-Image Pretraining (MLIP). In MLIP, we leverage the frequency transform's sensitivity to both high and low-frequency variations, which complements the spatial domain's sensitivity limited to low-frequency variations only. By incorporating frequency transforms and token-level alignment, we expand CILP's single supervision into multi-domain and multi-level supervision, enabling a more thorough exploration of informative image features. Additionally, we introduce a token merging method guided by comprehensive semantics from the frequency and spatial domains. This allows us to merge tokens to multi-granularity tokens with a controllable compression rate to accelerate CLIP. Extensive experiments validate the effectiveness of our design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
