arXiv 2311.164452026-08-09cs.CV

CLAP：通过增强提示的对比学习实现内容与风格的分离

CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts

Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）

Yichao Cai, Yuhang Liu, Zhen Zhang, Javen Qinfeng Shi

备注：: Accepted as a conference paper at ECCV 2024; v7: Minor corrections to the adversarial robustness results and corresponding text. Conclusions unchanged

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对CLIP等对比视觉语言模型特征混合内容与风格、分布偏移下泛化受限的问题，提出结合图像与文本增强的CLAP对比学习方法，解耦内容与风格特征，有效提升零样本、少样本分类性能及扰动鲁棒性。

摘要：

对比视觉语言模型（如CLIP）因所学习特征的出色泛化能力，在各类下游任务中受到广泛关注。然而，它们学习到的特征往往混合了内容与风格信息，这在一定程度上限制了其在分布偏移下的泛化能力。为解决这一局限，我们从多模态数据的因果生成视角出发，提出结合数据增强的对比学习方法，从原始表征中解耦出内容特征。为此，我们首先探索了图像增强技术，并开发了一种将其无缝集成到预训练类CLIP模型中的方法，以提取纯净的内容特征。更进一步，鉴于文本数据固有的语义丰富性与逻辑结构，我们探索利用文本增强将潜在内容与风格特征分离。这使得类CLIP模型的编码器能够聚焦于潜在内容信息，优化预训练类CLIP模型所学习的表征。我们在多个数据集上开展的大量实验表明，该方法在零样本和少样本分类任务中取得了显著提升，同时增强了对各类扰动的鲁棒性。这些结果凸显了我们提出的方法在优化视觉语言表征、推动多模态学习前沿发展方面的有效性。

英文摘要：

Contrastive vision-language models, such as CLIP, have garnered considerable attention for various downstream tasks, mainly due to the remarkable ability of the learned features for generalization. However, the features they learned often blend content and style information, which somewhat limits their generalization capabilities under distribution shifts. To address this limitation, we adopt a causal generative perspective for multimodal data and propose contrastive learning with data augmentation to disentangle content features from the original representations. To achieve this, we begin with exploring image augmentation techniques and develop a method to seamlessly integrate them into pre-trained CLIP-like models to extract pure content features. Taking a step further, recognizing the inherent semantic richness and logical structure of text data, we explore the use of text augmentation to isolate latent content from style features. This enables CLIP-like model's encoders to concentrate on latent content information, refining the learned representations by pre-trained CLIP-like models. Our extensive experiments across diverse datasets demonstrate significant improvements in zero-shot and few-shot classification tasks, alongside enhanced robustness to various perturbations. These results underscore the effectiveness of our proposed methods in refining vision-language representations and advancing the state-of-the-art in multimodal learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。