arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16307cs.MMcs.CR
收藏

多模态不可学习样本:保护数据免受多模态对比学习的影响

Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning

Xinwei Liu, Xiaojun Jia, Yuan Xun, Siyuan Liang, Xiaochun Cao

AI总结:

针对多模态对比学习的隐私保护问题,提出多步误差最小化方法MEM,通过联合优化图像噪声和文本触发器生成不可学习样本,有效误导模型学习捷径,实现高迁移性的数据保护。

摘要:

多模态对比学习(MCL)通过学习从互联网上抓取的数百万图文对,在零样本分类中展现出显著进步。然而,这种依赖带来了隐私风险,黑客可能未经授权地利用图文数据进行模型训练,其中可能包含个人和隐私敏感信息。近期研究提出通过向训练图像添加不可察觉的扰动来生成不可学习样本,以构建捷径进行保护。然而,这些方法是为单模态分类设计的,在MCL中很大程度上尚未被探索。我们首先通过评估现有方法在图文对上的性能来探索这一背景,发现它们不能有效泛化到多模态数据,并且由于MCL中缺乏标签以及数据对的分散性,在构建捷径方面表现出有限的影响。在本文中,我们提出了多步误差最小化(MEM),一种用于生成多模态不可学习样本的新型优化过程。它扩展了误差最小化(EM)框架,以同时优化图像噪声和额外的文本触发器,从而扩大优化空间,并有效地误导模型学习噪声特征与文本触发器之间的捷径。具体而言,我们采用投影梯度下降来解决噪声最小化问题,并使用HotFlip来近似梯度并替换单词以寻找最佳文本触发器。大量实验证明了MEM的有效性,受保护后的检索结果接近随机猜测的一半,并且在不同模型间具有高可迁移性。我们的代码可在https://github.com/thinwayliu/Multimodal-Unlearnable-Examples获取。

英文摘要:

Multimodal contrastive learning (MCL) has shown remarkable advances in zero-shot classification by learning from millions of image-caption pairs crawled from the Internet. However, this reliance poses privacy risks, as hackers may unauthorizedly exploit image-text data for model training, potentially including personal and privacy-sensitive information. Recent works propose generating unlearnable examples by adding imperceptible perturbations to training images to build shortcuts for protection. However, they are designed for unimodal classification, which remains largely unexplored in MCL. We first explore this context by evaluating the performance of existing methods on image-caption pairs, and they do not generalize effectively to multimodal data and exhibit limited impact to build shortcuts due to the lack of labels and the dispersion of pairs in MCL. In this paper, we propose Multi-step Error Minimization (MEM), a novel optimization process for generating multimodal unlearnable examples. It extends the Error-Minimization (EM) framework to optimize both image noise and an additional text trigger, thereby enlarging the optimized space and effectively misleading the model to learn the shortcut between the noise features and the text trigger. Specifically, we adopt projected gradient descent to solve the noise minimization problem and use HotFlip to approximate the gradient and replace words to find the optimal text trigger. Extensive experiments demonstrate the effectiveness of MEM, with post-protection retrieval results nearly half of random guessing, and its high transferability across different models. Our code is available on the https://github.com/thinwayliu/Multimodal-Unlearnable-Examples

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑