重新审视并红队测试个性化扩散模型中的保护扰动
Rethinking and Red-Teaming Protective Perturbation in Personalized Diffusion Models
AI总结：
本文通过快捷学习视角分析PDM微调过程，揭示对抗扰动导致的潜在空间错位问题，提出数据净化与对比解耦学习框架以提升保护性能。
摘要：
个性化扩散模型（PDMs）通过最小训练数据适应预训练文本到图像模型生成特定主题图像，但易受微小对抗扰动影响，导致在受污染数据集上微调时性能下降。现有净化方法试图红队测试保护扰动以打破保护，但常过度净化图像导致信息丢失。本文通过快捷学习视角深入分析PDMs微调过程，假设并实证对抗扰动在CLIP嵌入空间中导致图像与文本提示之间的潜在空间错位。这种错位使模型在微调时错误地将噪声模式与唯一标识符关联，导致泛化能力差。基于此，我们提出系统性的红队测试框架，包括数据净化和对比解耦学习。首先使用现成的图像修复技术重新对齐潜在空间中的图像与原始语义内容。然后引入带有噪声标记的对比解耦学习以解耦个性化概念与虚假噪声模式的学习。本研究不仅揭示了PDMs中的快捷学习漏洞，还提供了一个全面的评估框架用于开发更强的保护措施。我们的广泛评估显示其在现有净化方法和对抗扰动下的鲁棒性优势。
英文摘要：
Personalized diffusion models (PDMs) have become prominent for adapting pre-trained text-to-image models to generate images of specific subjects using minimal training data. However, PDMs are susceptible to minor adversarial perturbations, leading to significant degradation when fine-tuned on corrupted datasets. These vulnerabilities are exploited to create protective perturbations that prevent unauthorized image generation. Existing purification methods attempt to red-team the protective perturbation to break the protection but often over-purify images, resulting in information loss. In this work, we conduct an in-depth analysis of the fine-tuning process of PDMs through the lens of shortcut learning. We hypothesize and empirically demonstrate that adversarial perturbations induce a latent-space misalignment between images and their text prompts in the CLIP embedding space. This misalignment causes the model to erroneously associate noisy patterns with unique identifiers during fine-tuning, resulting in poor generalization. Based on these insights, we propose a systematic red-teaming framework that includes data purification and contrastive decoupling learning. We first employ off-the-shelf image restoration techniques to realign images with their original semantic content in latent space. Then, we introduce contrastive decoupling learning with noise tokens to decouple the learning of personalized concepts from spurious noise patterns. Our study not only uncovers shortcut learning vulnerabilities in PDMs but also provides a thorough evaluation framework for developing stronger protection. Our extensive evaluation demonstrates its advantages over existing purification methods and its robustness against adaptive perturbations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
