DuMo:用于精准概念擦除的双编码器调制网络
DuMo: Dual Encoder Modulation Network for Precise Concept Erasure
AI总结:
DuMo通过冻结骨干U-NET并利用EPR模块修改跳跃连接特征,结合TLMO动态调节不同层与时间步的擦除尺度,在精准擦除目标概念的同时最大程度保留非目标概念,并在多项概念擦除任务中取得最优性能。
摘要:
文本到图像模型卓越的生成能力引发了关于生成不安全(Not-Safe-For-Work,NSFW)内容以及潜在版权侵权的重大安全担忧。为了解决这些问题,以往的方法通过消除不适当的概念来保护模型。然而,这些模型会改变骨干网络的参数,并对图像的结构(低频)成分产生显著影响,从而削弱模型保留非目标概念的能力。在本工作中,我们提出了双编码器调制网络(Dual encoder Modulation network,DuMo),它能够精准擦除不适当的目标概念,同时对非目标概念的损害最小。与以往方法不同,DuMo采用带先验知识的擦除器(Eraser with PRior Knowledge,EPR)模块,该模块修改U-NET的跳跃连接特征,并主要在图像的细节(高频)成分上实现概念擦除。为了在擦除过程中尽量减少对非目标概念的破坏,骨干U-NET的参数被冻结,并且来自原始跳跃连接特征的先验知识被引入到擦除过程中。同时,我们观察到EPR在不同时间步和不同层上对图像结构和细节表现出不同的擦除偏好。因此,我们采用了一种新颖的时间-层调制过程(Time-Layer MOdulation process,TLMO),它调整EPR模块输出在不同层和时间步上的擦除尺度,自动平衡擦除效果与模型生成能力。我们的方法在显式内容擦除、卡通概念移除和艺术风格擦除上取得了最先进的性能,明显优于其他替代方法。代码可在https://github.com/Maplebb/DuMo获取。
英文摘要:
The exceptional generative capability of text-to-image models has raised substantial safety concerns regarding the generation of Not-Safe-For-Work (NSFW) content and potential copyright infringement. To address these concerns, previous methods safeguard the models by eliminating inappropriate concepts. Nonetheless, these models alter the parameters of the backbone network and exert considerable influences on the structural (low-frequency) components of the image, which undermines the model's ability to retain non-target concepts. In this work, we propose our Dual encoder Modulation network (DuMo), which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. In contrast to previous methods, DuMo employs the Eraser with PRior Knowledge (EPR) module which modifies the skip connection features of the U-NET and primarily achieves concept erasure on details (high-frequency) components of the image. To minimize the damage to non-target concepts during erasure, the parameters of the backbone U-NET are frozen and the prior knowledge from the original skip connection features is introduced to the erasure process. Meanwhile, the phenomenon is observed that distinct erasing preferences for the image structure and details are demonstrated by the EPR at different timesteps and layers. Therefore, we adopt a novel Time-Layer MOdulation process (TLMO) that adjusts the erasure scale of EPR module's outputs across different layers and timesteps, automatically balancing the erasure effects and model's generative ability. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods. Code is available at https://github.com/Maplebb/DuMo
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。