arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.05066cs.CV
收藏

美丽的图像,有毒的词语:理解并解决生成图像中的冒犯性文本

Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in Generated Images

Aditya Kumar, Tom Blanchard, Adam Dziedzic, Franziska Boenisch

AI总结:

本文提出了一种针对生成图像中冒犯性文本的微调策略,并发布了ToxicBench基准,用于评估和改进文本到图像模型的安全性。

摘要:

最先进的扩散模型(DMs)能够生成高度逼真的图像。尽管先前的研究在视觉领域成功地减轻了不适宜内容(NSFW),但我们发现了一个新的威胁:在图像中生成包含NSFW内容的文本。这包括侮辱性语言、种族歧视用语和性暗示用语,对用户构成重大风险。我们证明所有最先进的DMs(例如SD3、SDXL、Flux、DeepFloyd IF)都容易受到这一问题的影响。通过广泛的实验,我们展示了现有的缓解技术在视觉内容上有效,但在防止有害文本生成的同时显著降低了良性文本生成的质量。作为解决这一威胁的初步步骤,我们介绍了一种新的微调策略,该策略仅针对DMs中的文本生成层。因此,我们构建了一个安全微调数据集,通过将每个NSFW提示与两幅图像配对:一幅包含NSFW术语,另一幅则将该术语替换为精心设计的无害替代词,同时保持图像其余部分不变。通过在该数据集上训练,模型学会避免生成有害文本,同时保持良性内容和整体图像质量。最后,为了推动该领域的研究,我们发布了ToxicBench,一个开源的评估基准,用于评估图像中的NSFW文本生成。它包括我们精心挑选的微调数据集、一组有害提示、新的评估指标和一个评估NSFW性以及文本和图像质量的流程。我们的基准旨在指导未来在文本到图像模型中减轻NSFW文本生成的努力,从而促进其安全部署。

英文摘要:

State-of-the-art Diffusion Models (DMs) produce highly realistic images. While prior work has successfully mitigated Not Safe For Work (NSFW) content in the visual domain, we identify a novel threat: the generation of NSFW text embedded within images. This includes offensive language, such as insults, racial slurs, and sexually explicit terms, posing significant risks to users. We show that all state-of-the-art DMs (e.g., SD3, SDXL, Flux, DeepFloyd IF) are vulnerable to this issue. Through extensive experiments, we demonstrate that existing mitigation techniques, effective for visual content, fail to prevent harmful text generation while substantially degrading benign text generation. As an initial step toward addressing this threat, we introduce a novel fine-tuning strategy that targets only the text-generation layers in DMs. Therefore, we construct a safety fine-tuning dataset by pairing each NSFW prompt with two images: one with the NSFW term, and another where that term is replaced with a carefully crafted benign alternative while leaving the image unchanged otherwise. By training on this dataset, the model learns to avoid generating harmful text while preserving benign content and overall image quality. Finally, to advance research in the area, we release ToxicBench, an open-source benchmark for evaluating NSFW text generation in images. It includes our curated fine-tuning dataset, a set of harmful prompts, new evaluation metrics, and a pipeline that assesses both NSFW-ness and text and image quality. Our benchmark aims to guide future efforts in mitigating NSFW text generation in text-to-image models, thereby contributing to their safe deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑