arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19149cs.CVcs.AIcs.CRcs.LG
收藏

针对文本到图像模型的多模态语用越狱

Multimodal Pragmatic Jailbreak on Text-to-image Models

LMU Munich, Germany(慕尼黑大学);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心);Saarland University, Germany(萨尔兰大学);Max Planck Institute for Informatics, Germany(马克斯·普朗克信息研究所);Cornell University, USA(康奈尔大学);University of Oxford, UK(牛津大学)

Tong Liu, Zhixin Lai, Jiawen Wang, Gengyuan Zhang, Shuo Chen, Philip Torr, Vera Demberg, Volker Tresp, Jindong Gu

AI总结:

本文提出多模态语用越狱方法,通过单独安全的图像与视觉文本组合触发 T2I 模型生成不安全内容,并构建数据集评测九款模型及现有过滤器的失效问题。

摘要:

扩散模型最近在图像质量和对文本提示的忠实度方面取得了显著进展。与此同时,此类生成模型的安全性已成为一个日益受到关注的领域。本工作介绍了一种新型越狱,它会触发 T2I 模型生成带有视觉文本的图像,其中图像和文本虽然单独来看都被认为是安全的,但结合起来却构成不安全内容。为系统地探究这一现象,我们提出了一个数据集,用于在此类越狱下评估当前基于扩散的文本到图像(T2I)模型。我们对九个具有代表性的 T2I 模型进行了基准测试,其中包括两个闭源商业模型。实验结果揭示了一种令人担忧的产生不安全内容的倾向:所有受测模型都会遭受此类越狱,不安全生成率在约 10% 到 70% 之间,其中 DALLE 3 表现出几乎最高的不安全性。在现实场景中,关键词黑名单、定制提示过滤器和 NSFW 图像过滤器等各种过滤器通常被用于缓解这些风险。我们评估了此类过滤器针对我们的越狱的有效性,并发现,尽管这些过滤器可能对单模态检测有效,但它们无法抵御我们的越狱。我们还从文本渲染能力和训练数据的角度研究了此类越狱的根本原因。我们的工作为进一步开发更安全、更可靠的 T2I 模型奠定了基础。项目页面位于 https://multimodalpragmatic.github.io/。

英文摘要:

Diffusion models have recently achieved remarkable advancements in terms of image quality and fidelity to textual prompts. Concurrently, the safety of such generative models has become an area of growing concern. This work introduces a novel type of jailbreak, which triggers T2I models to generate the image with visual text, where the image and the text, although considered to be safe in isolation, combine to form unsafe content. To systematically explore this phenomenon, we propose a dataset to evaluate the current diffusion-based text-to-image (T2I) models under such jailbreak. We benchmark nine representative T2I models, including two closed-source commercial models. Experimental results reveal a concerning tendency to produce unsafe content: all tested models suffer from such type of jailbreak, with rates of unsafe generation ranging from around 10\% to 70\% where DALLE 3 demonstrates almost the highest unsafety. In real-world scenarios, various filters such as keyword blocklists, customized prompt filters, and NSFW image filters, are commonly employed to mitigate these risks. We evaluate the effectiveness of such filters against our jailbreak and found that, while these filters may be effective for single modality detection, they fail to work against our jailbreak. We also investigate the underlying reason for such jailbreaks, from the perspective of text rendering capability and training data. Our work provides a foundation for further development towards more secure and reliable T2I models. Project page at https://multimodalpragmatic.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑