通过分布建模实现文本到图像生成系统可扩展的红队测试
DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modeling
AI总结：
DREAM通过分布建模实现文本到图像生成系统的可扩展红队测试，有效发现多样化的有害提示，提升安全性和多样性。
摘要：
尽管集成了安全对齐和外部过滤器，文本到图像（T2I）生成系统仍然容易生成有害内容，如色情或暴力图像。这引发了对意外暴露和潜在滥用的严重担忧。红队测试，旨在主动识别能够从T2I系统中引发不安全输出的多样化提示，正日益被认可为在真实世界部署前评估和改进安全性的关键方法。然而，现有的自动化红队测试方法往往将提示发现视为一个孤立的、提示层面的优化任务，这限制了其可扩展性、多样性和整体效果。为弥合这一差距，本文提出DREAM，一种可扩展的红队测试框架，以自动从给定的T2I系统中发现多样化的有问题提示。与先前工作优化提示个别不同，DREAM直接建模目标系统有问题提示的概率分布，这使得在有效性和多样性上进行显式优化，并在训练后允许高效的大规模采样。为了在不直接访问代表性训练样本的情况下实现这一点，我们借鉴了能量模型并重新公式化目标为一个简单且可处理的形式。我们进一步引入GC-SPSA，一种高效的优化算法，通过长且可能非可微的T2I流水线提供稳定的梯度估计。在推理过程中，我们还提出了一种多样性意识的采样策略以增强提示的多样性。DREAM的有效性通过广泛的实验得到了验证，证明了其在广泛的T2I模型和安全过滤器中，无论是提示成功率还是多样性方面都表现出最先进的性能。我们的代码可在https://github.com/AntigoneRandy/DREAM上获得。
英文摘要：
Despite the integration of safety alignment and external filters, text-to-image (T2I) generative systems are still susceptible to producing harmful content, such as sexual or violent imagery. This raises serious concerns about unintended exposure and potential misuse. Red teaming, which aims to proactively identify diverse prompts that can elicit unsafe outputs from the T2I system, is increasingly recognized as an essential method for assessing and improving safety before real-world deployment. However, existing automated red teaming approaches often treat prompt discovery as an isolated, prompt-level optimization task, which limits their scalability, diversity, and overall effectiveness. To bridge this gap, in this paper, we propose DREAM, a scalable red teaming framework to automatically uncover diverse problematic prompts from a given T2I system. Unlike prior work that optimizes prompts individually, DREAM directly models the probabilistic distribution of the target system's problematic prompts, which enables explicit optimization over both effectiveness and diversity, and allows efficient large-scale sampling after training. To achieve this without direct access to representative training samples, we draw inspiration from energy-based models and reformulate the objective into a simple and tractable form. We further introduce GC-SPSA, an efficient optimization algorithm that provides stable gradient estimates through the long and potentially non-differentiable T2I pipeline. During inference, we also propose a diversity-aware sampling strategy to enhance prompt variety. The effectiveness of DREAM is validated through extensive experiments, demonstrating state-of-the-art performance across a wide range of T2I models and safety filters in terms of both prompt success rate and diversity. Our code is available at https://github.com/AntigoneRandy/DREAM
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
