SafeGen-Bench: 图像条件文本到视频生成中的安全性基准测试
SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation
AI总结:
针对图像条件文本到视频生成中安全文本和图像组合仍可能产生有害内容的问题,提出SafeGen-Bench基准,定义10个恶意类别并评估现有模型,发现当前模型难以避免生成恶意内容,且单模态护栏防御不足。
摘要:
随着文本到图像扩散模型的快速发展,像Sora这样的生成式视频模型(T2V模型)现在可以从文本提示或初始图像生成短视频。然而,合成视频生成——尤其是在初始图像引导下——常常带来风险,包括可能创建非法、政治敏感或不道德的内容。现有基准已开始考虑生成视频的安全性,但它们主要关注用恶意文本提示测试模型,忽略了文本提示和图像组合仍可能导致有害视频内容的场景。在实践中,这是一个常见且具有挑战性的问题:从安全文本和图像输入生成的视频仍可能传达有害信息。为弥补这一差距,我们引入了SafeGen-Bench,一个专门设计用于评估条件T2V模型安全性的基准。我们的基准定义了10个恶意类别,重点关注与时间序列和描绘行为相关的风险。SafeGen-Bench包含从多样图像和视频源中精心选择的起始帧,并配以相应的文本提示以模拟真实输入。我们在SafeGen-Bench上评估了多种条件T2V模型,结果表明当前模型难以持续避免生成恶意内容,不安全分数高达44.5,尤其是在需要高质量的条件下。此外,我们评估了基于文本和基于图像的护栏在我们的基准上的有效性,发现单模态护栏单独不足以提供稳健防御,在七个恶意类别中失败率达80%。我们希望SafeGen-Bench能促进更安全、更可控的条件T2V模型的开发。
英文摘要:
With the rapid advancements in text-to-image diffusion models, generative video models (T2V models) like Sora can now produce short synthetic videos from a text prompt or an initial image. However, synthetic video generation -- especially when guided by an initial image -- often poses risks, including the potential creation of illegal, politically sensitive, or unethical content. Existing benchmarks have started to consider the safety of generated videos, but they primarily focus on testing models with malicious text prompts, ignoring the scenario where text prompt and image combination may still lead to harmful video content. In practice, this is a common and challenging issue: videos generated from safe text and image inputs can nonetheless convey harmful information. To bridge this gap, we introduce SafeGen-Bench, a benchmark specifically designed to evaluate the safety of conditional T2V models. Our benchmark defines 10 malicious categories, concentrating on risks related to both temporal sequences and depicted behaviors. SafeGen-Bench consists of carefully selected start frames from diverse image and video sources, paired with corresponding text prompts to simulate realistic inputs. We evaluate a variety of conditional T2V models on SafeGen-Bench, and the results indicate that current models struggle to consistently avoid generating malicious content with unsafety scores reaching up to 44.5, especially under conditions requiring high quality. Furthermore, we assess the effectiveness of both text-based and image-based guardrails on our benchmark, finding that unimodal guardrails alone were insufficient to provide a robust defense, with an 80\% failure rate across seven malicious categories. We hope that SafeGen-Bench will foster the development of safer and more controllable conditional T2V models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。