arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.12581cs.CRcs.AIcs.CVcs.CY
收藏

理解不安全视频生成

Towards Understanding Unsafe Video Generation

University of Virginia(弗吉尼亚大学);Penn State University(宾夕法尼亚州立大学);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全研究中心)

Yan Pang, Aiping Xiong, Yang Zhang, Tianhao Wang

AI总结:

本文系统研究了视频生成模型的不安全内容生成问题,构建了首个不安全视频数据集,并提出在模型内部采样过程中起作用的潜在变量防御方法(LVD),实现0.90防御准确率且资源消耗降低10倍。

摘要:

视频生成模型(VGMs)已展现出合成高质量输出的能力。理解其生成不安全内容(如暴力或恐怖视频)的潜在可能性至关重要。在本工作中,我们对不安全视频生成进行了全面理解。\n首先,为确认这些模型确实可能生成不安全视频,我们选取了从4chan和Lexica收集的不安全内容生成提示,并使用三个开源最先进(SOTA)视频生成模型来生成不安全视频。在过滤掉重复和生成质量差的内容后,我们从原始5607个视频中创建了包含2112个不安全视频的初始集合。通过对这些生成视频进行聚类和主题编码分析,我们识别出5类不安全视频:扭曲/怪异、恐怖、色情、暴力/血腥和政治。经IRB批准,我们随后招募在线参与者帮助标注生成的视频。基于403名参与者提交的标注,我们从初始视频集中识别出937个不安全视频。利用标注信息及对应的提示,我们创建了首个由视频生成模型生成的不安全视频数据集。\n随后,我们研究了防止生成不安全视频的可能防御机制。现有图像生成领域的防御方法侧重于过滤输入提示或输出结果。我们提出了一种新方法,称为潜在变量防御(LVD),该方法在模型内部的采样过程中起作用。LVD在采样大量不安全提示时,可实现0.90的防御准确率,同时将时间和计算资源消耗减少10倍。

英文摘要:

Video generation models (VGMs) have demonstrated the capability to synthesize high-quality output. It is important to understand their potential to produce unsafe content, such as violent or terrifying videos. In this work, we provide a comprehensive understanding of unsafe video generation. First, to confirm the possibility that these models could indeed generate unsafe videos, we choose unsafe content generation prompts collected from 4chan and Lexica, and three open-source SOTA VGMs to generate unsafe videos. After filtering out duplicates and poorly generated content, we created an initial set of 2112 unsafe videos from an original pool of 5607 videos. Through clustering and thematic coding analysis of these generated videos, we identify 5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic, Violent/Bloody, and Political. With IRB approval, we then recruit online participants to help label the generated videos. Based on the annotations submitted by 403 participants, we identified 937 unsafe videos from the initial video set. With the labeled information and the corresponding prompts, we created the first dataset of unsafe videos generated by VGMs. We then study possible defense mechanisms to prevent the generation of unsafe videos. Existing defense methods in image generation focus on filtering either input prompt or output results. We propose a new approach called Latent Variable Defense (LVD), which works within the model's internal sampling process. LVD can achieve 0.90 defense accuracy while reducing time and computing resources by 10x when sampling a large number of unsafe prompts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑