Video generation models (VGMs) have demonstrated the capability to synthesize high-quality output. It is important to understand their potential to produce unsafe content, such as violent or terrifying videos. In this work, we provide a comprehensive understanding of unsafe video generation.
First, to confirm the possibility that these models could indeed generate unsafe videos, we choose unsafe content generation prompts collected from 4chan and Lexica, and three open-source SOTA VGMs to generate unsafe videos. After filtering out duplicates and poorly generated content, we created an initial set of 2112 unsafe videos from an original pool of 5607 videos. Through clustering and thematic coding analysis of these generated videos, we identify 5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic, Violent/Bloody, and Political. With IRB approval, we then recruit online participants to help label the generated videos. Based on the annotations submitted by 403 participants, we identified 937 unsafe videos from the initial video set. With the labeled information and the corresponding prompts, we created the first dataset of unsafe videos generated by VGMs.
We then study possible defense mechanisms to prevent the generation of unsafe videos. Existing defense methods in image generation focus on filtering either input prompt or output results. We propose a new approach called Latent Variable Defense (LVD), which works within the model's internal sampling process. LVD can achieve 0.90 defense accuracy while reducing time and computing resources by 10x when sampling a large number of unsafe prompts.
VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models
National Anti-Counterfeit Engineering Research Center, Huazhong University of Science and Technology(华中科技大学反伪工程研究中心);School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);Sydney AI Centre, The University of Sydney(悉尼大学AI中心)
Jailbreaking on Text-to-Video Models via Scene Splitting Strategy
Yonsei University(延世大学);Korea Institute of Science and Technology(韩国科学技术院);AIM Intelligence(AIM智能);Seoul National University(首尔国立大学);Kyung Hee University(庆熙大学)
Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
Shanghai Academy of AI for Science(上海人工智能实验室);Fudan University(复旦大学);Independent Researcher(独立研究者);Singapore University of Technology and Design(新加坡科技设计大学)
Two Frames Matter: A Temporal Attack for Text-to-Video Model Jailbreaking
College of Science, Mathematics and Technology, Wenzhou-Kean University(科学、数学与技术学院,温州市凯恩大学);State Key Laboratory of Complex & Critical Software Environment, Beihang University(复杂与关键软件环境国家重点实验室,北航);AI Security Lab(360人工智能安全实验室)
SPARK: Jailbreaking T2V Models by Synergistically Prompting Auditory and Recontextualized Knowledge
State Key Laboratory of Complex \& Critical Software Environment, Beihang University;College of Science, Mathematics;Technology, Wenzhou-Kean University;360 AI Security Lab;Faculty of Innovation Engineering, Macau University of Science;Hong Kong University of Science;University of Chinese Academy of Sciences