PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models
Department of Computer Science, University of Maryland（计算机科学系，马里兰大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机科学与数据科学学院）；Kahlert School of Computing, The University of Utah（犹他大学Kahlert计算学院）；Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院）
Lingzhi Yuan, Xinfeng Li, Chejian Xu, Guanhong Tao, Xiaojun Jia, Yihao Huang, Wei Dong, Yang Liu, Bo Li
备注：
Accepted for publication in IEEE Transactions on Information Forensics and Security (TIFS)
Recent text-to-image (T2I) models have exhibited remarkable performance in generating high-quality images from text descriptions. However, these models are vulnerable to misuse, particularly generating not-safe-for-work (NSFW) content, such as sexually explicit, violent, political, and disturbing images, raising serious ethical concerns. In this work, we present PromptGuard, a novel content moderation technique that draws inspiration from the system prompt mechanism in large language models (LLMs) for safety alignment. Unlike LLMs, T2I models lack a direct interface for enforcing behavioral guidelines. Our key idea is to optimize a safety soft prompt that functions as an implicit system prompt within the T2I model's textual embedding space. This universal soft prompt (P*) directly moderates NSFW inputs, enabling safe yet realistic image generation without affecting inference efficiency or requiring proxy models. We further enhance its reliability and helpfulness through a divide-and-conquer strategy that optimizes category-specific soft prompts and combines them into unified safety guidance. Extensive experiments across five datasets demonstrate that PromptGuard effectively mitigates NSFW content generation while preserving high-quality benign outputs. PromptGuard is 3.8 times faster than prior content moderation methods while outperforming eight state-of-the-art defenses. Evaluations using both a multi-head safety classifier and a VLM-based guardrail further confirm its robustness, with average unsafe ratios of 5.84% and 6.18%, respectively. Our code and dataset are available at https://t2i-promptguard.github.io/.
Patronus: Safeguarding Text-to-Image Models against White-Box Adversaries
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；College of Electrical Engineering and the Ubiquitous System Security Lab (USSLab), Zhejiang University（浙江大学电子工程学院和无处不在系统安全实验室）；ETH Zurich（苏黎世联邦理工学院）
RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Virginia Tech（弗吉尼亚理工大学）；Salesforce Research（Salesforce研究院）；University of California Berkeley（加州大学伯克利分校）；University of Chicago（芝加哥大学）
PromptSafe: Gated Prompt Tuning for Safe Text-to-Image Generation
Beihang University（北航）；Beijing University of Posts and Telecommunications（北京邮电大学）；Taishan University（泰山大学）；Nanyang Technological University（南洋理工大学）；Henan University of Science and Technology（河南科技大学）
On the Generation and Mitigation of Harmful Geometry in Image-to-3D Models
The Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））；Xidian University（西安电子科技大学）；Zhejiang University（浙江大学）；Wuhan University（武汉大学）
