With recent advancements in diffusion models, users can generate high-quality images by writing text prompts in natural language. However, generating images with desired details requires proper prompts, and it is often unclear how a model reacts to different prompts or what the best prompts are. To help researchers tackle these critical challenges, we introduce DiffusionDB, the first large-scale text-to-image prompt dataset totaling 6.5TB, containing 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. We analyze the syntactic and semantic characteristics of prompts. We pinpoint specific hyperparameter values and prompt styles that can lead to model errors and present evidence of potentially harmful model usage, such as the generation of misinformation. The unprecedented scale and diversity of this human-actuated dataset provide exciting research opportunities in understanding the interplay between prompts and generative models, detecting deepfakes, and designing human-AI interaction tools to help users more easily use these models. DiffusionDB is publicly available at: https://poloclub.github.io/diffusiondb.
EDITOR: Effective and Interpretable Prompt Inversion for Text-to-Image Diffusion Models
University of Massachusetts, Amherst(马萨诸塞大学阿姆赫斯特分校);Georgia Institute of Technology(佐治亚理工学院);Rutgers University(罗格斯大学);University of Utah(犹他大学);Dolby Laboratories(杜比实验室)
Couple to Control: Joint Initial Noise Design in Diffusion Models
Department of Computer Science(计算机科学系);Rutgers University(罗格斯大学);Department of EECS(电子工程与计算机科学系);University of Michigan(密歇根大学);Department of Statistics(统计学系)
MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping
Tencent(腾讯);Nanyang Technological University(南洋理工大学);Hong Kong University of Science and Technology(香港科技大学);Fuzhou University(福州大学);University of Hong Kong(香港大学);National University of Singapore(新加坡国立大学)
SORCE: Small Object Retrieval in Complex Environments
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);Sensetime Research(商汤科技研究院);Tongji University(同济大学);Beijing Institute of Technology(北京理工大学);Shanghai AI Lab(上海人工智能实验室)
Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval
School of Cyber Science and Engineering, Nanjing University of Science and Technology(南京理工大学网络空间安全学院);State Key Laboratory of Media Convergence and Communication, Communication University of China(中国传媒大学媒体融合与传播国家重点实验室);Lenovo Research(联想研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Laboratory for Advanced Computing and Intelligence Engineering(先进计算与智能工程实验室);College of Computer Science, Nankai University(南开大学计算机学院)
Lung-CADex: Fully automatic Zero-Shot Detection and Classification of Lung Nodules in Thoracic CT Images
Faculty of Electronics and Electrical Engineering, University of Engineering and Technology, Taxila(塔克西拉工程技术大学电子与电气工程学院);Center for Research in Computer Vision, University of Central Florida(中佛罗里达大学计算机视觉研究中心);Sheikh Zayed Institute for Pediatric Surgical Innovation, Childrens National Hospital(儿童国家医院谢赫扎耶德儿科外科创新研究所);School of Medicine and Health Sciences, George Washington University(乔治华盛顿大学医学与健康科学学院)