尽管现代文本到图像(T2I)模型在生成复杂提示的图像方面表现出色,但当输入为描述性段落时,它们难以捕捉关键细节。这种限制源于训练分布中简短标题的普遍性。现有方法试图通过在长提示上微调T2I模型或投影超大输入到正常提示空间来弥合这一差距。我们提出Prompt Refraction for Intricate Scene Modeling(PRISM),一种组合方法,使预训练T2I模型能够处理长序列输入。PRISM使用轻量模块从长提示中提取组成部分表示。T2I模型对每个组件独立进行噪声预测,然后通过能量基联合将输出合并到单个去噪步骤中。我们评估了PRISM在多种模型架构上的性能,显示其与在相同训练数据上微调的模型表现相当。此外,PRISM在挑战性公开基准上表现出优越的泛化能力,在超过500个标记的提示上,比基线模型高出7.4%。
英文摘要:
While modern text-to-image (T2I) models excel at generating images from intricate prompts, they struggle to capture the key details when the inputs are descriptive paragraphs. This limitation stems from the prevalence of concise captions that shape their training distributions. Existing methods attempt to bridge this gap by either fine-tuning T2I models on long prompts, which generalizes poorly to longer lengths; or by projecting the oversize inputs into normal-prompt space and compromising fidelity. We propose Prompt Refraction for Intricate Scene Modeling (PRISM), a compositional approach that enables pre-trained T2I models to process long sequence inputs. PRISM uses a lightweight module to extract constituent representations from the long prompts. The T2I model makes independent noise predictions for each component, and their outputs are merged into a single denoising step using energy-based conjunction. We evaluate PRISM across a wide range of model architectures, showing comparable performances to models fine-tuned on the same training data. Furthermore, PRISM demonstrates superior generalization, outperforming baseline models by 7.4% on prompts over 500 tokens in a challenging public benchmark.
Training-Free Text-to-Image Compositional Food Generation via Prompt Grafting
Elmore Family School of Electrical and Computer Engineering, Purdue University(电气与计算机工程学院,普渡大学);Department of Systems Design Engineering, University of Waterloo(系统设计工程系,滑铁卢大学)
When Composition Doesn't Add Up: Humans Identifying Defects in AI-Generated Images
Dundee International Institute of Central South University(中南大学邓迪国际学院);Central South University(中南大学);Faculty of Science, Engineering and Business, University of Dundee(邓迪大学科学、工程与商学院);University of Dundee(邓迪大学)
Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Models
Zhejiang University(浙江大学);Shanghai Institute for Advanced Study of Zhejiang University(浙江大学上海高等研究院);Endoscopy Center, Zhongshan Hospital, Fudan University(复旦大学中山医院内窥镜中心);Shanghai Collaborative Innovation Center of Endoscopy(上海内窥镜协同创新中心);Digital Medical Research Center, School of Basic Medical Sciences, Fudan University(复旦大学基础医学院数字医疗研究中心);Shanghai Key Laboratory of MICCAI(上海MICCAI重点实验室);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟商学院);Data Science Institute, Imperial College London(伦敦帝国学院数据科学研究所)
Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimization
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University(上海智能信息处理关键实验室,复旦大学计算机学院);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心)
Weave Forcing: Compositional Memory Routing for Interactive Long Video Generation
University of Electronic Science and Technology of China (UESTC)(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学)
When Safe Concepts Become Unsafe: Multi-Concept Compositional Vulnerabilities in Text-to-Image Models
School of Cyber Science and Engineering, Xi'an Jiaotong University(西安交通大学计算机科学与工程学院);CISPA Helmholtz Center for Information Security(信息安全研究中心);School of Cyber Science and Engineering, Wuhan University(武汉大学计算机科学与工程学院)
ORCA: Hunting Compositional Failures in Text-to-Image Diffusion
Wellcome Sanger Institute(威康桑格研究所);Cambridge Stem Cell Institute, University of Cambridge(剑桥大学剑桥干细胞研究所);Cambridge Centre for AI in Medicine, University of Cambridge(剑桥大学剑桥人工智能医学中心);University of Oxford(牛津大学);Apple(苹果公司);University of Helsinki(赫尔辛基大学);Institute for Molecular Medicine Finland (FIMM), University of Helsinki(赫尔辛基大学芬兰分子医学研究所);School of Computing and Information Systems, University of Melbourne(墨尔本大学计算与信息系统学院);Department of Medicine, University of Cambridge(剑桥大学医学系)