Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to ``direct'' the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to providing the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces ``activation'' at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement.
AdsorbFlow: energy-conditioned flow matching enables fast and realistic adsorbate placement
Beijing Key Laboratory of Artificial Intelligence for Advanced Chemical Engineering Materials, Department of Chemical Engineering, Tsinghua University(北京人工智能先进化工材料重点实验室,化学工程系,清华大学)
MAGIC: Few-Shot Mask-Guided Anomaly Inpainting with Prompt Perturbation, Spatially Adaptive Guidance, and Context Awareness
Department of AI Semiconductor Engineering, Hanyang University, Seoul, Korea(汉阳大学人工智能半导体工程系);Department of Electronic Engineering, Hanyang University, Seoul, Korea(汉阳大学电子工程系)
PIS: A Generalized Physical Inversion Solver for Arbitrary Sparse Observations via Set Conditioned Flow Matching
School of Information, University of California, Berkeley, USA(信息学院,加州大学伯克利分校);College of Civil Engineering, Tongji University, Shanghai, China(土木学院,同济大学);Institute of Mechanical Sciences(机械科学研究院)
TD3B: Transition-Directed Discrete Diffusion for Allosteric Binder Generation
Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);Department of Bioengineering, University of Pennsylvania(宾夕法尼亚大学生物工程系);Department of Computer and Information Science, University of Pennsylvania(宾夕法尼亚大学计算机与信息科学系);Centre for Computational Biology, Duke-NUS Medical School(杜克-新加坡国立大学医学学校计算生物学中心)
DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
Shenzhen Campus of Sun Yat-Sen University(中山大学深圳校区);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Shenzhen Campus of the Chinese University of Hong Kong(香港中文大学深圳校区)
ProteinGuide: On-the-fly property guidance for protein sequence generative models
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley(电气工程与计算机科学系,加州大学伯克利分校);Center for Computational Biology, University of California, Berkeley(计算生物学中心,加州大学伯克利分校);Department of Molecular and Cell Biology, University of California, Berkeley(分子与细胞生物学系,加州大学伯克利分校);Innovative Genomics Institute, University of California, Berkeley(创新基因组研究所,加州大学伯克利分校);Howard Hughes Medical Institute, University of California, Berkeley(霍华德·休斯医学研究所,加州大学伯克利分校)
Spine Vision X-Ray Image based GUI Planning of Pedicle Screws Using Enhanced YOLOv5 for Vertebrae Segmentation
Healthcare Technology Innovation Centre (HTIC), Indian Institute of Technology Madras(印度理工学院马德拉斯分校医疗技术创新中心);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);Department of Electrical Engineering, Indian Institute of Technology Madras(印度理工学院马德拉斯分校电气工程系)
Point2Insert: Video Object Insertion via Sparse Point Guidance
Institute of Artificial Intelligence, China Telecom (TeleAI)(人工智能研究院,中国电信(TeleAI));Sun Yat-sen University(孙中山大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Fudan University(复旦大学)