T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
ARC Lab, Tencent PCG(腾讯平台与内容事业群ARC实验室);University of Macau(澳门大学);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Peking University Shenzhen Graduate School(北京大学深圳研究生院)
The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.g., color and structure) is needed. In this paper, we aim to ``dig out" the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly. Specifically, we propose to learn simple and lightweight T2I-Adapters to align internal knowledge in T2I models with external control signals, while freezing the original large T2I models. In this way, we can train various adapters according to different conditions, achieving rich control and editing effects in the color and structure of the generation results. Further, the proposed T2I-Adapters have attractive properties of practical value, such as composability and generalization ability. Extensive experiments demonstrate that our T2I-Adapter has promising generation quality and a wide range of applications.
AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Models
VCIP, School of Computer Science, Nankai University(VCIP,计算机学院,南开大学);The Chinese University of Hong Kong(香港中文大学);vivo Mobile Communication Co., Ltd(vivo移动通信有限公司)
CAR: Controllable Autoregressive Modeling for Visual Generation
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);Tencent Youtu Lab(腾讯优图实验室);Southern University of Science and Technology(南方科技大学);University of Washington(华盛顿大学)
FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editing
Xi’an Key Laboratory of Big Data and Intelligent Vision, Xidian University, Xi’an 710071, China(西安大数据与智能视觉重点实验室,西安电子科技大学,西安710071,中国);Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China(协同智能系统重点实验室,教育部,西安电子科技大学,西安710071,中国);School of Computer Science and Technology, Xidian University, Xi’an 710071, China(计算机科学与技术学院,西安电子科技大学,西安710071,中国)
VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers
Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University(海德堡大学医学院心血管医学人工智能研究所);Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital(海德堡大学医院心脏病学、血管学、肺病学系);DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim(德国心血管研究中心(DZHK)海德堡/曼海姆合作站点);University Medical Center of the Johannes Gutenberg-University Mainz(美因茨约翰内斯·古腾堡大学医学中心)
eBIRD: Event-based Intensity Image Reconstruction Using Controllable Diffusion Models
Institute of Engineering Sciences, Universidad de O’Higgins(奥伊金斯大学工程科学学院);The Iniciativa de Datos e Inteligencia Artificial, University of Chile(智利大学数据与人工智能倡议)
Language-Grounded Multi-Domain Image Translation via Semantic Difference Guidance
Department of Artificial Intelligence, Chung-Ang University(人工智能系, Chung-Ang 大学);Hyundai Mobis;Korea Advanced Institute of Science and Technology(韩国科学技术院)
ECGTwin: Personalized ECG Generation Using Controllable Diffusion Model
State Key Laboratory of General Artificial Intelligence, Beijing, China(一般人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University, Beijing, China(智能科学与技术学院);National Institute of Health Data Science, Peking University, Beijing, China(健康数据科学国家研究院);Department of Cardiology, Peking University People’s Hospital, Beijing, China(心血管内科)
Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulation
University of Washington, Seattle, WA 98195, USA(华盛顿大学);Nanjing University of Posts and Telecommunications, Jiangsu, Nanjing, China(南京邮电大学);Nanjing Foreign Language School, Jiangsu, Nanjing, China(南京外国语学校)