Taking advantage of the many recent advances in deep learning, text-to-image generative models currently have the merit of attracting the general public attention. Two of these models, DALL-E 2 and Imagen, have demonstrated that highly photorealistic images could be generated from a simple textual description of an image. Based on a novel approach for image generation called diffusion models, text-to-image models enable the production of many different types of high resolution images, where human imagination is the only limit.
However, these models require exceptionally large amounts of computational resources to train, as well as handling huge datasets collected from the internet. In addition, neither the codebase nor the models have been released. It consequently prevents the AI community from experimenting with these cutting-edge models, making the reproduction of their results complicated, if not impossible.
In this thesis, we aim to contribute by firstly reviewing the different approaches and techniques used by these models, and then by proposing our own implementation of a text-to-image model. Highly based on DALL-E 2, we introduce several slight modifications to tackle the high computational cost induced. We thus have the opportunity to experiment in order to understand what these models are capable of, especially in a low resource regime. In particular, we provide additional and analyses deeper than the ones performed by the authors of DALL-E 2, including ablation studies.
Besides, diffusion models use so-called guidance methods to help the generating process. We introduce a new guidance method which can be used in conjunction with other guidance methods to improve image quality. Finally, the images generated by our model are of reasonably good quality, without having to sustain the significant training costs of state-of-the-art text-to-image models.
Oxford-Man Institute of Quantitative Finance, University of Oxford(牛津大学牛津-曼数量金融研究所);Mathematical Institute, University of Oxford(牛津大学数学研究所);University of Cambridge(剑桥大学);Imperial College London(伦敦帝国学院);Royal Brompton Hospital(皇家布朗普顿医院);King’s College London(伦敦国王学院)
Provable Separations between Memorization and Generalization in Diffusion Models
Department of Industrial Engineering and Management Sciences, Northwestern University(工业工程与管理科学系,西北大学);Department of Statistics and Data Science, Northwestern University(统计与数据科学系,西北大学);School of Mathematics, Georgia Institute of Technology(数学系,佐治亚理工学院)
Visual Car Brand Classification by Implementing a Synthetic Image Dataset Creation Pipeline
TH OWL University of Applied Sciences and Arts(TH OWL应用科学与艺术大学);Fraunhofer IOSB-INA(弗劳恩霍夫IOSB-INA研究所);Institute for Industrial Information Technology (inIT)(工业信息技术研究所(inIT))
DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models
Autonomous Intelligent Systems group, Computer Science Institute VI – Intelligent Systems and Robotics – and the Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany(自主智能系统组,计算机科学研究所VI——智能系统与机器人——和机器人中心以及拉马尔人工智能与机器学习研究所,波恩大学,德国)
FOCUS: Frequency-Optimized Conditioning of DiffUSion Models for mitigating catastrophic forgetting during Test-Time Adaptation
Centre for Frontier AI Research (CFAR)(前沿人工智能研究中心);Agency for Science, Technology and Research (A*STAR)(科技研究局);College of Computing and Data Science(计算与数据科学学院);Nanyang Technological University(南洋理工大学);Institute of Infocomm Research (I2R)(信息通信研究所);School of Computer Science and Technology(计算机科学与技术学院)
Advancing AI-Powered Medical Image Synthesis: Insights from MedVQA-GI Challenge Using CLIP, Fine-Tuned Stable Diffusion, and Dream-Booth + LoRA
Department of Computer Science, SCMNS School, Morgan State University, Baltimore, Maryland 21251, USA;Electrical \& Computer Engineering Dept., School of Engineering, Morgan State University, Baltimore, Maryland 21251, USA
IDF-CR: Iterative Diffusion Process for Divide-and-Conquer Cloud Removal in Remote-sensing Images
School of Information Engineering, Guangdong University of Technology(广东工业大学信息工程学院);CRRC Academy Co., Ltd.(中车研究院有限公司);School of Computer Science, Sun Yat-sen University(中山大学计算机学院)
Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?
National Yang Ming Chiao Tung University(国立阳明交通大学);University of Illinois at Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Chicago(芝加哥大学);IBM Research(IBM研究院)
Physics-Constrained Diffusion Reconstruction with Posterior Correction for Quantitative and Fast PET Imaging
Beijing Normal University(北京师范大学);Beijing Friendship Hospital, Capital Medical University(首都医科大学北京友谊医院);The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China(中国科学技术大学生命科学与医学部附属第一医院);Jiangnan University(江南大学);University of Melbourne(墨尔本大学)