Diffusion models are a powerful class of generative models capable of producing high-quality images from pure noise using a simple text prompt. While most methods which introduce additional spatial constraints into the generated images (e.g., bounding boxes) require fine-tuning, a smaller and more recent subset of these methods take advantage of the models' attention mechanism, and are training-free. These methods generally fall into one of two categories. The first entails modifying the cross-attention maps of specific tokens directly to enhance the signal in certain regions of the image. The second works by defining a loss function over the cross-attention maps, and using the gradient of this loss to guide the latent. While previous work explores these as alternative strategies, we provide an interpretation for these methods which highlights their complimentary features, and demonstrate that it is possible to obtain superior performance when both methods are used in concert.
Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Indian Statistical Institute（印度统计研究所）
Despite significant progress on current state-of-the-art image generation models, synthesis of document images containing multiple and complex object layouts is a challenging task. This paper presents a novel approach, called DocSynth, to automatically synthesize document images based on a given layout. In this work, given a spatial layout (bounding boxes with object categories) as a reference by the user, our proposed DocSynth model learns to generate a set of realistic document images consistent with the defined layout. Also, this framework has been adapted to this work as a superior baselin…
University of British Columbia（不列颠哥伦比亚大学）；Bank of Montreal（蒙特利尔银行）
Recent approaches have achieved great success in image generation from structured inputs, e.g., semantic segmentation, scene graph or layout. Although these methods allow specification of objects and their locations at image-level, they lack the fidelity and semantic control to specify visual appearance of these objects at an instance-level. To address this limitation, we propose a new image generation method that enables instance-level attribute control. Specifically, the input to our attribute-guided generative model is a tuple that contains: (1) object bounding boxes, (2) object categories …
We present an end-to-end deep learning framework for indoor panoramic image inpainting. Although previous inpainting methods have shown impressive performance on natural perspective images, most fail to handle panoramic images, particularly indoor scenes, which usually contain complex structure and texture content. To achieve better inpainting quality, we propose to exploit both the global and local context of indoor panorama during the inpainting process. Specifically, we take the low-level layout edges estimated from the input panorama as a prior to guide the inpainting model for recovering …
Diffusion models are able to generate photorealistic images in arbitrary scenes. However, when applying diffusion models to image translation, there exists a trade-off between maintaining spatial structure and high-quality content. Besides, existing methods are mainly based on test-time optimization or fine-tuning model for each input image, which are extremely time-consuming for practical applications. To address these issues, we propose a new approach for flexible image translation by learning a layout-aware image condition together with a text condition. Specifically, our method co-encodes …
Enhancing Retinal Vessel Segmentation Generalization via Layout-Aware Generative Modelling
Faculty of Biomedical Engineering, Technion-IIT（以色列理工学院生物医学工程学院）；Faculty of Mathematics, Technion-IIT（以色列理工学院数学学院）；KU Leuven（鲁汶大学）；UZ Leuven（鲁汶大学医院）；São Paulo Federal University（圣保罗联邦大学）；Massachusetts Institute of Technology（麻省理工学院）；School of Electrical and Computer Engineering, Ben-Gurion University of the Negev（本-古里安大学内盖夫分校电气与计算机工程学院）
CalliMaster: Mastering Page-level Chinese Calligraphy via Layout-guided Spatial Planning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Faculty of Engineering and IT, University of Technology Sydney（悉尼大学工程与信息学院）；Xiamen University（厦门大学）；The Hong Kong University of Science and Technology（香港科技大学）
ETH Zurich（苏黎世联邦理工学院）；University of Hong Kong（香港大学）；HKUST（香港科技大学）；University of Trento（特伦托大学）；University of Wurzburg（维尔茨堡大学）
We propose a novel ECGAN for the challenging semantic image synthesis task. Although considerable improvement has been achieved, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it difficult to synthesize local details and structures. 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent r…
Computer Vision Lab ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；KU Leuven（鲁汶大学）
Attribute image manipulation has been a very active topic since the introduction of Generative Adversarial Networks (GANs). Exploring the disentangled attribute space within a transformation is a very challenging task due to the multiple and mutually-inclusive nature of the facial images, where different labels (eyeglasses, hats, hair, identity, etc.) can co-exist at the same time. Several works address this issue either by exploiting the modality of each domain/attribute using a conditional random vector noise, or extracting the modality from an exemplary image. However, existing methods cann…
Enhancing Object Coherence in Layout-to-Image Synthesis
School of Computer Science, Fudan University（复旦大学计算机科学学院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）
RSGen: Enhancing Layout-Driven Remote Sensing Image Generation with Diverse Edge Guidance
School of Future Science and Engineering, Soochow University, Suzhou, China（未来科学与工程学院，苏州大学）；D-Robotics, Beijing, China（北京D-机器人）；Fraunhofer Institute for Applied Information Technology, Sankt Augustin, Germany（弗劳恩霍夫应用信息技术研究所）；School of Mathematical and Computing Sciences, Heriot-Watt University Malaysia, Putrajaya, Malaysia（数学与计算科学学院，赫瑞-瓦德大学马来西亚分校）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（自动化研究所，中国科学院北京分院）
Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints
Hong Kong University of Science and Technology（香港理工大学）；LightIllusion, China（中国LightIllusion公司）；Alibaba Group（阿里巴巴集团）；Simon Fraser University, Canada（加拿大Simon Fraser大学）
Lung-DDPM: Semantic Layout-guided Diffusion Models for Thoracic CT Image Synthesis
Centre de recherche du CHU de Québec-Université Laval（魁北克大学拉瓦尔医院研究中心）；Centre de recherche de l’Institut universitaire de cardiologie et de pneumologie de Québec-Université Laval（魁北克大学拉瓦尔心血管与呼吸病学研究所）
EchoGen: Cycle-Consistent Learning for Unified Layout-Image Generation and Understanding
School of Cyber Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；Tongji University（同济大学）；Sun Yat-sen University（中山大学）；Fudan University（复旦大学）；Hefei University of Technology（合肥工业大学）；Anhui Province Key Laboratory of Digital Security（安徽省数字安全重点实验室）
Interactive image synthesis from user-guided input is a challenging task when users wish to control the scene structure of a generated image with ease.Although remarkable progress has been made on layout-based image synthesis approaches, in order to get realistic fake image in interactive scene, existing methods require high-precision inputs, which probably need adjustment several times and are unfriendly to novice users. When placement of bounding boxes is subject to perturbation, layout-based models suffer from "missing regions" in the constructed semantic layouts and hence undesirable artif…
The Chinese University of Hong Kong（香港中文大学）；University of Science and Technology of China（中国科学技术大学）；SenseTime Research（商汤科技研究院）
Semantic image synthesis aims at generating photorealistic images from semantic layouts. Previous approaches with conditional generative adversarial networks (GAN) show state-of-the-art performance on this task, which either feed the semantic label maps as inputs to the generator, or use them to modulate the activations in normalization layers via affine transformations. We argue that convolutional kernels in the generator should be aware of the distinct semantic labels at different locations when generating images. In order to better exploit the semantic layout for the image generator, we pro…
Beuth University of Applied Sciences Berlin（柏林博伊特应用科学大学）
We propose a novel approach for the synthesis of sign language videos using GANs. We extend the previous work of Stoll et al. by using the human semantic parser of the Soft-Gated Warping-GAN from to produce photorealistic videos guided by region-level spatial layouts. Synthesizing target poses improves performance on independent and contrasting signers. Therefore, we have evaluated our system with the highly heterogeneous MS-ASL dataset with over 200 signers resulting in a SSIM of 0.893. Furthermore, we introduce a periodic weighting approach to the generator that reactivates the training and …
Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Mayo Clinic（梅奥诊所）
Real-world non-mydriatic retinal fundus photography is prone to artifacts, imperfections and low-quality when certain ocular or systemic co-morbidities exist. Artifacts may result in inaccuracy or ambiguity in clinical diagnoses. In this paper, we proposed a simple but effective end-to-end framework for enhancing poor-quality retinal fundus images. Leveraging the optimal transport theory, we proposed an unpaired image-to-image translation scheme for transporting low-quality images to their high-quality counterparts. We theoretically proved that a Generative Adversarial Networks (GAN) model wit…
Fudan University（复旦大学）；East China Normal University（华东师范大学）；Videt Lab（Videt实验室）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）
Document layout analysis (DLA) plays an important role in information extraction and document understanding. At present, document layout analysis has reached a milestone achievement, however, document layout analysis of non-Manhattan is still a challenge. In this paper, we propose an image layer modeling method to tackle this challenge. To measure the proposed image layer modeling method, we propose a manually-labeled non-Manhattan layout fine-grained segmentation dataset named FPD. As far as we know, FPD is the first manually-labeled non-Manhattan layout fine-grained segmentation dataset. To …
