Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (e.g., document and web designs) with constraints representing design intentions. Although recent diffusion-based models have achieved state-of-the-art FID scores, they tend to exhibit more pronounced misalignment compared to earlier transformer-based models. In this work, we propose the $\textbf{LA}$yout $\textbf{C}$onstraint diffusion mod$\textbf{E}$l (LACE), a unified model to handle a broad range of layout generation tasks, such as arranging elements with specified attributes and refining or completing a coarse layout design. The model is based on continuous diffusion models. Compared with existing methods that use discrete diffusion models, continuous state-space design can enable the incorporation of differentiable aesthetic constraint functions in training. For conditional generation, we introduce conditions via masked input. Extensive experiment results show that LACE produces high-quality layouts and outperforms existing state-of-the-art baselines.
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；ProjectHAT
We explore computational approaches for visual guidance to aid in creating aesthetically pleasing art and graphic design. Our work complements and builds on previous work that developed models for how humans look at images. Our approach comprises three steps. First, we collected a dataset of art masterpieces and labeled the visual fixations with state-of-art vision models. Second, we clustered the visual guidance templates of the art masterpieces with unsupervised learning. Third, we developed a pipeline using generative adversarial networks to learn the principles of visual guidance and that …
Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization
The Australian National University（澳大利亚国立大学）；University of Liverpool（利物浦大学）；Southeast University（东南大学）；Microsoft（微软公司）；Microsoft Research Asia（微软亚洲研究院）
In this paper, we study the graphic layout generation problem of producing high-quality visual-textual presentation designs for given images. We note that image compositions, which contain not only global semantics but also spatial information, would largely affect layout results. Hence, we propose a deep generative model, dubbed as composition-aware graphic layout GAN (CGL-GAN), to synthesize layouts based on the global and spatial visual contents of input images. To obtain training images from images that already contain manually designed graphic layout data, previous work suggests masking d…
SUNY at Buffalo（布法罗纽约州立大学）；Tianjin University（天津大学）
Deep convolutional neural networks (CNN) have recently been shown to generate promising results for aesthetics assessment. However, the performance of these deep CNN methods is often compromised by the constraint that the neural network only takes the fixed-size input. To accommodate this requirement, input images need to be transformed via cropping, warping, or padding, which often alter image composition, reduce image resolution, or cause image distortion. Thus the aesthetics of the original images is impaired because of potential loss of fine grained details and holistic image layout. Howev…
AesRM: Improving Video Aesthetics with Expert-Level Feedback
The University of Hong Kong（香港大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；University of Sydney（悉尼大学）；Alibaba Group（阿里巴巴集团）
Fudan University（复旦大学）；East China Normal University（华东师范大学）；Videt Lab（Videt实验室）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）
Document layout analysis (DLA) plays an important role in information extraction and document understanding. At present, document layout analysis has reached a milestone achievement, however, document layout analysis of non-Manhattan is still a challenge. In this paper, we propose an image layer modeling method to tackle this challenge. To measure the proposed image layer modeling method, we propose a manually-labeled non-Manhattan layout fine-grained segmentation dataset named FPD. As far as we know, FPD is the first manually-labeled non-Manhattan layout fine-grained segmentation dataset. To …
POSTA: A Go-to Framework for Customized Artistic Poster Generation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology（香港科技大学）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Tencent（腾讯）
Text logo design heavily relies on the creativity and expertise of professional designers, in which arranging element layouts is one of the most important procedures. However, few attention has been paid to this task which needs to take many factors (e.g., fonts, linguistics, topics, etc.) into consideration. In this paper, we propose a content-aware layout generation network which takes glyph images and their corresponding text as input and synthesizes aesthetic layouts for them automatically. Specifically, we develop a dual-discriminator module, including a sequence discriminator and an imag…
Simon Fraser University（西蒙弗雷泽大学）；Autodesk Research（欧特克研究院）
This paper proposes a novel graph-constrained generative adversarial network, whose generator and discriminator are built upon relational architecture. The main idea is to encode the constraint into the graph structure of its relational networks. We have demonstrated the proposed architecture for a new house layout generation problem, whose task is to take an architectural constraint as a graph (i.e., the number and types of rooms with their spatial adjacency) and produce a set of axis-aligned bounding boxes of rooms. We measure the quality of generated house layouts with the three metrics: th…
University of Surrey（萨里大学）；Reutlingen University（罗伊特林根大学）；Hunan University of Science Technology（湖南科技大学）；Xi’an Polytechnic University（西安工程大学）
Facial beauty prediction (FBP) aims to develop a machine that automatically makes facial attractiveness assessment. In the past those results were highly correlated with human ratings, therefore also with their bias in annotating. As artificial intelligence can have racist and discriminatory tendencies, the cause of skews in the data must be identified. Development of training data and AI algorithms that are robust against biased information is a new challenge for scientists. As aesthetic judgement usually is biased, we want to take it one step further and propose an Unbiased Convolutional Neu…
Peking University（北京大学）；Institute of Computer Science and Technology（计算机科学技术研究所）
In this work, we present a new framework for the stylization of text-based binary images. First, our method stylizes the stroke-based geometric shape like text, symbols and icons in the target binary image based on an input style image. Second, the composition of the stylized geometric shape and a background image is explored. To accomplish the task, we propose legibility-preserving structure and texture transfer algorithms, which progressively narrow the visual differences between the binary image and the style image. The stylization is then followed by a context-aware layout design algorithm…
LEARN: A Story-Driven Layout-to-Image Generation Framework for STEM Instruction
Graduate School of Advanced Science and Engineering, Hiroshima University（Hiroshima大学研究生院）；Department of Computer Science, Weifang University of Science and Technology（潍坊科技大学计算机科学系）
Trinity College Dublin（都柏林圣三一学院）；Accenture Labs（埃森哲实验室）
Aspect ratio and spatial layout are two of the principal factors determining the aesthetic value of a photograph. But, incorporating these into the traditional convolution-based frameworks for the task of image aesthetics assessment is problematic. The aspect ratio of the photographs gets distorted while they are resized/cropped to a fixed dimension to facilitate training batch sampling. On the other hand, the convolutional filters process information locally and are limited in their ability to model the global spatial layout of a photograph. In this work, we present a two-stage framework base…
GardenDesigner: Encoding Aesthetic Principles into Jiangnan Garden Construction via a Chain of Agents
Shanghai University（上海大学）；Shanghai Engineering Research Center of Motion Picture Special Effects（上海电影特效工程技术研究中心）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Framework
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Meituan（美团）；Xiamen University（厦门大学）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology（香港科技大学）
CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）；School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；Alibaba Group（阿里巴巴集团）
Centre for Research and Technology Hellas（希腊研究与技术中心）
It has been shown that global scene understanding tasks like layout estimation can benefit from wider field of views, and specifically spherical panoramas. While much progress has been made recently, all previous approaches rely on intermediate representations and postprocessing to produce Manhattan-aligned estimates. In this work we show how to estimate full room layouts in a single-shot, eliminating the need for postprocessing. Our work is the first to directly infer Manhattan-aligned outputs. To achieve this, our data-driven model exploits direct coordinate regression and is supervised end-…
InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；School of Future Technology, South China University of Technology（华南理工大学未来技术学院）
Designing visually appealing layouts for multimedia documents containing text, graphs and images requires a form of creative intelligence. Modelling the generation of layouts has recently gained attention due to its importance in aesthetics and communication style. In contrast to standard prediction tasks, there are a range of acceptable layouts which depend on user preferences. For example, a poster designer may prefer logos on the top-left while another prefers logos on the bottom-right. Both are correct choices yet existing machine learning models treat layouts as a single choice prediction…
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）
Layout generation is a novel task in computer vision, which combines the challenges in both object localization and aesthetic appraisal, widely used in advertisements, posters, and slides design. An accurate and pleasant layout should consider both the intra-domain relationship within layout elements and the inter-domain relationship between layout elements and the image. However, most previous methods simply focus on image-content-agnostic layout generation, without leveraging the complex visual information from the image. To this end, we explore a novel paradigm entitled image-conditioned la…
Prim2Room: Layout-Controllable Room Mesh Generation from Primitives
Nanyang Technological University（南洋理工大学）；Tencent XR Vision Labs（腾讯XR视觉实验室）；Institute for Infocomm Research A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；Australia National University（澳大利亚国立大学）
CraftSVG: Multi-Object Text-to-SVG Synthesis via Layout Guided Diffusion
Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；Smallest AI；Indian Statistical Institute, Kolkata（印度统计研究所）；Institute for People Centred Artificial Intelligence, University of Surrey（以人为中心的人工智能研究所，萨里大学）
