While recent image generation models demonstrate a remarkable ability to handle a wide variety of image generation tasks, this flexibility makes them hard to control via prompting or simple inference adaptation alone, rendering them unsuitable for use cases with strict product requirements. In this paper, we introduce Pinterest Canvas, our large-scale image generation system built to support image editing and enhancement use cases at Pinterest. Canvas is first trained on a diverse, multimodal dataset to produce a foundational diffusion model with broad image-editing capabilities. However, rather than relying on one generic model to handle every downstream task, we instead rapidly fine-tune variants of this base model on task-specific datasets, producing specialized models for individual use cases. We describe key components of Canvas and summarize our best practices for dataset curation, training, and inference. We also showcase task-specific variants through case studies on background enhancement and aspect-ratio outpainting, highlighting how we tackle their specific product requirements. Online A/B experiments demonstrate that our enhanced images receive a significant 18.0% and 12.5% engagement lift, respectively, and comparisons with human raters further validate that our models outperform third-party models on these tasks. Finally, we showcase other Canvas variants, including multi-image scene synthesis and image-to-video generation, demonstrating that our approach can generalize to a wide variety of potential downstream tasks.
We propose a new recurrent generative model for generating images from text captions while attending on specific parts of text captions. Our model creates images by incrementally adding patches on a "canvas" while attending on words from text caption at each timestep. Finally, the canvas is passed through an upscaling network to generate images. We also introduce a new method for generating visual-semantic sentence embeddings based on self-attention over text. We compare our model's generated images with those generated Reed et. al.'s model and show that our model is a stronger baseline for te…
From Prompting to Composing: A Spatial Canvas Interface for Poster Generation
Fudan University（复旦大学）；Microsoft Research（微软研究院）；The University of Sydney（悉尼大学）；USTC（中国科学技术大学）；University of Waterloo（滑铁卢大学）；The University of Hong Kong（香港大学）
CanvasMAR: Improving Masked Autoregressive Video Prediction With Canvas
Institute for Artificial Intelligence, Peking University, Beijing, China（人工智能研究院，北京大学，北京，中国）；School of Intelligence Science and Technology, Peking University, Beijing, China（智能科学与技术学院，北京大学，北京，中国）；State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China（通用人工智能国家重点实验室，北京大学，北京，中国）
LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Model
Fudan University（复旦大学）；School of Data Science, Fudan University（复旦大学数据科学学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Alibaba Group（阿里巴巴集团）
Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TReCS, a sequential model that exploits this grounding to generate images. TReCS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approa…
KAUST（阿卜杜拉国王科技大学）；Adobe Research（Adobe研究院）；University College London（伦敦大学学院）
While GANs can produce photo-realistic images in ideal conditions for certain domains, the generation of full-body human images remains difficult due to the diversity of identities, hairstyles, clothing, and the variance in pose. Instead of modeling this complex domain with a single GAN, we propose a novel method to combine multiple pretrained GANs, where one GAN generates a global canvas (e.g., human body) and a set of specialized GANs, or insets, focus on different parts (e.g., faces, shoes) that can be seamlessly inserted onto the global canvas. We model the problem as jointly exploring the…
The prevalent approach in self-supervised image generation is to operate on pixel level representations. While this approach can produce high quality images, it cannot benefit from the simplicity and innate quality of vectorization. Here we present a drawing agent that operates on stroke-level representation of images. At each time step, the agent first assesses the current canvas and decides whether to stop or keep drawing. When a 'draw' decision is made, the agent outputs a program indicating the stroke to be drawn. As a result, it produces a final raster image by drawing the strokes on a ca…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；NVIDIA（英伟达）；Canadian Institute for Advanced Research（加拿大高等研究院）
In this paper, we present a non-parametric structured latent variable model for image generation, called NP-DRAW, which sequentially draws on a latent canvas in a part-by-part fashion and then decodes the image from the canvas. Our key contributions are as follows. 1) We propose a non-parametric prior distribution over the appearance of image parts so that the latent variable ``what-to-draw'' per step becomes a categorical random variable. This improves the expressiveness and greatly eases the learning compared to Gaussians used in the literature. 2) We model the sequential dependency structur…
Instituto de Ingeniería del Conocimiento（知识工程研究所）；Universidad Autónoma de Madrid（马德里自治大学）
Diffusion methods have been proven to be very effective to generate images while conditioning on a text prompt. However, and although the quality of the generated images is unprecedented, these methods seem to struggle when trying to generate specific image compositions. In this paper we present Mixture of Diffusers, an algorithm that builds over existing diffusion models to provide a more detailed control over composition. By harmonizing several diffusion processes acting on different regions of a canvas, it allows generating larger images, where the location of each object and style is contr…
Motivated by the recent progress in generative models, we introduce a model that generates images from natural language descriptions. The proposed model iteratively draws patches on a canvas, while attending to the relevant words in the description. After training on Microsoft COCO, we compare our model with several baseline generative models on image generation and retrieval tasks. We demonstrate that our model produces higher quality samples than other approaches and generates images with novel scene compositions corresponding to previously unseen captions in the dataset.
MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
Beijing University of Technology（北京工业大学）；East China University of Science（华东理工大学）；Shandong University（山东大学）；Tsinghua University（清华大学）；Rutgers University（罗格斯大学）；University of Washington（华盛顿大学）；University of Copenhagen（哥本哈根大学）
Beijing University of Post and Telecommunication（北京邮电大学）；Tsinghua University（清华大学）；Institute of AI (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；School of Artificial Intelligence（人工智能学院）；Northwestern Polytechnical University (NWPU)（西北工业大学）
Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. Starting from random noise, such text-to-image diffusion models gradually synthesize images in an iterative fashion while conditioning on text prompts. We find that their synthesis behavior qualitatively changes throughout this process: Early in sampling, generation strongly relies on the text prompt to generate text-aligned content, while later, the text conditioning is almost entirely ignored. This suggests that sharing model parameters throughout the entire generation…
EPEdit: Redefining Image Editing with Generative AI and User-Centric Design
University of Science, VNU-HCM（越南胡志明市科学大学）；Vietnam National University, Ho Chi Minh City（越南国家大学，胡志明市）；University of Information Technology, VNU-HCM（越南胡志明市信息科技大学）；University of Dayton（Dayton 大学）
