Recent advances in diffusion models have significantly improved the synthesis of materials, textures, and 3D shapes. By conditioning these models via text or images, users can guide the generation, reducing the time required to create digital assets. In this paper, we address the synthesis of structured, stationary patterns, where diffusion models are generally less reliable and, more importantly, less controllable.
Our approach leverages the generative capabilities of diffusion models specifically adapted for the pattern domain. It enables users to exercise direct control over the synthesis by expanding a partially hand-drawn pattern into a larger design while preserving the structure and details of the input. To enhance pattern quality, we fine-tune an image-pretrained diffusion model on structured patterns using Low-Rank Adaptation (LoRA), apply a noise rolling technique to ensure tileability, and utilize a patch-based approach to facilitate the generation of large-scale assets.
We demonstrate the effectiveness of our method through a comprehensive set of experiments, showing that it outperforms existing models in generating diverse, consistent patterns that respond directly to user input.
Modular Diffusion Models for Structured Visual Recognition
Department of Computer Science, University of British Columbia（不列颠哥伦比亚大学计算机科学系）；Vector Institute for AI（向量人工智能研究所）；CompVis, Ludwig Maximilian University of Munich（慕尼黑路德维希·马克西米利安大学计算机视觉实验室）；Munich Center for Machine Learning（慕尼黑机器学习中心）；CIFAR AI Chair（CIFAR人工智能教席）
GRE-Diff: Gaussian Room Embeddings for Structured Layout Diffusion
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University（广东省视觉媒体与多维智能重点实验室，计算机科学与软件学院，深圳大学）；School of Computer Science, University of Guelph（圭尔夫大学计算机科学学院）
