DreamRenderer：驯服大规模文本到图像模型中的多实例属性控制
DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Models
AI总结：
针对现有文本到图像模型在多实例内容控制中存在的属性泄漏问题，提出无训练方法DreamRenderer，通过桥接图像标记和分层属性绑定策略，在保持视觉和谐的同时提升控制精度，相关基准测试结果显著优于现有模型。
摘要：
图像条件生成方法（如深度和边缘条件方法）已展现出卓越的精确图像合成能力。然而，现有模型在准确控制多个实例（或区域）的内容方面仍存在困难。即使是FLUX和3DIS等最先进的模型也面临挑战，例如实例之间的属性泄漏，这限制了用户控制。为解决这些问题，我们引入DreamRenderer，一种基于FLUX模型的无训练方法。DreamRenderer允许用户通过边界框或掩码控制每个实例的内容，同时确保整体视觉和谐。我们提出两个关键创新：1）用于硬文本属性绑定的桥接图像标记，使用复制的图像标记作为桥接标记，确保仅在文本数据上预训练的T5文本嵌入在联合注意力期间为每个实例绑定正确的视觉属性；2）仅在关键层应用硬图像属性绑定。通过对FLUX的分析，我们识别出负责实例属性渲染的关键层，并仅在这些层应用硬图像属性绑定，在其他层使用软绑定。这种方法在保持图像质量的同时确保精确控制。在COCO-POS和COCO-MIG基准上的评估表明，DreamRenderer比FLUX提高了17.7%的图像成功率，并将GLIGEN和3DIS等布局到图像模型的性能提高了高达26.8%。项目页面：https://limuloo.github.io/DreamRenderer/。
英文摘要：
Image-conditioned generation methods, such as depth- and canny-conditioned approaches, have demonstrated remarkable abilities for precise image synthesis. However, existing models still struggle to accurately control the content of multiple instances (or regions). Even state-of-the-art models like FLUX and 3DIS face challenges, such as attribute leakage between instances, which limits user control. To address these issues, we introduce DreamRenderer, a training-free approach built upon the FLUX model. DreamRenderer enables users to control the content of each instance via bounding boxes or masks, while ensuring overall visual harmony. We propose two key innovations: 1) Bridge Image Tokens for Hard Text Attribute Binding, which uses replicated image tokens as bridge tokens to ensure that T5 text embeddings, pre-trained solely on text data, bind the correct visual attributes for each instance during Joint Attention; 2) Hard Image Attribute Binding applied only to vital layers. Through our analysis of FLUX, we identify the critical layers responsible for instance attribute rendering and apply Hard Image Attribute Binding only in these layers, using soft binding in the others. This approach ensures precise control while preserving image quality. Evaluations on the COCO-POS and COCO-MIG benchmarks demonstrate that DreamRenderer improves the Image Success Ratio by 17.7% over FLUX and enhances the performance of layout-to-image models like GLIGEN and 3DIS by up to 26.8%. Project Page: https://limuloo.github.io/DreamRenderer/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
