DiffBrush：只需亲手绘画
DiffBrush:Just Painting the Art by Your Hands
AI总结：
DiffBrush是一种兼容T2I扩散模型的免训练绘画编辑方法，通过引导潜在表示和注意力图，使用户手绘草图能控制生成图像的布局与对象属性。
摘要：
近年来，图像生成与编辑算法的快速发展使得普通用户也能生成逼真的图像。然而，当前AI绘画生态主要依赖文本驱动的扩散模型（T2I），这难以精确捕捉用户需求。此外，实现与其他模态的兼容需要高昂的训练成本。为此，我们提出了DiffBrush，它与T2I模型兼容，允许用户通过绘画来生成和编辑图像。通过操控和调整扩散模型的内部表示，DiffBrush引导模型生成的图像向用户手绘草图收敛，以满足用户的特定需求，而无需额外训练。DiffBrush通过在扩散模型的去噪过程中持续引导潜在表示和实例级注意力图，实现了对图像中对象的颜色、语义和实例的控制。此外，我们提出了一种潜在重生机制，该机制优化扩散模型中随机采样的噪声，以获得更好的图像生成布局。最后，用户只需在画布上粗略绘制实例的掩码（颜色可接受），DiffBrush即可在相应位置自然生成相应的实例。
英文摘要：
The rapid development of image generation and editing algorithms in recent years has enabled ordinary user to produce realistic images. However, the current AI painting ecosystem predominantly relies on text-driven diffusion models (T2I), which pose challenges in accurately capturing user requirements. Furthermore, achieving compatibility with other modalities incurs substantial training costs. To this end, we introduce DiffBrush, which is compatible with T2I models and allows users to draw and edit images. By manipulating and adapting the internal representation of the diffusion model, DiffBrush guides the model-generated images to converge towards the user's hand-drawn sketches for user's specific needs without additional training. DiffBrush achieves control over the color, semantic, and instance of objects in images by continuously guiding the latent and instance-level attention map during the denoising process of the diffusion model. Besides, we propose a latent regeneration, which refines the randomly sampled noise in the diffusion model, obtaining a better image generation layout. Finally, users only need to roughly draw the mask of the instance (acceptable colors) on the canvas, DiffBrush can naturally generate the corresponding instance at the corresponding location.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
