BrushNet：一种采用分解双分支扩散的即插即用图像修复模型
BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion
AI总结：
针对扩散模型图像修复中的语义不一致与质量下降问题，提出双分支即插即用 BrushNet，并配套 BrushData、BrushBench，在七项指标上优于现有方法。
摘要：
图像修复是恢复受损图像的过程，随着扩散模型（DMs）的出现已取得显著进展。尽管有这些进展，当前用于修复的 DM 改编方法（包括修改采样策略或开发专门用于修复的 DMs）仍经常受到语义不一致和图像质量下降的困扰。为应对这些挑战，我们的工作提出了一种新颖范式：将带掩码图像特征和噪声潜变量划分到不同分支。这种划分显著降低了模型的学习负担，有助于以分层方式精细融入必要的带掩码图像信息。在此，我们提出 BrushNet，这是一种新颖的即插即用双分支模型，旨在将像素级带掩码图像特征嵌入任何预训练 DM，确保获得连贯且增强的图像修复结果。此外，我们提出 BrushData 和 BrushBench，以促进基于分割的修复训练和性能评估。大量实验分析表明，BrushNet 在图像质量、掩码区域保持和文本连贯性等七项关键指标上均优于现有模型。
英文摘要：
Image inpainting, the process of restoring corrupted images, has seen significant advancements with the advent of diffusion models (DMs). Despite these advancements, current DM adaptations for inpainting, which involve modifications to the sampling strategy or the development of inpainting-specific DMs, frequently suffer from semantic inconsistencies and reduced image quality. Addressing these challenges, our work introduces a novel paradigm: the division of masked image features and noisy latent into separate branches. This division dramatically diminishes the model's learning load, facilitating a nuanced incorporation of essential masked image information in a hierarchical fashion. Herein, we present BrushNet, a novel plug-and-play dual-branch model engineered to embed pixel-level masked image features into any pre-trained DM, guaranteeing coherent and enhanced image inpainting outcomes. Additionally, we introduce BrushData and BrushBench to facilitate segmentation-based inpainting training and performance assessment. Our extensive experimental analysis demonstrates BrushNet's superior performance over existing models across seven key metrics, including image quality, mask region preservation, and textual coherence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
