PixelHacker：具有结构与语义一致性的图像修复
PixelHacker: Image Inpainting with Structural and Semantic Consistency
AI总结：
本文提出基于扩散模型的图像修复方法 PixelHacker，通过潜在类别引导范式与大规模数据集预训练，在结构与语义一致性上全面超越现有 SOTA。
摘要：
图像修复是介于图像编辑与图像生成之间的基础研究领域。近期最先进（SOTA）方法探索了新型注意力机制、轻量化架构和上下文感知建模，展现出令人瞩目的性能。然而，它们在处理复杂结构（如纹理、形状、空间关系）和语义（如颜色一致性、物体修复及逻辑正确性）时仍面临困难，易导致伪影与不当生成。为应对此挑战，我们设计了一种简单有效的修复范式，称为潜在类别引导，并进一步提出基于扩散模型的 PixelHacker。具体而言，我们首先构建了一个包含 1400 万张图像-掩码对的大型数据集，对前景和背景进行标注（分别潜在包含 116 和 21 个类别）。随后，我们通过两个固定大小的嵌入分别编码潜在前景与背景表示，并通过线性注意力间歇性将这些特征注入去噪过程。最后，通过在我们的数据集上预训练并在开源基准上微调，得到 PixelHacker。大量实验表明，PixelHacker 在多种数据集（Places2、CelebA-HQ 和 FFHQ）上全面超越 SOTA，并在结构和语义上展现出显著一致性。项目页面：https://hustvl.github.io/PixelHacker。
英文摘要：
Image inpainting is a fundamental research area between image editing and image generation. Recent state-of-the-art (SOTA) methods have explored novel attention mechanisms, lightweight architectures, and context-aware modeling, demonstrating impressive performance. However, they often struggle with complex structure (e.g., texture, shape, spatial relations) and semantics (e.g., color consistency, object restoration, and logical correctness), leading to artifacts and inappropriate generation. To address this challenge, we design a simple yet effective inpainting paradigm called latent categories guidance, and further propose a diffusion-based model named PixelHacker. Specifically, we first construct a large dataset containing 14 million image-mask pairs by annotating foreground and background (potential 116 and 21 categories, respectively). Then, we encode potential foreground and background representations separately through two fixed-size embeddings, and intermittently inject these features into the denoising process via linear attention. Finally, by pre-training on our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker. Extensive experiments show that PixelHacker comprehensively outperforms the SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits remarkable consistency in both structure and semantics. Project page at https://hustvl.github.io/PixelHacker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
