从局部到整体：面向可控人物图像生成的统一参考框架
From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation
AI总结：
针对多外观条件人物图像生成难题，提出Parts2Whole框架，通过语义感知外观编码器与掩码增强共享自注意力机制实现多参考图定制化肖像生成。
摘要：
近期，可控人物图像生成领域的进展已实现利用结构信号（如姿态、深度）或面部外观进行零样本生成。然而，以人物外观的多个局部为条件生成人物图像仍具挑战。为此，我们提出Parts2Whole，一种新颖的框架，旨在从多张参考图像（包括姿态图像和人物外观的多个方面）生成定制化肖像。为实现该目标，我们首先开发了一个语义感知的外观编码器，以保留不同人物局部的细节；该编码器根据每张图像的文本标签将其处理为一系列多尺度特征图，而非单一图像token，从而保留图像维度信息。其次，我们的框架通过共享自注意力机制支持多图像条件生成，该机制在扩散过程中跨参考特征和目标特征进行操作。我们通过引入来自参考人物图像的掩码信息来增强原始注意力机制，从而允许对任意局部进行精确选择。大量实验表明，我们的方法优于现有替代方案，为多局部可控人物图像定制提供了先进的能力。项目页面见 https://huanngzh.github.io/Parts2Whole/。
英文摘要：
Recent advancements in controllable human image generation have led to zero-shot generation using structural signals (e.g., pose, depth) or facial appearance. Yet, generating human images conditioned on multiple parts of human appearance remains challenging. Addressing this, we introduce Parts2Whole, a novel framework designed for generating customized portraits from multiple reference images, including pose images and various aspects of human appearance. To achieve this, we first develop a semantic-aware appearance encoder to retain details of different human parts, which processes each image based on its textual label to a series of multi-scale feature maps rather than one image token, preserving the image dimension. Second, our framework supports multi-image conditioned generation through a shared self-attention mechanism that operates across reference and target features during the diffusion process. We enhance the vanilla attention mechanism by incorporating mask information from the reference human images, allowing for the precise selection of any part. Extensive experiments demonstrate the superiority of our approach over existing alternatives, offering advanced capabilities for multi-part controllable human image customization. See our project page at https://huanngzh.github.io/Parts2Whole/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
