Sketch2Human：基于解耦几何与外观控制的深度人体生成
Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Control
AI总结：
提出Sketch2Human，首个由语义草图和参考图像引导的全身人体图像生成系统，基于StyleGAN-Human潜空间并设计新颖训练方案实现解耦的几何与外观控制，性能优于现有方法。
摘要：
几何和外观控制的全局人体图像生成是一项有趣但具有挑战性的任务。现有的解决方案要么是无条件的，要么依赖于粗略的条件（例如，姿态、文本），因此缺乏对身体和服装的显式几何和外观控制。草图提供了这种编辑能力，并已被各种基于草图的人脸生成和编辑解决方案所采用。然而，由于姿态、体型以及服装形状和纹理的高复杂性和多样性，直接将基于草图的人脸生成适应于全身生成通常无法产生高保真和多样化的结果。最近基于几何可控的扩散方法主要依赖提示来生成外观，并且当输入粗略时，很难平衡其结果的真实性和对草图的忠实度。这项工作提出了 Sketch2Human，这是第一个由语义草图（用于几何控制）和参考图像（用于外观控制）引导的可控全身人体图像生成系统。我们的解决方案基于 StyleGAN-Human 的潜在空间，以反演的几何和外观潜码作为输入。具体来说，我们提出了一个草图编码器，该编码器使用从 StyleGAN-Human 的潜在空间采样的大型合成数据集进行训练，并由草图直接监督，而不是真实图像。考虑到 StyleGAN-Human 中部分几何和纹理的纠缠信息以及解耦数据集的缺失，我们设计了一种新颖的训练方案，该方案创建保留几何和转移外观的训练数据，以调整生成器从而实现解耦的几何和外观控制。尽管我们的方法是用合成数据训练的，但它也能处理手绘草图。定性和定量评估证明了我们的方法相对于最先进方法的优越性能。
英文摘要：
Geometry- and appearance-controlled full-body human image generation is an interesting but challenging task. Existing solutions are either unconditional or dependent on coarse conditions (e.g., pose, text), thus lacking explicit geometry and appearance control of body and garment. Sketching offers such editing ability and has been adopted in various sketch-based face generation and editing solutions. However, directly adapting sketch-based face generation to full-body generation often fails to produce high-fidelity and diverse results due to the high complexity and diversity in the pose, body shape, and garment shape and texture. Recent geometrically controllable diffusion-based methods mainly rely on prompts to generate appearance and it is hard to balance the realism and the faithfulness of their results to the sketch when the input is coarse. This work presents Sketch2Human, the first system for controllable full-body human image generation guided by a semantic sketch (for geometry control) and a reference image (for appearance control). Our solution is based on the latent space of StyleGAN-Human with inverted geometry and appearance latent codes as input. Specifically, we present a sketch encoder trained with a large synthetic dataset sampled from StyleGAN-Human's latent space and directly supervised by sketches rather than real images. Considering the entangled information of partial geometry and texture in StyleGAN-Human and the absence of disentangled datasets, we design a novel training scheme that creates geometry-preserved and appearance-transferred training data to tune a generator to achieve disentangled geometry and appearance control. Although our method is trained with synthetic data, it can handle hand-drawn sketches as well. Qualitative and quantitative evaluations demonstrate the superior performance of our method to state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
