SCAM!使用语义交叉注意调制在图像之间转移人
SCAM! Transferring humans between images with Semantic Cross Attention Modulation
摘要:
最近的一个大机构的工作目标语义条件的图像生成。大多数这样的方法集中在姿势转移的较窄的任务,而忽略了主体转移的更具挑战性的任务,该任务不仅包括转移姿势,还包括转移外观和背景。在这项工作中,我们介绍了SCAM(语义交叉注意力调制),一个系统,编码丰富和多样的信息在每个语义区域的图像(包括前景和背景),从而实现精确的生成,强调精细的细节。这是由语义注意力Transformer编码器和相应的生成器实现的,语义注意力编码器为每个语义区域提取多个潜在向量,相应的生成器通过使用语义交叉注意力调制来利用这些多个潜在向量。它仅使用重建设置进行训练,而受试者转移在测试时执行。我们的分析表明,我们提出的架构是成功的编码在每个语义区域的外观的多样性。在iDesigner和CelebAMask-HD数据集上进行的大量实验表明,SCAM的性能优于SEAN和SPADE;此外,它在主题转移方面开创了新的技术水平。
英文摘要:
A large body of recent work targets semantically conditioned image generation. Most such methods focus on the narrower task of pose transfer and ignore the more challenging task of subject transfer that consists in not only transferring the pose but also the appearance and background. In this work, we introduce SCAM (Semantic Cross Attention Modulation), a system that encodes rich and diverse information in each semantic region of the image (including foreground and background), thus achieving precise generation with emphasis on fine details. This is enabled by the Semantic Attention Transformer Encoder that extracts multiple latent vectors for each semantic region, and the corresponding generator that exploits these multiple latents by using semantic cross attention modulation. It is trained only using a reconstruction setup, while subject transfer is performed at test time. Our analysis shows that our proposed architecture is successful at encoding the diversity of appearance in each semantic region. Extensive experiments on the iDesigner and CelebAMask-HD datasets show that SCAM outperforms SEAN and SPADE; moreover, it sets the new state of the art on subject transfer.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。