arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.05239cs.CVcs.AIcs.LG
收藏

面向文本驱动的人体图像生成中扩散模型里人体中心先验的有效利用研究

Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation

Carnegie Mellon University(卡内基梅隆大学);Alibaba DAMO Academy(阿里巴巴达摩院);Fudan University(复旦大学);University of New South Wales(新南威尔士大学);Australian National University(澳大利亚国立大学);INF Technology(INF科技)

Junyan Wang, Zhenhong Sun, Zhiyu Tan, Xuanbai Chen, Weihua Chen, Hao Li, Cheng Zhang, Yang Song

AI总结:

针对文本到图像扩散模型生成人体图像存在解剖结构缺陷的问题,提出将人体中心先验融入微调阶段的方法,通过人体中心对齐损失及尺度感知、逐步约束提升生成质量,无需推理时额外条件。

摘要:

普通的文本到图像扩散模型难以生成准确的人体图像,通常会产生解剖结构缺陷,例如不自然的姿势或比例失调的肢体。现有方法大多通过使用额外图像微调模型,或在图像生成阶段添加额外控制条件(即姿态、深度图等人体中心先验)来解决该问题。本文探索将这些人体中心先验直接整合到模型微调阶段,从根本上消除推理阶段对额外条件的需求。我们通过提出一种人体中心对齐损失来实现该思路,以强化交叉注意力图中文本提示包含的人体相关信息。为确保微调过程中的语义细节丰富度与人体结构准确性,我们基于对交叉注意力层的深入分析,在扩散过程中引入了尺度感知与逐步约束。大量实验表明,我们的方法在根据用户编写的提示合成高质量人体图像方面,相较现有最先进的文本到图像模型有大幅提升。项目页面:https://hcplayercvpr2024.github.io。

英文摘要:

Vanilla text-to-image diffusion models struggle with generating accurate human images, commonly resulting in imperfect anatomies such as unnatural postures or disproportionate limbs.Existing methods address this issue mostly by fine-tuning the model with extra images or adding additional controls -- human-centric priors such as pose or depth maps -- during the image generation phase. This paper explores the integration of these human-centric priors directly into the model fine-tuning stage, essentially eliminating the need for extra conditions at the inference stage. We realize this idea by proposing a human-centric alignment loss to strengthen human-related information from the textual prompts within the cross-attention maps. To ensure semantic detail richness and human structural accuracy during fine-tuning, we introduce scale-aware and step-wise constraints within the diffusion process, according to an in-depth analysis of the cross-attention layer. Extensive experiments show that our method largely improves over state-of-the-art text-to-image models to synthesize high-quality human images based on user-written prompts. Project page: \url{https://hcplayercvpr2024.github.io}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑