arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.04608cs.CV
收藏

EmoGen:基于文本到图像扩散模型的情感图像内容生成

EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models

Shenzhen University(深圳大学)

Jingyuan Yang, Jiawei Feng, Hui Huang

AI总结:

本文提出情感图像内容生成(EICG)任务,通过构建情感空间并与CLIP对齐,设计属性损失和情感置信度,生成语义清晰且情感忠实的图像,在多项指标上优于现有方法。

摘要:

近年来,图像生成任务取得了显著进展,用户可以创作出高质量、令人惊叹的图像。然而,现有的文本到图像扩散模型擅长生成具体概念(如狗),但在处理更抽象的概念(如情感)时面临挑战。已有一些努力通过颜色和风格调整来修改图像情感,但在固定图像内容下有效传达情感方面存在局限。在本工作中,我们提出了情感图像内容生成(EICG),这是一项根据情感类别生成语义清晰且情感忠实图像的新任务。具体而言,我们构建了一个情感空间,并设计了一个映射网络将其与强大的对比语言-图像预训练(CLIP)空间对齐,为抽象情感提供具体解释。我们进一步提出了属性损失和情感置信度,以确保生成图像的语义多样性和情感保真度。我们的方法在定量和定性上均优于最先进的文本到图像方法,其中我们设计了三个自定义指标:情感准确率、语义清晰度和语义多样性。除生成外,我们的方法还可辅助情感理解并启发情感艺术设计。

英文摘要:

Recent years have witnessed remarkable progress in image generation task, where users can create visually astonishing images with high-quality. However, existing text-to-image diffusion models are proficient in generating concrete concepts (dogs) but encounter challenges with more abstract ones (emotions). Several efforts have been made to modify image emotions with color and style adjustments, facing limitations in effectively conveying emotions with fixed image contents. In this work, we introduce Emotional Image Content Generation (EICG), a new task to generate semantic-clear and emotion-faithful images given emotion categories. Specifically, we propose an emotion space and construct a mapping network to align it with the powerful Contrastive Language-Image Pre-training (CLIP) space, providing a concrete interpretation of abstract emotions. Attribute loss and emotion confidence are further proposed to ensure the semantic diversity and emotion fidelity of the generated images. Our method outperforms the state-of-the-art text-to-image approaches both quantitatively and qualitatively, where we derive three custom metrics, i.e., emotion accuracy, semantic clarity and semantic diversity. In addition to generation, our method can help emotion understanding and inspire emotional art design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑