arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.07734cs.CVcs.AI
收藏

TreeGAN:将类层次结构扩展到图像生成中

TreeGAN: Incorporating Class Hierarchy into Image Generation

University of California, San Diego(加州大学圣地亚哥分校);Beijing University of Technology(北京工业大学)

Ruisi Zhang, Luntian Mou, Pengtao Xie

摘要:

条件图像生成(CIG)是计算机视觉和机器学习领域的一个研究热点。给定一个类,CIG将该类的名称作为输入,并生成属于该类的一组图像。在现有的CIG工作中,对于不同的类,它们对应的图像是独立生成的,没有考虑类之间的关系。在现实世界的应用中,类被组织成一个层次结构,它们的层次关系是用于生成高保真图像的信息。在本文中,我们的目标是利用类层次结构的条件图像生成。我们提出了两种结合类层次结构的方法:事前控制和事后约束。在先验控制中,我们首先对类层次结构进行编码,然后将其作为先验输入到条件生成器中以生成图像。在后约束中,在生成图像之后,我们测量它们与类层次的一致性,并使用一致性得分来指导生成器的训练。基于这两个思想,我们提出了一个TreeGAN模型,它由三个模块组成:(1)类层次编码器(CHE),它以类的层次结构和文本名称为输入,学习每个类的嵌入,嵌入捕捉类之间的层次关系;(2)条件图像生成器(CIG),其将类的CHE生成的嵌入作为输入,并且生成属于该类的图像集合;(3)一致性检查器,对生成的图像进行层次分类,并检查生成的图像是否与类层次兼容;一致性得分用于指导CIG生成层次兼容的图像。在各种数据集上的实验证明了我们方法的有效性。

英文摘要:

Conditional image generation (CIG) is a widely studied problem in computer vision and machine learning. Given a class, CIG takes the name of this class as input and generates a set of images that belong to this class. In existing CIG works, for different classes, their corresponding images are generated independently, without considering the relationship among classes. In real-world applications, the classes are organized into a hierarchy and their hierarchical relationships are informative for generating high-fidelity images. In this paper, we aim to leverage the class hierarchy for conditional image generation. We propose two ways of incorporating class hierarchy: prior control and post constraint. In prior control, we first encode the class hierarchy, then feed it as a prior into the conditional generator to generate images. In post constraint, after the images are generated, we measure their consistency with the class hierarchy and use the consistency score to guide the training of the generator. Based on these two ideas, we propose a TreeGAN model which consists of three modules: (1) a class hierarchy encoder (CHE) which takes the hierarchical structure of classes and their textual names as inputs and learns an embedding for each class; the embedding captures the hierarchical relationship among classes; (2) a conditional image generator (CIG) which takes the CHE-generated embedding of a class as input and generates a set of images belonging to this class; (3) a consistency checker which performs hierarchical classification on the generated images and checks whether the generated images are compatible with the class hierarchy; the consistency score is used to guide the CIG to generate hierarchy-compatible images. Experiments on various datasets demonstrate the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑