arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.10477cs.CVcs.IRcs.LG
收藏

从文本生成组合颜色表示

Generating Compositional Color Representations from Text

Stanford University(斯坦福大学);Carnegie Mellon University(卡内基梅隆大学);Adobe Research(奥多比研究院);Adobe(奥多比);Advanced Micro Devices Inc.(超威半导体公司)

Paridhi Maheshwari, Nihal Jain, Praneetha Vaddamanu, Dhananjay Raut, Shraiysh Vaishay, Vishwa Vinay

摘要:

我们认为跨模态的任务,产生颜色表示的文本短语。由于图像搜索引擎上很大一部分用户查询都遵循(属性,对象)结构,我们提出了一个生成对抗网络,为这种二元组生成颜色配置文件。我们设计管道来学习组合-将可见的属性和对象组合为不可见的对的能力。我们从现有的公共资源中提出了一种新的数据集管理管道。我们描述了如何使用图传播技术编译一组感兴趣的短语,然后映射到图像。虽然这个数据集专门用于我们对颜色的研究,但该方法可以扩展到其他感兴趣的视觉维度。我们提供了详细的消融研究,测试我们的GAN架构的行为与损失函数的对比学习文献。我们表明,生成模型实现了较低的弗雷歇初始距离比歧视,因此预测的颜色配置文件,更好地匹配那些从真实图像。最后,我们展示了图像检索和分类的改进性能,表明颜色在这些下游任务中起着至关重要的作用。

英文摘要:

We consider the cross-modal task of producing color representations for text phrases. Motivated by the fact that a significant fraction of user queries on an image search engine follow an (attribute, object) structure, we propose a generative adversarial network that generates color profiles for such bigrams. We design our pipeline to learn composition - the ability to combine seen attributes and objects to unseen pairs. We propose a novel dataset curation pipeline from existing public sources. We describe how a set of phrases of interest can be compiled using a graph propagation technique, and then mapped to images. While this dataset is specialized for our investigations on color, the method can be extended to other visual dimensions where composition is of interest. We provide detailed ablation studies that test the behavior of our GAN architecture with loss functions from the contrastive learning literature. We show that the generative model achieves lower Frechet Inception Distance than discriminative ones, and therefore predicts color profiles that better match those from real images. Finally, we demonstrate improved performance in image retrieval and classification, indicating the crucial role that color plays in these downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑