arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.05258cs.CV
收藏

作为多视图表示学习数据源的生成模型

Generative Models as a Data Source for Multiview Representation Learning

Massachusetts Institute of Technology(麻省理工学院)

Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola

摘要:

生成模型现在能够生成高度逼真的图像,这些图像看起来几乎无法与它们所训练的数据区分开来。这就提出了一个问题:如果我们有足够好的生成模型,我们还需要数据集吗?我们研究这个问题的设置学习通用的视觉表示从黑盒生成模型,而不是直接从数据。给定一个现成的图像生成器,而不能访问其训练数据,我们从该生成器输出的样本中训练表示。我们比较了几个表示学习的方法,可以应用到这种设置,使用潜在的空间生成器生成多个“视图”相同的语义内容。我们表明,对于对比方法,这种多视图数据可以自然地用于识别正对(潜在空间中的附近)和负对(潜在空间中相距较远)。我们发现,由此产生的表示竞争对手,甚至优于那些直接从真实数据中学习,但良好的性能需要注意的采样策略和训练方法。生成模型可以被视为数据集的压缩和组织副本,我们设想未来会有越来越多的“模型动物园”激增,而数据集变得越来越笨拙,缺失或私有。本文提出了几种技术来处理视觉表示学习在这样的未来。代码可以在我们的项目页面https://ali-design.github.io/GenRep/上找到。

英文摘要:

Generative models are now capable of producing highly realistic images that look nearly indistinguishable from the data on which they are trained. This raises the question: if we have good enough generative models, do we still need datasets? We investigate this question in the setting of learning general-purpose visual representations from a black-box generative model rather than directly from data. Given an off-the-shelf image generator without any access to its training data, we train representations from the samples output by this generator. We compare several representation learning methods that can be applied to this setting, using the latent space of the generator to generate multiple "views" of the same semantic content. We show that for contrastive methods, this multiview data can naturally be used to identify positive pairs (nearby in latent space) and negative pairs (far apart in latent space). We find that the resulting representations rival or even outperform those learned directly from real data, but that good performance requires care in the sampling strategy applied and the training method. Generative models can be viewed as a compressed and organized copy of a dataset, and we envision a future where more and more "model zoos" proliferate while datasets become increasingly unwieldy, missing, or private. This paper suggests several techniques for dealing with visual representation learning in such a future. Code is available on our project page https://ali-design.github.io/GenRep/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑