arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.08343cs.LGcs.AIcs.CV
收藏

使用逆对比损失学习不变表示

Learning Invariant Representations using Inverse Contrastive Loss

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)

Aditya Kumar Akash, Vishnu Suresh Lokhande, Sathya N. Ravi, Vikas Singh

摘要:

学习不变表示是许多机器学习任务中关键的第一步。一种常见的方法对应于所谓的信息瓶颈原理,其中仔细选择和优化互信息的应用相关函数。不幸的是,在实践中,这些函数不适合于优化目的,因为这些损失与模型参数的度量结构无关。我们引入了一类损失的学习表示,是不变的一些无关变量的利益,通过反转类对比损失,即,逆对比度损失(ICL)。我们表明,如果无关变量是二进制的,那么优化ICL是等价于优化正则化的MMD发散。更一般地说,我们还表明,如果我们提供了一个度量的样本空间,我们制定的ICL可以分解成一个给定的距离度量的凸函数的总和。我们的实验结果表明,通过优化ICL获得的模型实现了显着更好的不变性的外来变量为一个固定的期望水平的准确性。在各种实验设置中,我们显示了ICL学习连续和离散无关变量的不变表示的适用性。

英文摘要:

Learning invariant representations is a critical first step in a number of machine learning tasks. A common approach corresponds to the so-called information bottleneck principle in which an application dependent function of mutual information is carefully chosen and optimized. Unfortunately, in practice, these functions are not suitable for optimization purposes since these losses are agnostic of the metric structure of the parameters of the model. We introduce a class of losses for learning representations that are invariant to some extraneous variable of interest by inverting the class of contrastive losses, i.e., inverse contrastive loss (ICL). We show that if the extraneous variable is binary, then optimizing ICL is equivalent to optimizing a regularized MMD divergence. More generally, we also show that if we are provided a metric on the sample space, our formulation of ICL can be decomposed into a sum of convex functions of the given distance metric. Our experimental results indicate that models obtained by optimizing ICL achieve significantly better invariance to the extraneous variable for a fixed desired level of accuracy. In a variety of experimental settings, we show applicability of ICL for learning invariant representations for both continuous and discrete extraneous variables.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑