InfoNCE 是识别参数化模型中的变分推断
InfoNCE is variational inference in a recognition parameterised model
AI总结:
本文证明 InfoNCE 目标等价于识别参数化模型中的 ELBO,并从 VAE 视角为 InfoNCE 提供了比互信息更合理的理论动机。
摘要:
本文证明 InfoNCE 目标等价于一类新的概率生成模型——识别参数化模型(RPM)中的 ELBO。当学习最优先验时,RPM 的 ELBO 等于互信息(MI;相差一个常数),从而与 InfoNCE 等已有的自监督学习方法建立了联系。然而,实际的 InfoNCE 方法并不使用 MI 作为目标;MI 对任意可逆变换具有不变性,因此使用 MI 目标可能导致高度纠缠的表示(Tschannen 等,2019)。相反,实际的 InfoNCE 目标是 MI 的一个简化下界,即使在无限样本极限下也是松散的。因此,一个有效工作的目标(即实际的 InfoNCE 目标)似乎被解释为一个不工作的目标(即会产生任意纠缠表示的真实 MI)的松散下界。我们为实际的 InfoNCE 目标给出了另一种动机。特别地,我们证明在无限样本极限下,并且对于特定的先验选择,实际的 InfoNCE 目标等于 ELBO(相差一个常数);而 ELBO 等于具有确定性识别模型的边缘似然。因此,我们认为我们的 VAE 视角比 MI 视角为 InfoNCE 提供了更好的动机,因为实际的 InfoNCE 目标仅被 MI 松散地界定,但等于 ELBO/边缘似然(相差一个常数)。
英文摘要:
Here, we show that the InfoNCE objective is equivalent to the ELBO in a new class of probabilistic generative model, the recognition parameterised model (RPM). When we learn the optimal prior, the RPM ELBO becomes equal to the mutual information (MI; up to a constant), establishing a connection to pre-existing self-supervised learning methods such as InfoNCE. However, practical InfoNCE methods do not use the MI as an objective; the MI is invariant to arbitrary invertible transformations, so using an MI objective can lead to highly entangled representations (Tschannen et al., 2019). Instead, the actual InfoNCE objective is a simplified lower bound on the MI which is loose even in the infinite sample limit. Thus, an objective that works (i.e. the actual InfoNCE objective) appears to be motivated as a loose bound on an objective that does not work (i.e. the true MI which gives arbitrarily entangled representations). We give an alternative motivation for the actual InfoNCE objective. In particular, we show that in the infinite sample limit, and for a particular choice of prior, the actual InfoNCE objective is equal to the ELBO (up to a constant); and the ELBO is equal to the marginal likelihood with a deterministic recognition model. Thus, we argue that our VAE perspective gives a better motivation for InfoNCE than MI, as the actual InfoNCE objective is only loosely bounded by the MI, but is equal to the ELBO/marginal likelihood (up to a constant).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。