基于能量的模型及其配分函数的联合学习
Joint Learning of Energy-based Models and their Partition Function
AI总结:
本文提出联合学习能量模型及其对数配分函数的神经网络参数化方法,为组合上很大的离散空间提供了无需MCMC的可处理EBMs学习准则,并首次实现sparsemax损失在该类空间中的可处理优化。
摘要:
基于能量的模型(EBMs)为使用神经网络参数化概率分布提供了一种灵活的框架。然而,由于需要计算配分函数(归一化常数),通过精确最大似然估计(MLE)学习EBMs通常是不可行的。在本文中,我们提出了一种新的公式,用于在组合上很大的离散空间(如集合或排列)中近似学习概率EBMs。我们的核心思想是联合学习一个能量模型及其对数配分,两者均以神经网络进行参数化。我们的方法不仅提供了一种新颖的可处理目标准则,可以通过随机梯度下降(不依赖MCMC)学习EBMs,而且还提供了一种在未见数据点上估计对数配分函数的新方法。在理论方面,我们表明,当在连续函数空间中优化时,我们的方法能够恢复最优的MLE解。此外,我们表明我们的方法自然地扩展到更广泛的Fenchel-Young损失族,使我们能够在组合上很大的空间中获得第一个可处理的sparsemax损失优化方法。我们在多标签分类和标签排序上展示了我们的方法。
英文摘要:
Energy-based models (EBMs) offer a flexible framework for parameterizing probability distributions using neural networks. However, learning EBMs by exact maximum likelihood estimation (MLE) is generally intractable, due to the need to compute the partition function (normalization constant). In this paper, we propose a novel formulation for approximately learning probabilistic EBMs in combinatorially-large discrete spaces, such as sets or permutations. Our key idea is to jointly learn both an energy model and its log-partition, both parameterized as a neural network. Our approach not only provides a novel tractable objective criterion to learn EBMs by stochastic gradient descent (without relying on MCMC), but also a novel means to estimate the log-partition function on unseen data points. On the theoretical side, we show that our approach recovers the optimal MLE solution when optimizing in the space of continuous functions. Furthermore, we show that our approach naturally extends to the broader family of Fenchel-Young losses, allowing us to obtain the first tractable method for optimizing the sparsemax loss in combinatorially-large spaces. We demonstrate our approach on multilabel classification and label ranking.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。