Differential Contrastive Divergence

Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood

The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores. It has risen in popularity recently thanks to the impressive results obtained in image generation by parameterizing the distribution with Convolutional Neural Networks (CNN). However, the motivation and theoretical foundations behind modern EBMs are often absent from recent papers and this sometimes results in some confusion. In particular, the theoretical justifications behind the popular MCMC-based learning…

A Neighbourhood-Based Stopping Criterion for Contrastive Divergence Learning

Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence learning algorithm (CD), an approximation to the gradient of the data log-likelihood. A simple reconstruction error is often used as a stopping criterion for CD, although several authors \cite{schulz-et-al-Convergence-Contrastive-Divergence-2010-NIPSw, fischer-igel-Divergence-Contrastive-Divergence-2010-ICANN} have raised doubts concerning the feasibility of this procedure. In many cases the evolution curve …

RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence

Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task-irrelevant one that can hinder the generalization capability of representation. Motivated by this, we present a new robust contrastive learning scheme, coined RényiCL, which can effectively manage harder augmentations by utilizing Rényi divergence. Our …

Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE

We propose a number of new algorithms for learning deep energy models and demonstrate their properties. We show that our SteinCD performs well in term of test likelihood, while SteinGAN performs well in terms of generating realistic looking images. Our results suggest promising directions for learning better models by combining GAN-style methods with traditional energy-based learning.

A Contrastive Divergence for Combining Variational Inference and MCMC

We develop a method to combine Markov chain Monte Carlo (MCMC) and variational inference (VI), leveraging the advantages of both inference approaches. Specifically, we improve the variational distribution by running a few MCMC steps. To make inference tractable, we introduce the variational contrastive divergence (VCD), a new divergence that replaces the standard Kullback-Leibler (KL) divergence used in VI. The VCD captures a notion of discrepancy between the initial variational distribution and its improved version (obtained after running the MCMC steps), and it converges asymptotically to th…

噪声对比估计与对比散度之间的联系

On the connection between Noise-Contrastive Estimation and Contrastive Divergence

本文证明排序NCE和条件NCE本质上可视为最大似然估计方法，其中RNCE等价于结合条件重要性采样的ML估计，且二者均为对比散度的特例，从而打通了NCE与ML-IS、CD方法之间的联系。

使用 Beta 散度的变分自监督对比学习

Variational Self-Supervised Contrastive Learning Using Beta Divergence

针对多标签场景下未标注含噪数据的判别表示学习问题，提出结合 beta 散度的变分对比自监督方法 VCL，并在人脸理解多标签实验中取得优于现有自监督方法的准确率。

对比散度算法的近最优性

Near-Optimality of Contrastive Divergence Algorithms

本文对对比散度算法进行非渐近分析，证明在正则性假设下其收敛速率可提升至参数速率，且渐近方差接近Cramér-Rao下界，实现近最优性。

持久对比散度算法的时间一致收敛界

Uniform-in-time convergence bounds for Persistent Contrastive Divergence algorithms

本研究提出PCD的连续时间公式，将其表示为多尺度SDE系统，推导了PCD迭代与MLE解误差的显式时间一致界，还基于S-ROCK积分器实现高效方案，提出带显式误差保证的EBM训练新方法。

基于对比散度的贝叶斯伪核心集

Bayesian Pseudo-Coresets via Contrastive Divergence

针对贝叶斯伪核心集构建中前向KL散度依赖近似或高维MCMC混合缓慢的问题，提出用对比散度优化伪核心集，支持有限步MCMC并在多数据集上优于现有BPC方法。

可扩展的最大熵人口合成 via 持续对比散度

Scalable Maximum Entropy Population Synthesis via Persistent Contrastive Divergence

本文提出GibbsPCDSolver方法，通过持续对比散度实现高效最大熵人口合成，解决了传统方法在计算效率和样本多样性上的不足。

基于分歧的多视图对比学习相似度函数

Divergence-Based Similarity Function for Multi-View Contrastive Learning

本文提出基于分歧的多视图对比学习相似度函数，通过分布分歧建模多视图联合结构，提升多种任务性能并提高效率。

广义对比散度：通过逆强化学习联合训练基于能量的模型与扩散模型

Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning

本文提出 GCD 目标，将 EBM 与扩散模型联合训练建模为极小极大博弈，并建立其与逆强化学习的等价关系，从而无需 MCMC 即可训练 EBM 并改善扩散模型样本质量。

保守目标模型是一种特殊的基于对比散度的能量模型

Conservative objective models are a special kind of contrastive divergence-based energy model

本文从理论上证明保守目标模型是一种特殊的基于对比散度的能量模型，提出用Langevin MCMC采样器替换梯度上升采样器，并解耦模型以分别建模无条件与条件概率，从而获得更好的样本。

Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator

Generative Adversarial Networks (GANs) have shown great promise in modeling high dimensional data. The learning objective of GANs usually minimizes some measure discrepancy, \textit{e.g.}, $f$-divergence~($f$-GANs) or Integral Probability Metric~(Wasserstein GANs). With $f$-divergence as the objective function, the discriminator essentially estimates the density ratio, and the estimated ratio proves useful in further improving the sample quality of the generator. However, how to leverage the information contained in the discriminator of Wasserstein GANs (WGAN) is less explored. In this paper, …

Population-Contrastive-Divergence: Does Consistency help with RBM training?

Estimating the log-likelihood gradient with respect to the parameters of a Restricted Boltzmann Machine (RBM) typically requires sampling using Markov Chain Monte Carlo (MCMC) techniques. To save computation time, the Markov chains are only run for a small number of steps, which leads to a biased estimate. This bias can cause RBM training algorithms such as Contrastive Divergence (CD) learning to deteriorate. We adopt the idea behind Population Monte Carlo (PMC) methods to devise a new RBM training algorithm termed Population-Contrastive-Divergence (pop-CD). Compared to CD, it leads to a consi…

Contrastive Divergence Learning is a Time Reversal Adversarial Game

Contrastive divergence (CD) learning is a classical method for fitting unnormalized statistical models to data samples. Despite its wide-spread use, the convergence properties of this algorithm are still not well understood. The main source of difficulty is an unjustified approximation which has been used to derive the gradient of the loss. In this paper, we present an alternative derivation of CD that does not require any approximation and sheds new light on the objective that is actually being optimized by the algorithm. Specifically, we show that CD is an adversarial learning procedure, whe…

对比学习中的锚点散度用于语义几何

Anchor Divergence for Semantic Geometry in Contrastive Learning

本文提出锚点散度方法，通过对比学习、指数族与信息几何的对应，在固定表示上定义上下文相关的语义几何，实验证明其能高效指定特定语义相似性。

Weighted Contrastive Divergence

Learning algorithms for energy based Boltzmann architectures that rely on gradient descent are in general computationally prohibitive, typically due to the exponential number of terms involved in computing the partition function. In this way one has to resort to approximation schemes for the evaluation of the gradient. This is the case of Restricted Boltzmann Machines (RBM) and its learning algorithm Contrastive Divergence (CD). It is well-known that CD has a number of shortcomings, and its approximation to the gradient has several drawbacks. Overcoming these defects has been the basis of much…

Training Deep Energy-Based Models with f-Divergence Minimization

Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging because of the intractable partition function. They are typically trained via maximum likelihood, using contrastive divergence to approximate the gradient of the KL divergence between data and model distribution. While KL divergence has many desirable properties, other f-divergences have shown advantages in training implicit density generative models such as generative adversarial networks. In this paper, we propose a general variational framework termed f-EBM to train EBMs using a…

Convergence of Contrastive Divergence with Annealed Learning Rate in Exponential Family

In our recent paper, we showed that in exponential family, contrastive divergence (CD) with fixed learning rate will give asymptotically consistent estimates \cite{wu2016convergence}. In this paper, we establish consistency and convergence rate of CD with annealed learning rate $η_t$. Specifically, suppose CD-$m$ generates the sequence of parameters $\{θ_t\}_{t \ge 0}$ using an i.i.d. data sample $\mathbf{X}_1^n \sim p_{θ^*}$ of size $n$, then $δ_n(\mathbf{X}_1^n) = \limsup_{t \to \infty} \Vert \sum_{s=t_0}^t η_s θ_s / \sum_{s=t_0}^t η_s - θ^* \Vert$ converges in probability to 0 at a rate of …

Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism

Recently, a variety of methods under the name of non-contrastive learning (like BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric architectural designs, aligning positive pairs alone is sufficient to attain good performance in self-supervised visual learning. Despite some understandings of some specific modules (like the predictor in BYOL), there is yet no unified theoretical understanding of how these seemingly different asymmetric designs can all avoid feature collapse, particularly considering methods that also work without the predictor (like DINO). In this work, we p…

Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data

Conceived in the early 1990s, Experience Replay (ER) has been shown to be a successful mechanism to allow online learning algorithms to reuse past experiences. Traditionally, ER can be applied to all machine learning paradigms (i.e., unsupervised, supervised, and reinforcement learning). Recently, ER has contributed to improving the performance of deep reinforcement learning. Yet, its application to many practical settings is still limited by the memory requirements of ER, necessary to explicitly store previous observations. To remedy this issue, we explore a novel approach, Online Contrastive…

对比表示学习的几何力学：对齐势、熵分散和跨模态散度

The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence

本文通过测度论框架，在大批量极限下证明InfoNCE目标与确定性能量景观的等价性，揭示单模态与对称多模态之间的几何分岔，并指出跨模态散度项导致模态间隙。

Adiabatic Persistent Contrastive Divergence Learning

This paper studies the problem of parameter learning in probabilistic graphical models having latent variables, where the standard approach is the expectation maximization algorithm alternating expectation (E) and maximization (M) steps. However, both E and M steps are computationally intractable for high dimensional data, while the substitution of one step to a faster surrogate for combating against intractability can often cause failure in convergence. We propose a new learning algorithm which is computationally efficient and provably ensures convergence to a correct optimum. Its key idea is…

面向鲁棒属性图聚类的混合协同增强与对比样本自适应差分感知

Hybrid-Collaborative Augmentation and Contrastive Sample Adaptive-Differential Awareness for Robust Attributed Graph Clustering

针对现有对比属性图聚类忽略边级增强及样本对差异的问题，提出融合混合协同增强与对比样本自适应差分感知的鲁棒聚类方法，在六个基准数据集上性能优于现有最优方法。

用扩散对比散度训练基于能量的模型

Training Energy-Based Models with Diffusion Contrastive Divergences

本文提出扩散对比散度（DCD）训练能量模型，用不依赖EBM参数的扩散过程替代朗之万动力学，在提升计算效率的同时避免短程MCMC的梯度项问题，并在合成数据、图像去噪与生成任务中验证了其优势。

$f$-MICL：理解与泛化基于InfoNCE的对比学习

$f$-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning

本文提出 $f$-MICL 框架，通过 $f$-散度将基于KL的互信息泛化，并推导出 $f$-高斯相似度，在视觉和自然语言基准上超越 InfoNCE。

具有差分隐私的对比可解释聚类

Contrastive Explainable Clustering with Differential Privacy

本文提出将对比解释与差分隐私结合用于k-median和k-means聚类，证明差分隐私解释能达到与非隐私解释基本相同的效用界限，在保护隐私的同时提供个性化且不显著损害聚类效用的解释。

对比学习的一种基于近似充分统计量的统计理论

A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics

本文提出基于近似充分统计量的理论框架，分析SimCLR等对比学习，证明其编码器近似充分且可有效适应下游任务，并给出线性回归和主题分类示例。

通过局部模态初始化的无偏对比散度实现深度玻尔兹曼机的端到端训练

End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive Divergence with Local Mode Initialization

针对深度玻尔兹曼机梯度估计有偏问题，提出基于 Metropolis-Hastings 耦合与局部模态初始化的无偏对比散度，实现无需贪心预训练的端到端训练，并在 MNIST 上达到 10.33 的 FID。

差分信息驱动的样本选择加速多模态对比学习

Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning

本文提出差分信息驱动的样本选择方法DISSect，利用当前模型与历史模型预测相关性的差分高效判别噪声对应，从而在冷启动和在线场景下加速多模态对比学习，并在多个基准上取得一致优势。

PICL：用于偏微分方程的物理信息对比学习

PICL: Physics Informed Contrastive Learning for Partial Differential Equations

本研究提出基于广义对比损失的物理信息对比预训练框架PICL，通过控制方程系数和物理信息驱动的系统演化提升神经算子在多方程上的泛化能力，显著提高了Fourier Neural Operator在多类偏微分方程任务中的精度。

关于没有局部时钟的差分时间差学习发散性研究

On the Divergence of Differential Temporal Difference Learning without Local Clocks

本文探讨了在平均回报强化学习中，差分时间差学习在局部时钟下收敛但在全局时钟下发散的现象，解决了Wan等[2021]和Blaser等[2026]提出的问题。

超越I-Con：探索表示学习中距离度量的新维度

Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning

Beyond I-Con框架通过探索替代统计散度，改进了表示学习中的对比学习和降维任务，展示了不同散度选择对优化效果的重要性。

Auditing Differential Privacy in High Dimensions with the Kernel Quantum Rényi Divergence

Differential privacy (DP) is the de facto standard for private data release and private machine learning. Auditing black-box DP algorithms and mechanisms to certify whether they satisfy a certain DP guarantee is challenging, especially in high dimension. We propose relaxations of differential privacy based on new divergences on probability distributions: the kernel Rényi divergence and its regularized version. We show that the regularized kernel Rényi divergence can be estimated from samples even in high dimensions, giving rise to auditing procedures for $\varepsilon$-DP, $(\varepsilon,δ)$-DP …

Learning RBM with a DC programming Approach

By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of the above formulation and the hyperparameters of the two algorithms can be chosen such that the amount of computation per mini-batch is identical. We show that for a given computational budget the proposed algorithm almost always reaches a higher log-likelihood more rapidly, compar…

递归 KL 散度优化：一种面向表征学习的动态框架

Recursive KL Divergence Optimization: A Dynamic Framework for Representation Learning

本文提出 Recursive KL Divergence Optimization (RKDO) 动态框架，将表征学习构建为跨数据邻域的 KL 散度演化，实验证明其相比静态方法损失值约低30%、计算资源减少60%至80%，为资源受限应用提供了更高效的优化空间。

Local Differential Privacy Is Equivalent to Contraction of $E_γ$-Divergence

We investigate the local differential privacy (LDP) guarantees of a randomized privacy mechanism via its contraction properties. We first show that LDP constraints can be equivalently cast in terms of the contraction coefficient of the $E_γ$-divergence. We then use this equivalent formula to express LDP guarantees of privacy mechanisms in terms of contraction coefficients of arbitrary $f$-divergences. When combined with standard estimation-theoretic tools (such as Le Cam's and Fano's converse methods), this result allows us to study the trade-off between privacy and utility in several testing …

Posterior Differential Regularization with f-divergence for Improving Model Robustness

We address the problem of enhancing model robustness through regularization. Specifically, we focus on methods that regularize the model posterior difference between clean and noisy inputs. Theoretically, we provide a connection of two recent methods, Jacobian Regularization and Virtual Adversarial Training, under this framework. Additionally, we generalize the posterior differential regularization to the family of $f$-divergences and characterize the overall regularization framework in terms of Jacobian matrix. Empirically, we systematically compare those regularizations and standard BERT tra…

scBeacon：通过对比孪生网络识别跨生物学条件的配对细胞簇实现单细胞生物标志物提取

scBeacon: single-cell biomarker extraction via identifying paired cell clusters across biological conditions with contrastive siamese networks

scBeacon利用对比孪生网络和VQ-VAE，通过无监督识别跨条件的配对细胞簇，实现精准的单细胞差异基因分析，从而提取关键生物标志物，提升诊断准确性。

DP-DCAN：用于单细胞聚类的差分隐私深度对比自编码器网络

DP-DCAN: Differentially Private Deep Contrastive Autoencoder Network for Single-cell Clustering

针对现有全网络扰动的差分隐私方法用于单细胞聚类时性能损耗大的问题，提出仅对部分网络加噪的DP-DCAN模型，在6个数据集上性能优于传统方案且对抗攻击鲁棒性强。

假设检验解释与Rényi差分隐私

Hypothesis Testing Interpretations and Renyi Differential Privacy

本文研究基于Rényi散度的差分隐私松弛形式的假设检验解释，提出了满足该类解释的散度条件，推导了改进的隐私定义转换规则。

A Better Bound Gives a Hundred Rounds: Enhanced Privacy Guarantees via $f$-Divergences

We derive the optimal differential privacy (DP) parameters of a mechanism that satisfies a given level of Rényi differential privacy (RDP). Our result is based on the joint range of two $f$-divergences that underlie the approximate and the Rényi variations of differential privacy. We apply our result to the moments accountant framework for characterizing privacy guarantees of stochastic gradient descent. When compared to the state-of-the-art, our bounds may lead to about 100 more stochastic gradient descent iterations for training deep learning models for the same privacy budget.

Towards the Generalization of Contrastive Self-Supervised Learning

Recently, self-supervised learning has attracted great attention, since it only requires unlabeled data for model training. Contrastive learning is one popular method for self-supervised learning and has achieved promising empirical performance. However, the theoretical understanding of its generalization ability is still limited. To this end, we define a kind of $(σ,δ)$-measure to mathematically quantify the data augmentation, and then provide an upper bound of the downstream classification error rate based on the measure. It reveals that the generalization ability of contrastive self-supervi…

A Differential Geometry Perspective on Orthogonal Recurrent Models

Recently, orthogonal recurrent neural networks (RNNs) have emerged as state-of-the-art models for learning long-term dependencies. This class of models mitigates the exploding and vanishing gradients problem by design. In this work, we employ tools and insights from differential geometry to offer a novel perspective on orthogonal RNNs. We show that orthogonal RNNs may be viewed as optimizing in the space of divergence-free vector fields. Specifically, based on a well-known result in differential geometry that relates vector fields and linear operators, we prove that every divergence-free vecto…

Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences

Differential privacy comes equipped with multiple analytical tools for the design of private data analyses. One important tool is the so-called "privacy amplification by subsampling" principle, which ensures that a differentially private mechanism run on a random subsample of a population provides higher privacy guarantees than when run on the entire population. Several instances of this principle have been studied for different random subsampling methods, each with an ad-hoc analysis. In this paper we present a general method that recovers and improves prior analyses, yields lower bounds and …

Contraction of $E_γ$-Divergence and Its Applications to Privacy

We investigate the contraction coefficients derived from strong data processing inequalities for the $E_γ$-divergence. By generalizing the celebrated Dobrushin's coefficient from total variation distance to $E_γ$-divergence, we derive a closed-form expression for the contraction of $E_γ$-divergence. This result has fundamental consequences in two privacy settings. First, it implies that local differential privacy can be equivalently expressed in terms of the contraction of $E_γ$-divergence. This equivalent formula can be used to precisely quantify the impact of local privacy in (Bayesian and m…

量子信息排序与差分隐私

Quantum Information Ordering and Differential Privacy

本文提出通过量子态信息性顺序定义量子差分隐私，研究隐私化假设检验和参数估计的限制，并建立量子通道的收缩界。

Privacy Amplification of Iterative Algorithms via Contraction Coefficients

We investigate the framework of privacy amplification by iteration, recently proposed by Feldman et al., from an information-theoretic lens. We demonstrate that differential privacy guarantees of iterative mappings can be determined by a direct application of contraction coefficients derived from strong data processing inequalities for $f$-divergences. In particular, by generalizing the Dobrushin's contraction coefficient for total variation distance to an $f$-divergence known as $E_γ$-divergence, we derive tighter bounds on the differential privacy parameters of the projected noisy stochastic…