惊奇-雷尼自由能
Surprisal-Rényi Free Energy
AI总结:
本文提出惊奇-雷尼自由能,通过揭示均值-方差权衡和大偏差结构,澄清了前向和反向KL极限背后的几何和大偏差特性。
摘要:
前向和反向的Kullback-Leibler(KL)发散出现在学习和推断中作为极限目标,却诱导出显著不同的归纳偏置,这些偏置无法仅在期望水平上解释。在本工作中,我们引入了惊奇-雷尼自由能(SRFE),这是一种基于对数矩的似然比函数,它不属于$f$-发散的范畴。我们展示了SRFE能够恢复前向和反向KL发散作为奇异端点极限,并推导了围绕这两个极限的局部展开,其中对数似然比的方差出现在一阶修正中。这揭示了显式均值-方差权衡,支配着偏离KL主导区域的偏差。我们进一步建立了SRFE作为加权KL发散和的唯一极小值的吉布斯型变分特征,并证明SRFE直接通过切尔诺夫型界控制超额编码长度的大偏差,从而获得精确的最小描述长度解释。这些结果将SRFE识别为一种对变异性及尾部敏感的自由能函数,阐明了前向和反向KL极限背后的几何和大偏差结构,而无需统一或包含不同的学习框架。
英文摘要:
The forward and reverse Kullback-Leibler (KL) divergences arise as limiting objectives in learning and inference, yet they induce markedly different inductive biases that cannot be explained by expectation terms alone. We introduce the Surprisal-Rényi Free Energy (SRFE), a normalized Rényi-Chernoff log-moment functional of the likelihood ratio. Although SRFE is algebraically related to Rényi divergence for fixed values of the order parameter, its endpoint-symmetric normalization yields a distinct free-energy viewpoint for analyzing how forward- and reverse KL behavior emerges near singular limits. We show that SRFE recovers the forward and reverse KL divergences as endpoint limits and derive local expansions around both regimes in which the variance of the log-likelihood ratio appears as a first-order correction away from the KL objective. These expansions reveal an explicit mean-variance tradeoff governing departures from KL-dominated behavior. We further establish a Gibbs-type variational characterization in terms of weighted KL projections and prove that SRFE controls large deviations of excess code length through Chernoff-type bounds, giving a Minimum Description Length interpretation of the functional. We evaluate SRFE on two synthetic settings (two-dimensional Gaussian-mixture variational inference and multi-class classification) and the long-tailed CIFAR-10 and CIFAR-100 (CIFAR-LT) datasets. On CIFAR-LT, retraining the best configuration of each loss over five seeds, SRFE matches cross-entropy accuracy on CIFAR-10-LT and gives mixed results on CIFAR-100-LT, improving few-shot accuracy at one of three imbalance factors. Together, these results position SRFE as a variance- and tail-sensitive, normalized Rényi-Chernoff free energy that clarifies the geometric, statistical, and large-deviation structure underlying the transition between forward- and reverse-KL limits.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。