Tomohiro Nishiyama
The divergence minimization problem plays an important role in various fields. In this note, we focus on differentiable and strictly convex divergences. For some minimization problems, we show the minimizer conditions and the uniqueness of the minimizer without assuming a specific form of divergences. Furthermore, we show geometric properties related to the minimization problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
We develop and analyze $M$-estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a non-asymptotic variational characterization of $f$-divergences, which allows the problem of estimating divergences to be tackled via convex empirical risk optimization. The resulting estimators are simple to implement, requiring only the solution of standard convex programs. We present an analysis of consistency and convergence for these estimators. Given conditions only on the ratios of densities, we show that our estimators can achieve …
We generalize the family of $α$-divergences using a pair of strictly comparable weighted means. In particular, we obtain the $1$-divergence in the limit case $α\rightarrow 1$ (a generalization of the Kullback-Leibler divergence) and the $0$-divergence in the limit case $α\rightarrow 0$ (a generalization of the reverse Kullback-Leibler divergence). We state the condition for a pair of quasi-arithmetic means to be strictly comparable, and report the formula for the quasi-arithmetic $α$-divergences and its subfamily of bipower homogeneous $α$-divergences which belong to the Csisár's $f$-divergenc…
The generalized Kullback-Leibler divergence (K-Ld) in Tsallis statistics [constrained by the additive duality of generalized statistics (dual generalized K-Ld)] is here reconciled with the theory of Bregman divergences for expectations defined by normal averages, within a measure-theoretic framework. Specifically, it is demonstrated that the dual generalized K-Ld is a scaled Bregman divergence. The Pythagorean theorem is derived from the minimum discrimination information-principle using the dual generalized K-Ld as the measure of uncertainty, with constraints defined by normal averages. The m…
The importance of power-law distributions is attributed to the fact that most of the naturally occurring phenomenon exhibit this distribution. While exponential distributions can be derived by minimizing KL-divergence w.r.t some moment constraints, some power law distributions can be derived by minimizing some generalizations of KL-divergence (more specifically some special cases of Csiszár f-divergences). Divergence minimization is very well studied in information theoretical approaches to statistics. In this work we study properties of minimization of Tsallis divergence, which is a special c…
Divergence-Minimization for Latent-Structure Models: Monotone Operators, Contraction Guarantees, and Robust Inference
本文提出发散最小化框架，通过残差调整发散量实现稳健推断，结合惩罚准则和样本分割确定混合组件数量，展示其在混合和图像分割任务中的稳定性与准确性。
We first introduce the class of strictly quasiconvex and strictly quasiconcave Jensen divergences which are oriented (asymmetric) distances, and study some of their properties. We then define the strictly quasiconvex Bregman divergences as the limit case of scaled and skewed quasiconvex Jensen divergences, and report a simple closed-form formula which shows that these divergences are only pseudo-divergences at countably many inflection points of the generators. To remedy this problem, we propose the $δ$-averaged quasiconvex Bregman divergences which integrate the pseudo-divergences over a smal…
Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences
本文研究带 f-散度正则化的经验风险最小化，在温和条件下给出唯一最优测度，并证明不同 f-散度正则化可通过风险函数变换相互等价。
We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes. Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection. We propose a general imitation learning framework for estimating and minimizi…
Bregman-divergence-based Arimoto-Blahut algorithm
本文将广义Arimoto-Blahut算法推广到Bregman散度系统，提出免最小化迭代算法，并应用于经典与量子率失真理论中最优条件分布的数值推导。
Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization
本文通过局部曲率分析揭示f-散度正则化与锐度感知最小化在参数和输入扰动下的等价性，并实验证明最大曲率惩罚可提升泛化性能。
Alternating Minimization Schemes for Computing Rate-Distortion-Perception Functions with $f$-Divergence Perception Constraints
针对带$f$-散度感知约束的率失真感知函数计算，提出两种交替最小化算法（牛顿法和松弛法），证明全局收敛及指数收敛速度，并用数值实验验证。
In this paper, we provide three applications for $f$-divergences: (i) we introduce Sanov's upper bound on the tail probability of the sum of independent random variables based on super-modular $f$-divergence and show that our generalized Sanov's bound strictly improves over ordinary one, (ii) we consider the lossy compression problem which studies the set of achievable rates for a given distortion and code length. We extend the rate-distortion function using mutual $f$-information and provide new and strictly better bounds on achievable rates in the finite blocklength regime using super-modula…
The (global) Lipschitz smoothness condition is crucial in establishing the convergence theory for most optimization methods. Unfortunately, most machine learning and signal processing problems are not Lipschitz smooth. This motivates us to generalize the concept of Lipschitz smoothness condition to the relative smoothness condition, which is satisfied by any finite-order polynomial objective function. Further, this work develops new Bregman-divergence based algorithms that are guaranteed to converge to a second-order stationary point for any relatively smooth problem. In addition, the proposed…
To learn directed behaviors in complex environments, intelligent agents need to optimize objective functions. Various objectives are known for designing artificial agents, including task rewards and intrinsic motivation. However, it is unclear how the known objectives relate to each other, which objectives remain yet to be discovered, and which objectives better describe the behavior of humans. We introduce the Action Perception Divergence (APD), an approach for categorizing the space of possible objective functions for embodied agents. We show a spectrum that reaches from narrow to general ob…
Minimization problems with respect to a one-parameter family of generalized relative entropies are studied. These relative entropies, which we term relative $α$-entropies (denoted $\mathscr{I}_α$), arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler divergence). Just like relative entropy, these relative $α$-entropies behave like squared Euclidean distance and satisfy the Pythagorean property. Minimiz…
A Structural Characterization of Entropy Functionals
该研究通过测度论框架刻画熵泛函的容许性，解决雷尼提出的广义均值替代问题，区分出雷尼族与香农熵，还得到新的容许熵和散度族。
Computation of Rate-Distortion-Perception Function under f-Divergence Perception Constraints
本文针对离散无记忆信源在f-散度感知约束下的率失真感知函数计算问题，利用凸规划性质提出带收敛保证的交替最小化松弛算法，推导指数收敛条件并通过数值模拟验证理论结果。
Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization
本文提出通过KL散度度量高斯混合模型发布精度，结合差分隐私机制，在保证隐私安全的同时保持模型效用。
A precise bare simulation approach to the minimization of some distances. II. Further Foundations
本文提出无维度扩展裸模拟方法，可在任意维离散设置下精确求解大量有向距离与广义熵的约束优化，同时得到最优值与优化器。
Bregman divergence based em algorithm and its application to classical and quantum rate distortion theory
本文在 Bregman 散度框架下提出通用 em 算法，证明其收敛性与速度，并用于直接求解经典及量子率失真理论中代价约束下的互信息最小化问题。
In information theory -- as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition -- many flexibilizations of the omnipresent Kullback-Leibler information distance (relative entropy) and of the closely related Shannon entropy have become frequently used tools. To tackle corresponding constrained minimization (respectively maximization) problems by a newly developed dimension-free bare (pure) simulation method, is the main goal of this paper. Almost no assumptions (like convexity) on the set of constraints are needed, …
We prove a two-way correspondence between the min-max optimization of general CPE loss function GANs and the minimization of associated $f$-divergences. We then focus on $α$-GAN, defined via the $α$-loss, which interpolates several GANs (Hellinger, vanilla, Total Variation) and corresponds to the minimization of the Arimoto divergence. We show that the Arimoto divergences induced by $α$-GAN equivalently converge, for all $α\in \mathbb{R}_{>0}\cup\{\infty\}$. However, under restricted learning models and finite samples, we provide estimation bounds which indicate diverse GAN behavior as a funct…
Information divergences allow one to assess how close two distributions are from each other. Among the large panel of available measures, a special attention has been paid to convex $φ$-divergences, such as Kullback-Leibler, Jeffreys-Kullback, Hellinger, Chi-Square, Renyi, and I$_α$ divergences. While $φ$-divergences have been extensively studied in convex analysis, their use in optimization problems often remains challenging. In this regard, one of the main shortcomings of existing methods is that the minimization of $φ$-divergences is usually performed with respect to one of their arguments,…
Divergences induced by dual subtractive and divisive normalizations of exponential families and their convex deformations
本文研究指数族在减式与除式归一化下诱导的散度关系，证明未归一化密度的α-散度等价于配分函数诱导的缩放α-偏斜Jensen散度，并利用比较凸性构造凸变形下的对偶平坦空间。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。