基于ADMM的可证明收敛的信息瓶颈解决方案
A Provably Convergent Information Bottleneck Solution via ADMM
摘要：
信息瓶颈（IB）方法能够优化数据压缩和学习表示的预测准确性之间的权衡，并且已经成功且鲁棒地应用于监督和无监督表示学习问题。然而，IB有一些限制。首先，IB问题很难优化。IB拉格朗日$\mathcal{L}_{IB}:=I(X;Z)-βI(Y;Z)$是非凸的，现有的解决方案只能保证局部收敛。因此，所获得的解依赖于初始化。其次，评估解决方案也是一项具有挑战性的任务。传统上，它采取表征信息平面的方法，即，针对从不同初始点获得的所有解绘制$I(Y;Z)$与$I(X;Z)$的关系。此外，IB拉格朗日量具有相变，同时改变乘数$β$。在相变时，$I(X;Z)$和$I(Y;Z)$都突然增加，并且对于现有的解决方案，收敛速率变得非常慢。最近的作品与IB采用变分代理界的IB拉格朗日。虽然允许有效的优化，如何接近这些代理IB拉格朗日不清楚。在这项工作中，我们解决了IB拉格朗日使用增广拉格朗日方法。增广变量，我们表明，IB目标可以解决交替方向的乘法器（ADMM）。与以往的工作不同，我们证明了所提出的算法是一致收敛的，无论$β$的值。从经验上讲，我们的基于梯度下降的方法的信息平面点，通过传统的Blahut-Arimoto为基础的求解器获得的，是收敛的惩罚系数比以前的ADMM求解器的范围更广。
英文摘要：
The Information bottleneck (IB) method enables optimizing over the trade-off between compression of data and prediction accuracy of learned representations, and has successfully and robustly been applied to both supervised and unsupervised representation learning problems. However, IB has several limitations. First, the IB problem is hard to optimize. The IB Lagrangian $\mathcal{L}_{IB}:=I(X;Z)-βI(Y;Z)$ is non-convex and existing solutions guarantee only local convergence. As a result, the obtained solutions depend on initialization. Second, the evaluation of a solution is also a challenging task. Conventionally, it resorts to characterizing the information plane, that is, plotting $I(Y;Z)$ versus $I(X;Z)$ for all solutions obtained from different initial points. Furthermore, the IB Lagrangian has phase transitions while varying the multiplier $β$. At phase transitions, both $I(X;Z)$ and $I(Y;Z)$ increase abruptly and the rate of convergence becomes significantly slow for existing solutions. Recent works with IB adopt variational surrogate bounds to the IB Lagrangian. Although allowing efficient optimization, how close are these surrogates to the IB Lagrangian is not clear. In this work, we solve the IB Lagrangian using augmented Lagrangian methods. With augmented variables, we show that the IB objective can be solved with the alternating direction method of multipliers (ADMM). Different from prior works, we prove that the proposed algorithm is consistently convergent, regardless of the value of $β$. Empirically, our gradient-descent-based method results in information plane points that are comparable to those obtained through the conventional Blahut-Arimoto-based solvers and is convergent for a wider range of the penalty coefficient than previous ADMM solvers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
