高维经验风险景观的拓扑探索:通用方法及在相位恢复中的应用
Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrieval
AI总结:
本文研究了高维经验风险景观的拓扑特性,提出了一种通用方法用于分析损失景观,并应用于相位恢复问题,揭示了BBP型相变及梯度流动力学的预测。
摘要:
我们考虑高维高斯单索引模型(广义线性模型)的经验风险最小化景观。目标是从依赖于标签对(x_i·θ,x_i·θ*)的损失函数R(θ)中恢复未知信号θ*∈R^d(其中d>>1)。在比例渐近 regime n≈d 中,我们使用Kac-Rice公式分析不同复杂度的景观——定义为期望的临界点数量——对应各种类型的临界点,包括局部极小值。我们首先展示了一些之前文献中建立的变分公式可以大大简化,减少到有限数量的标量参数上的显式变分问题,我们可以高效地数值求解。我们的框架还提供了关于临界点性质的详细预测,包括Hessian的谱性质以及标签的联合分布。我们将分析应用于实际的相位恢复问题,推导出完整的损失景观拓扑相图,特别指出BBP型过渡,其中局部极小值的Hessian(根据Kac-Rice公式预测)在信号方向上变得不稳定。我们测试了我们分析的预测能力以表征梯度流动力学,发现与局部优化算法的有限大小模拟有很好的一致,捕捉到诸如标签的实证分布等细粒度细节。总体而言,我们的结果为高维统计模型中损失景观和拓扑平凡现象的渐近研究开辟了新途径。
英文摘要:
We consider the landscape of empirical risk minimization for high-dimensional Gaussian single-index models (generalized linear models). The objective is to recover an unknown signal $\boldsymbolθ^\star \in \mathbb{R}^d$ (where $d \gg 1$) from a loss function $\hat{R}(\boldsymbolθ)$ that depends on pairs of labels $(\mathbf{x}_i \cdot \boldsymbolθ, \mathbf{x}_i \cdot \boldsymbolθ^\star)_{i=1}^n$, with $\mathbf{x}_i \sim \mathcal{N}(0, I_d)$, in the proportional asymptotic regime $n \asymp d$. Using the Kac-Rice formula, we analyze different complexities of the landscape -- defined as the expected number of critical points -- corresponding to various types of critical points, including local minima. We first show that some variational formulas previously established in the literature for these complexities can be drastically simplified, reducing to explicit variational problems over a finite number of scalar parameters that we can efficiently solve numerically. Our framework also provides detailed predictions for properties of the critical points, including the spectral properties of the Hessian and the joint distribution of labels. We apply our analysis to the real phase retrieval problem for which we derive complete topological phase diagrams of the loss landscape, characterizing notably BBP-type transitions where the Hessian at local minima (as predicted by the Kac-Rice formula) becomes unstable in the direction of the signal. We test the predictive power of our analysis to characterize gradient flow dynamics, finding excellent agreement with finite-size simulations of local optimization algorithms, and capturing fine-grained details such as the empirical distribution of labels. Overall, our results open new avenues for the asymptotic study of loss landscapes and topological trivialization phenomena in high-dimensional statistical models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。