非凸经验风险最小化中的拓扑平凡化
Topological trivialization in non-convex empirical risk minimization
AI总结：
研究非凸经验风险最小化中的拓扑平凡化现象，通过分析局部极小值数量的渐进行为，揭示样本与参数比例对模型收敛性的影响。
摘要：
给定数据集$\{({\boldsymbol x}_i,y_i): i\le n\}$，其中${\boldsymbol x}_i$是标准的$d$维高斯特征向量，$y_i\in{\mathbb R}$是响应变量，我们研究学习一个由${\boldsymbol θ}\in{\mathbb R}^d$参数化的模型的问题，通过最小化一个依赖于${\boldsymbol θ}$的损失函数，该损失函数通过${\boldsymbol θ}^{\sf T}{\boldsymbol x}_i$的一维投影来确定。尽管之前的研究大多处理的是凸损失函数，但我们的方法假设一般的（非凸）损失函数，从而涵盖了经典但理解有限的例子，如感知机和非凸鲁棒回归。我们使用Kac-Rice公式来控制经验风险的局部极小值数量的渐进行为，在比例渐进行为$n,d\to\infty$，$n/d\toα>1$下。具体来说，我们证明了一个有限维变分公式，用于指数增长率的预期局部极小值数量。进一步，我们提供了充分条件，使得指数增长率消失，并且所有经验风险最小化器具有相同的渐进行为（实际上，我们期望在这些情况下最小化器是唯一的）。我们将这种现象称为“速率平凡化”。如果总体风险有唯一的最小化器，我们的速率平凡化充分条件通常在样本/参数比$α$大于合适的常数$α_{\star}$时得到验证。之前一般结果要求$n\ge Cd \log d$。我们在非凸鲁棒回归的情况下展示了我们的结果。基于启发式论点和数值模拟，我们提出了关于平凡化相变$α_{\text{tr}}$的精确位置的猜想。
英文摘要：
Given data $\{({\boldsymbol x}_i,y_i): i\le n\}$, with ${\boldsymbol x}_i$ standard $d$-dimensional Gaussian feature vectors, and $y_i\in{\mathbb R}$ response variables, we study the general problem of learning a model parametrized by ${\boldsymbol θ}\in{\mathbb R}^d$, by minimizing a loss function that depends on ${\boldsymbol θ}$ via the one-dimensional projections ${\boldsymbol θ}^{\sf T}{\boldsymbol x}_i$. While previous work mostly dealt with convex losses, our approach assumes general (non-convex) losses hence covering classical, yet poorly understood examples such as the perceptron and non-convex robust regression. We use the Kac-Rice formula to control the asymptotics of the expected number of local minima of the empirical risk, under the proportional asymptotics $n,d\to\infty$, $n/d\toα>1$. Specifically, we prove a finite dimensional variational formula for the exponential growth rate of the expected number of local minima. Further we provide sufficient conditions under which the exponential growth rate vanishes and all empirical risk minimizers have the same asymptotic properties (in fact, we expect the minimizer to be unique in these circumstances). We refer to this phenomenon as `rate trivialization.' If the population risk has a unique minimizer, our sufficient condition for rate trivialization is typically verified when the samples/parameters ratio $α$ is larger than a suitable constant $α_{\star}$. Previous general results of this type required $n\ge Cd \log d$. We illustrate our results in the case of non-convex robust regression. Based on heuristic arguments and numerical simulations, we present a conjecture for the exact location of the trivialization phase transition $α_{\text{tr}}$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
