用梯度流学习高斯多索引模型:时间复杂性与方向收敛
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence
AI总结:
本文研究高斯数据上多索引函数的梯度流学习,刻画搜索阶段后神经元的方向收敛性,证明正交索引向量下可用约 k log k 个神经元高概率恢复全部索引向量,并揭示非正交情形存在尖锐阈值决定相关性损失是否有效。
摘要:
本文研究一个神经网络模型的梯度流动力学,该模型使用相关性损失在高维标准高斯数据上逼近多索引函数。具体而言,我们考虑的多索引函数是神经元之和 $f^*(x) \\!=\\! \sum_{j=1}^k \\! σ^*(v_j^T x)$,其中 $v_1, \dots, v_k$ 是单位向量,且 $σ^*$ 在其 Hermite 展开中缺少第一和第二 Hermite 多项式。已知在单索引情形($k\\!=\\!1$)下,克服搜索阶段需要多项式时间复杂度。我们首先将该结果推广到由任意方向向量刻画的多索引函数。在搜索阶段之后,尚不清楚网络神经元是收敛到索引向量,还是停留在次优解。当索引向量正交时,我们给出了不动点的完整刻画,并证明神经元收敛到最近的索引向量。因此,使用 $n \\! \asymp \\! k \log k$ 个神经元可确保在随机初始化下以高概率通过梯度流找到完整的索引向量集合。当对所有 $i \neq j$ 有 $ v_i^T v_j \\!=\\! β\\! \geq \\! 0$ 时,我们证明存在一个尖锐阈值 $β_c \\!=\\! c/(c+k)$,在该阈值处,计算索引向量平均值的不动点从鞍点转变为极小值点。数值模拟表明,当索引向量近乎正交时,使用相关性损失和适度过参数化足以学习所有索引向量;然而,当索引向量之间的点积超过某个阈值时,相关性损失会失效。
英文摘要:
This work focuses on the gradient flow dynamics of a neural network model that uses correlation loss to approximate a multi-index function on high-dimensional standard Gaussian data. Specifically, the multi-index function we consider is a sum of neurons $f^*(x) \!=\! \sum_{j=1}^k \! σ^*(v_j^T x)$ where $v_1, \dots, v_k$ are unit vectors, and $σ^*$ lacks the first and second Hermite polynomials in its Hermite expansion. It is known that, for the single-index case ($k\!=\!1$), overcoming the search phase requires polynomial time complexity. We first generalize this result to multi-index functions characterized by vectors in arbitrary directions. After the search phase, it is not clear whether the network neurons converge to the index vectors, or get stuck at a sub-optimal solution. When the index vectors are orthogonal, we give a complete characterization of the fixed points and prove that neurons converge to the nearest index vectors. Therefore, using $n \! \asymp \! k \log k$ neurons ensures finding the full set of index vectors with gradient flow with high probability over random initialization. When $ v_i^T v_j \!=\! β\! \geq \! 0$ for all $i \neq j$, we prove the existence of a sharp threshold $β_c \!=\! c/(c+k)$ at which the fixed point that computes the average of the index vectors transitions from a saddle point to a minimum. Numerical simulations show that using a correlation loss and a mild overparameterization suffices to learn all of the index vectors when they are nearly orthogonal, however, the correlation loss fails when the dot product between the index vectors exceeds a certain threshold.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。