arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.03347cs.LGmath.STstat.TH
收藏

多尺度单指标模型:用于分层特征学习的程式化模型

The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning

Courant Institute School of Mathematics, Computing and Data Science(柯朗研究所数学、计算与数据科学学院)

Joan Bruna

AI总结:

研究多尺度单指标模型用于分层学习,在特定假设下分析目标的维纳混沌,通过埃奇沃思展开进行细粒度分析,建立浅层网络近似下界,证明在线SGD在相关目标上的样本复杂度。

摘要:

我们将首次在[参考文献]中引入的多尺度单指标模型(MSIM)视为具有“尺度分离”的分层学习的程式化模型。各层在一个物理尺度上提取共享的单指标特征并传递给下一层,定义了一个易于处理的设置来研究深度架构如何学习多尺度表示。在关于链接函数和植入特征的非退化和离域假设下,对于固定深度K和局部尺度d,目标的一阶维纳混沌表现为一个受扰的尖峰张量,其中阶数为d^(-1/2)的扰动来自非线性——揭示了MSIM作为张量主成分分析模型的自然非线性类似物。虽然这种微扰图景足以基于张量展开实现有效的谱恢复,但对于基于反向传播梯度的方法分析来说不够精确。在这项工作中,我们通过使用埃奇沃思展开对维纳混沌进行细粒度分析来解决这一限制。在一阶混沌中,这给出了尺度为d^(-q/2)的有限秩层次结构。在更高阶混沌中,在自然的高阶混沌非抵消条件下,平衡扁平化表现出大小为d^(-ρ/2)且重数为d^ρ的阶梯状奇异值平台。利用这种高阶混沌结构,并在额外的缓慢埃尔米特能量尾部条件下,我们首先建立浅层网络近似下界,量化该模型中深度的益处。其次,也是最重要的,我们证明在相关目标上的在线随机梯度下降,其中所有层在相同的时间尺度上演化,使用n = \widetilde{O}(d^(K - 1))个样本实现1 - o_d(1)的恢复,恢复与线性对应物相同的样本复杂度。

英文摘要:

We consider the Multiscale Single-Index Model (MSIM), first introduced in \cite{oymak2021learning}, as a stylized model for hierarchical learning with \emph{scale separation}. Each layer extracts a shared single-index feature at one physical scale and passes it to the next, thus defining a tractable setting in which to study how deep architectures learn multiscale representations. Under non-degeneracy and delocalization assumptions on the link function and planted features respectively, for fixed depth $K$ and local scale $d$, the first Wiener chaos of the target behaves as a perturbed spiked tensor, where the perturbation of order $d^{-1/2}$ comes from the non-linearity -- revealing the MSIM as a natural non-linear analogue of the Tensor PCA model \cite{montanari2014statistical}. While this perturbative picture is sufficient to enable efficient spectral recovery based on Tensor unfolding (as already observed in \cite{oymak2021learning}), it is not precise enough for the analysis of backpropagation gradient-based methods. In this work, we address this limitation by performing a fine-grained analysis of the Wiener chaos using Edgeworth expansions. In the first chaos, this gives a finite-rank hierarchy at scales $d^{-q/2}$. In higher chaoses, balanced flattenings exhibit staircase singular-value plateaus of size $d^{-ρ/2}$ and multiplicity $d^ρ$ under a natural higher-chaos non-cancellation condition. Using this higher-chaos structure, and under an additional slow Hermite-energy tail condition, we first establish shallow-network approximation lower bounds, quantifying the benefit of depth in this model. Next, and most importantly, we prove that online SGD on the correlation objective, where all layers evolve in the same timescale, achieves $1 - o_d(1)$ recovery with $n = \widetilde{O}( d^{K-1})$ samples, recovering the same sample complexity as in the linear counterpart.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑