深度学习中类流形的反转动力学揭示了泛化背后的权衡
Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
AI总结:
本文通过数值实验发现深度学习优化动力学中类流形先分离后纠缠的非单调反转现象,并证明反转点由数据结构和少量“掉队者”决定,揭示了泛化背后的核心权衡。
摘要:
为了在分类问题中实现接近零的训练误差,前馈网络的各层必须解开具有不同标签的数据点流形,以促进判别。然而,过度的类别分离可能导致过拟合,因为良好的泛化需要学习不变特征,而这涉及一定程度的纠缠。我们报告了数值实验,展示了优化动力学如何以非单调趋势找到平衡这些对立倾向的表示。在快速分离阶段之后,一个较慢的重排过程(在数据集和架构之间保持一致)增加了类别纠缠。反转时的训练误差在子采样下、跨网络初始化和优化器下都是稳定的,这将其表征为仅由数据结构以及(非常微弱地)由架构决定的属性。该反转是由训练集中定义明确且最大稳定元素(称为“掉队者”)所引发的权衡的体现,这些元素对泛化特别有影响。
英文摘要:
To achieve near-zero training error in a classification problem, the layers of a feed-forward network have to disentangle the manifolds of data points with different labels, to facilitate the discrimination. However, excessive class separation can bring to overfitting since good generalisation requires learning invariant features, which involve some level of entanglement. We report on numerical experiments showing how the optimisation dynamics finds representations that balance these opposing tendencies with a non-monotonic trend. After a fast segregation phase, a slower rearrangement (conserved across data sets and architectures) increases the class entanglement.The training error at the inversion is stable under subsampling, and across network initialisations and optimisers, which characterises it as a property solely of the data structure and (very weakly) of the architecture. The inversion is the manifestation of tradeoffs elicited by well-defined and maximally stable elements of the training set, coined ``stragglers'', particularly influential for generalisation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。