arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.08505cs.LGcs.AI
收藏

利用正交性训练低秩神经网络

Harnessing Orthogonality to Train Low-Rank Neural Networks

Scientific Computing Center (SCC), Karlsruhe Institute of Technology (KIT)(卡尔斯鲁厄理工学院科学计算中心(SCC))

Daniel Coquelin, Katharina Flügel, Marie Weiel, Nicholas Kiefer, Charlotte Debus, Achim Streit, Markus Götz

AI总结:

本文通过分析权重SVD发现训练中正交基趋于稳定,据此提出OIALR低秩训练方法,能无缝集成现有工作流且精度损失极小,调参后可超越常规及最先进模型训练设置。

摘要:

本研究通过分析训练过程中权重的奇异值分解(SVD)来探索神经网络的学习动态。我们的研究表明,每个多维权重SVD表示中的正交基在训练过程中会趋于稳定。基于此,我们提出了正交性自适应低秩(OIALR)训练,这是一种利用神经网络内在正交性的新颖训练方法。正如在各种数据集和成熟的网络架构上的基准测试所证明的那样,OIALR能够以最小的精度损失无缝集成到现有的训练工作流中。通过适当的超参数调整,OIALR可以超越包括最先进模型在内的常规训练设置。

英文摘要:

This study explores the learning dynamics of neural networks by analyzing the singular value decomposition (SVD) of their weights throughout training. Our investigation reveals that an orthogonal basis within each multidimensional weight's SVD representation stabilizes during training. Building upon this, we introduce Orthogonality-Informed Adaptive Low-Rank (OIALR) training, a novel training method exploiting the intrinsic orthogonality of neural networks. OIALR seamlessly integrates into existing training workflows with minimal accuracy loss, as demonstrated by benchmarking on various datasets and well-established network architectures. With appropriate hyperparameter tuning, OIALR can surpass conventional training setups, including those of state-of-the-art models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑