我们提出了一种新的低秩初始化框架，用于训练低秩深度神经网络-网络，其中权重参数由两个低秩矩阵的乘积重新参数化。最成功的现有方法，谱初始化，从满秩设置的初始化分布中抽取样本，然后通过奇异值分解用一对低秩初始化矩阵在Frobenius范数中最佳地近似满秩初始化参数。我们的方法的灵感来自于这样一种认识，即近似对应于每一层的函数比近似参数值更重要。我们可以证明，ReLU网络的这两种方法之间存在显著的差距，特别是当近似权重的期望秩降低时，或者当层输入的维度增加时（当网络宽度在维度上是超线性时，后一点成立）。在此过程中，我们提供了第一个可证明有效的算法来解决固定参数秩$r$的ReLU低秩近似问题-以前，即使对于秩$1$，也不知道该问题在计算上是否易于解决。我们还提供了一种实用的算法来解决这个问题，它不比现有的谱初始化方法更昂贵，并通过训练ResNet和EfficientNet模型来验证我们的理论（He et al.，2016; Tan & Le，2019）在ImageNet上（Russakovsky等人，2015年）。
英文摘要：
We propose a novel low-rank initialization framework for training low-rank deep neural networks -- networks where the weight parameters are re-parameterized by products of two low-rank matrices. The most successful prior existing approach, spectral initialization, draws a sample from the initialization distribution for the full-rank setting and then optimally approximates the full-rank initialization parameters in the Frobenius norm with a pair of low-rank initialization matrices via singular value decomposition. Our method is inspired by the insight that approximating the function corresponding to each layer is more important than approximating the parameter values. We provably demonstrate that there is a significant gap between these two approaches for ReLU networks, particularly as the desired rank of the approximating weights decreases, or as the dimension of the inputs to the layer increases (the latter point holds when the network width is super-linear in dimension). Along the way, we provide the first provably efficient algorithm for solving the ReLU low-rank approximation problem for fixed parameter rank $r$ -- previously, it was unknown that the problem was computationally tractable to solve even for rank $1$. We also provide a practical algorithm to solve this problem which is no more expensive than the existing spectral initialization approach, and validate our theory by training ResNet and EfficientNet models (He et al., 2016; Tan & Le, 2019) on ImageNet (Russakovsky et al., 2015).
IDInit: A Universal and Stable Initialization Method for Neural Network Training
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）；MetaAI（元宇宙人工智能实验室）；Fudan University（复旦大学）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
Improved SVD-based Initialization for Nonnegative Matrix Factorization using Low-Rank Correction
Graduate School of Science and Engineering（研究生院）；PAF-Karachi Institute of Economics and Technology（卡拉奇经济和技术学院）；Department of Mathematics and Operational Research（数学与运筹学系）
矩阵分解是研究梯度下降隐式正则化的一个简单而自然的测试平台。Gunasekar et al.（2017）证明了具有无穷小初始化的梯度流收敛于最小化核范数的解，但最近的一系列论文认为范数最小化的语言不足以给出隐式正则化的完整表征。在这项工作中，我们提供了理论和经验证据，证明对于深度2矩阵分解，具有无穷小初始化的梯度流在数学上等价于一个简单的启发式秩最小化算法，贪婪低秩学习，在一些合理的假设下。这将先前工作中的秩最小化观点推广到更广泛的环境，并使我们能够构建反例来反驳Gunasekar等人的猜想。我们还将结果扩展到深度为$\ge 3$的情况，并且我们表明更深的好处是上述收敛对初始化幅度的依赖性要弱得多，因此这种秩最小化更有可能对实际规模的初始化生效。
Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction
Initialization Method for Factorization Machine Based on Low-Rank Approximation for Constructing a Corrected Approximate Ising Model
Graduate School of Science and Technology, Keio University（科技科学研究生院，庆应大学）；Recruit Co., Ltd.（Recruit公司）；Department of Applied Physics and Physico-Informatics, Keio University（应用物理与物理信息学系，庆应大学）；Keio University Sustainable Quantum Artificial Intelligence Center (KSQAIC), Keio University（庆应大学可持续量子人工智能中心（KSQAIC），庆应大学）；Human Biology-Microbiome-Quantum Research Center (WPI-Bio2Q), Keio University（人类生物学-微生物群-量子研究中心（WPI-Bio2Q），庆应大学）
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Georgia Institute of Technology（佐治亚理工学院）；Massachusetts Institute of Technology（麻省理工学院）
The power of small initialization in noisy low-tubal-rank tensor recovery
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences, China（机器人与智能系统国家重点实验室，沈阳自动化研究所，中国科学院，中国）；University of Chinese Academy of Sciences, China（中国科学院大学，中国）；School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院，新加坡）；The Center for Intelligent Decision-making and Machine Learning, School of Management, Xi’an Jiaotong University, China（智能决策与机器学习中心，西安交通大学管理学院，中国）
University of Science and Technology of China（中国科学技术大学）；Institute of Intelligent Machines, HFIPS, Chinese Academy of Sciences（中国科学院智能机械研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；University of Sydney（悉尼大学）；Anhui Disaster Warning & Agrometeorological Information Center（安徽省灾害预警与农业气象信息中心）
Learning Linearized Models from Nonlinear Systems under Initialization Constraints with Finite Data
Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）；School of Mechanical Engineering, Purdue University（帕克大学机械工程系）；Elmore Family School of Electrical and Computer Engineering, Purdue University（帕克大学埃尔莫家族电气与计算机工程学院）
Parameter Efficient Fine-tuning via Explained Variance Adaptation
ELLIS Unit, LIT AI Lab, Institute for Machine Learning, JKU Linz（ELLIS单位、LIT人工智能实验室、机器学习研究所、JKU林茨）；University College London（伦敦大学学院）；EMMI AI, Linz（EMMI AI、林茨）；NXAI GmbH, Linz, Austria（NXAI GmbH、林茨、奥地利）
本文提出 EVA 初始化方法，以激活方差最大方向构造 LoRA 子空间并支持自适应秩，在多类任务上实现更快收敛、更高平均得分及更少可训练参数。
California Institute of Technology（加州理工学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Carnegie Mellon University（卡内基梅隆大学）；Georgia Institute of Technology（佐治亚理工学院）；NVIDIA（英伟达）
Antai College of Economics and Management, Shanghai Jiao Tong University（上海交通大学安泰经济与管理学院）；University of Michigan（密歇根大学）；Michigan State University（密歇根州立大学）
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of International Business and Economics, China（对外经济贸易大学）；XiaoMi AI Lab, China（小米AI实验室）
