Weights initialization of neural networks for function approximation
School of Mathematics and Computational Science（数学与计算科学学院）；National Center for Applied Mathematics in Hunan（湖南应用数学研究中心）；Hunan Key Laboratory for Computation and Simulation in Science and Engineering（计算与工程科学与工程计算模拟湖南重点实验室）；University of Texas at El Paso（德克萨斯埃尔帕索大学）
Neural network-based function approximation plays a pivotal role in the advancement of scientific computing and machine learning. Yet, training such models faces several challenges: (i) each target function often requires training a new model from scratch; (ii) performance is highly sensitive to architectural and hyperparameter choices; and (iii) models frequently generalize poorly beyond the training domain. To overcome these challenges, we propose a reusable initialization framework based on basis function pretraining. In this approach, basis neural networks are first trained to approximate families of polynomials on a reference domain. Their learned parameters are then used to initialize networks for more complex target functions. To enhance adaptability across arbitrary domains, we further introduce a domain mapping mechanism that transforms inputs into the reference domain, thereby preserving structural correspondence with the pretrained models. Extensive numerical experiments in one- and two-dimensional settings demonstrate substantial improvements in training efficiency, generalization, and model transferability, highlighting the promise of initialization-based strategies for scalable and modular neural function approximation. The full code is made publicly available on Gitee.
在神经网络中，权值的正确初始化对网络的收敛性至关重要。目前对权重初始化的认识主要来自线性激活函数。在本文中，我开发了一个理论的重量初始化与非线性激活。首先，我推导出一个一般的权重初始化策略，任何神经网络使用激活函数可微0。接下来，我推导出了整流线性单元（RECTIFITED LINEAR UNIT）的权重初始化策略，并从理论上解释了为什么Xavier初始化对于RECTIFITED LINEAR UNIT激活是一个糟糕的选择。我的分析提供了一个明确的示范作用的非线性确定适当的权重初始化。
How to Initialize your Network? Robust Initialization for WeightNorm & ResNets
Salesforce Research（赛富时研究院）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；Montréal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；Université de Montréal（蒙特利尔大学）
A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks
Harbin Institute of Technology Shenzhen（哈尔滨工业大学（深圳））；University of Electronic Science and Technology of China（电子科技大学）；Tokyo Institute of Technology（东京工业大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Pengcheng Laboratory（鹏城实验室）
1-Lipschitz Network Initialization for Certifiably Robust Classification Applications: A Decay Problem
The Grainger College of Engineering, Industrial and Enterprise Systems Engineering Department（格拉inger工程学院，工业与企业系统工程系）；Construction Engineering Research Laboratory（建设工程研究实验室）；U.S. Army Corps of Engineers Engineering Research and Development Center（美国陆军工程兵团工程研究与发展中心）
Initialization Method for Factorization Machine Based on Low-Rank Approximation for Constructing a Corrected Approximate Ising Model
Graduate School of Science and Technology, Keio University（科技科学研究生院，庆应大学）；Recruit Co., Ltd.（Recruit公司）；Department of Applied Physics and Physico-Informatics, Keio University（应用物理与物理信息学系，庆应大学）；Keio University Sustainable Quantum Artificial Intelligence Center (KSQAIC), Keio University（庆应大学可持续量子人工智能中心（KSQAIC），庆应大学）；Human Biology-Microbiome-Quantum Research Center (WPI-Bio2Q), Keio University（人类生物学-微生物群-量子研究中心（WPI-Bio2Q），庆应大学）
WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（教育部新一代人工智能技术及其交叉应用重点实验室（东南大学））
Reducing Oversmoothing through Informed Weight Initialization in Graph Neural Networks
School of Electrical Engineering and Computer Science, National Technical University of Athens（雅典国立技术大学电气与计算机工程学院）；National Center for Scientific Research “Demokritos”（德谟克利特国家科学研究中心）
IDInit: A Universal and Stable Initialization Method for Neural Network Training
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）；MetaAI（元宇宙人工智能实验室）；Fudan University（复旦大学）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
Department of Computer Science（计算机科学系）；Information Engineering, National Cheng Kung University（信息工程系，国立成功大学）；Department of Artificial Intelligence, Chang Gung University（人工智能系，长庚大学）；Department of Computer Science, National Tsing Hua University（计算机科学系，国立清华大学）
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
Language Technologies Institute, School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院语言技术研究所）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
Pruning Neural Networks at Initialization: Why are We Missing the Mark?
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Element AI（Element AI公司）；University of Toronto（多伦多大学）；Vector Institute（矢量研究院）
最近的工作探索了在初始化时修剪神经网络的可能性。我们评估这样做的建议：SNIP（Lee等人，2019），GraSP（Wang et al.，2020）、SynFlow（Tanaka等人，2020年）和幅度修剪。尽管这些方法超越了随机修剪的平凡基线，但它们在训练后仍然低于幅度修剪的准确性，我们试图理解为什么。我们表明，与训练后修剪不同，随机打乱这些方法在每层中修剪的权重或采样新的初始值可以保留或提高准确性。因此，由这些方法做出的每权重修剪决策可以由要修剪的权重的分数的每层选择来代替。此属性表明了底层修剪算法、在初始化时修剪的愿望或两者的更广泛挑战。
