As a randomized learner model, SCNs are remarkable that the random weights and biases are assigned employing a supervisory mechanism to ensure universal approximation and fast learning. However, the randomness makes SCNs more likely to generate approximate linear correlative nodes that are redundant and low quality, thereby resulting in non-compact network structure. In the light of a fundamental principle in machine learning, that is, a model with fewer parameters holds improved generalization. This paper proposes orthogonal SCN, termed OSCN, to filtrate out the low-quality hidden nodes for network structure reduction by incorporating Gram-Schmidt orthogonalization technology. The universal approximation property of OSCN and an adaptive setting for the key construction parameters have been presented in details. In addition, an incremental updating scheme is developed to dynamically determine the output weights, contributing to improved computational efficiency. Finally, experimental results on two numerical examples and several real-world regression and classification datasets substantiate the effectiveness and feasibility of the proposed approach.
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University(人工智能微电子实验室、人工智能学院、计算机科学学院、上海交通大学);Xiaohongshu Inc.(小红书公司)
A Novel Orthogonal Direction Mesh Adaptive Direct Search Approach for SVM Hyperparameter Tuning
École de Technologie Supérieure - Université du Québec(魁北克大学高等技术学院);University of Santa Catarina(圣卡塔琳娜联邦大学);Pontifícia Universidade Católica do Paraná(巴拉那天主教大学)
On the equivalence of different adaptive batch size selection strategies for stochastic gradient descent methods
University of Nottingham(诺丁汉大学);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);RWTH Aachen University(亚琛工业大学);King Abdullah University of Science & Technology(阿卜杜拉国王科技大学);Stevens Institute of Technology(史蒂文斯理工学院)
University of Reading(雷丁大学);University of Southampton(南安普顿大学);King Abdulaziz University(阿卜杜勒阿齐兹国王大学);CSIRO Mathematics and Information Sciences(联邦科学与工业研究组织数学与信息科学研究所);Charles Sturt University(查尔斯特大学)