Efficient model selection for identifying a suitable pre-trained neural network to a downstream task is a fundamental yet challenging task in deep learning. Current practice requires expensive computational costs in model training for performance prediction. In this paper, we propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections. Therefore, a converged neural network is associated with an equilibrium state of a networked system composed of those edges. To this end, we construct a network mapping $ϕ$, converting a neural network $G_A$ to a directed line graph $G_B$ that is defined on those edges in $G_A$. Next, we derive a neural capacitance metric $β_{\rm eff}$ as a predictive measure universally capturing the generalization capability of $G_A$ on the downstream task using only a handful of early training results. We carried out extensive experiments using 17 popular pre-trained ImageNet models and five benchmark datasets, including CIFAR10, CIFAR100, SVHN, Fashion MNIST and Birds, to evaluate the fine-tuning performance of our framework. Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods.
Feature and Instance Joint Selection: A Reinforcement Learning Perspective
University of Central Florida（中佛罗里达大学）；Hong Kong University of Science and Technology（香港科技大学）；Baidu Talent Intelligence Center（百度人才智库）；Rutgers University（罗格斯大学）
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Department of Computer Science（计算机科学系）；Department of Mathematics（数学系）；Mathematical Institute（数学研究所）；University of Pisa（比萨大学）；Imperial College London（伦敦帝国学院）；University of Oxford（牛津大学）；Section of Mathematics and Artificial Intelligence（数学与人工智能系）；Max Planck Institute of Molecular Cell Biology and Genetics（马克斯·普朗克分子细胞生物学和遗传学研究所）；Center for Systems Biology（系统生物学中心）；Faculty of Mathematics（数学系）；TU Dresden（德累斯顿技术大学）
Regularization vs. Relaxation: A conic optimization perspective of statistical variable selection
Department of Mathematics, Washington State University（数学系，华盛顿州立大学）；Department of Statistics, University of Connecticut（统计系，康涅狄格大学）；Department of Industrial and Systems Engineering, University of Wisconsin-Madison（工业与系统工程系，威斯康星大学麦迪逊分校）
From Ambiguity to Action: A POMDP Perspective on Partial Multi-Label Ambiguity and Its Horizon-One Resolution
College of Computer Science and Technology, Jilin University, China（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China（教育部符号计算与知识工程重点实验室）
Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Paradigm. Inc.（第四范式有限公司）；Lenovo（联想）
