首页/论文/2012.09943

arXiv 2012.099432020-12-17stat.MLcs.LG

Guiding Neural Network Initialization via Marginal Likelihood Maximization

Indiana University（印第安纳大学）

Anthony S. Tai, Chunfeng Huang

英文摘要：

We propose a simple, data-driven approach to help guide hyperparameter selection for neural network initialization. We leverage the relationship between neural network and Gaussian process models having corresponding activation and covariance functions to infer the hyperparameter values desirable for model initialization. Our experiment shows that marginal likelihood maximization provides recommendations that yield near-optimal prediction performance on MNIST classification task under experiment constraints. Furthermore, our empirical results indicate consistency in the proposed technique, suggesting that computation cost for the procedure could be significantly reduced with smaller training sets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

On the Behavior of the Expectation-Maximization Algorithm for Mixture Models

University of Southern California（南加州大学）

Finite mixture models are among the most popular statistical models used in different data science disciplines. Despite their broad applicability, inference under these models typically leads to computationally challenging non-convex problems. While the Expectation-Maximization (EM) algorithm is the most popular approach for solving these non-convex problems, the behavior of this algorithm is not well understood. In this work, we focus on the case of mixture of Laplacian (or Gaussian) distribution. We start by analyzing a simple equally weighted mixture of two single dimensional Laplacian dist…

2018-09-24cs.LGstat.ML

Gradient-based training of Gaussian Mixture Models for High-Dimensional Streaming Data

Fulda University of Applied Sciences（富尔达应用技术大学）

We present an approach for efficiently training Gaussian Mixture Model (GMM) by Stochastic Gradient Descent (SGD) with non-stationary, high-dimensional streaming data. Our training scheme does not require data-driven parameter initialization (e.g., k-means) and can thus be trained based on a random initialization. Furthermore, the approach allows mini-batch sizes as low as 1, which are typical for streaming-data settings. Major problems in such settings are undesirable local optima during early training phases and numerical instabilities due to high data dimensionalities. We introduce an adapt…

2021-07-02cs.LGstat.ML

Improved model-based clustering performance using Bayesian initialization averaging

The Expectation-Maximization (EM) algorithm is a commonly used method for finding the maximum likelihood estimates of the parameters in a mixture model via coordinate ascent. A serious pitfall with the algorithm is that in the case of multimodal likelihood functions, it can get trapped at a local maximum. This problem often occurs when sub-optimal starting values are used to initialize the algorithm. Bayesian initialization averaging (BIA) is proposed as an ensemble method to generate high quality starting values for the EM algorithm. Competing sets of trial starting values are combined as a w…

2018-08-30stat.CO

An Experimental Comparison of Several Clustering and Initialization Methods

Microsoft Research（微软研究院）；Massachusetts Institute of Technology（麻省理工学院）

We examine methods for clustering in high dimensions. In the first part of the paper, we perform an experimental comparison between three batch clustering algorithms: the Expectation-Maximization (EM) algorithm, a winner take all version of the EM algorithm reminiscent of the K-means algorithm, and model-based hierarchical agglomerative clustering. We learn naive-Bayes models with a hidden root node, using high-dimensional discrete-variable data sets (both real and synthetic). We find that the EM algorithm significantly outperforms the other methods, and proceed to investigate the effect of va…

2015-05-16cs.LGstat.ML

Conjugate Mixture Models for Clustering Multimodal Data

INRIA Grenoble Rhône-Alpes（法国国立信息学与自动化研究院格勒诺布尔罗讷-阿尔卑斯研究中心）

The problem of multimodal clustering arises whenever the data are gathered with several physically different sensors. Observations from different modalities are not necessarily aligned in the sense there there is no obvious way to associate or to compare them in some common space. A solution may consist in considering multiple clustering tasks independently for each modality. The main difficulty with such an approach is to guarantee that the unimodal clusterings are mutually consistent. In this paper we show that multimodal clustering can be addressed within a novel framework, namely conjugate…

2020-12-09cs.CVcs.LGcs.SD

Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels

Tsinghua University（清华大学）；Huawei Technologies Co. Ltd.（华为技术有限公司）

In deep learning, neural networks serve as noisy channels between input data and its representation. This perspective naturally relates deep learning with the pursuit of constructing channels with optimal performance in information transmission and representation. While considerable efforts are concentrated on realizing optimal channel properties during network optimization, we study a frequently overlooked possibility that neural networks can be initialized toward optimal channels. Our theory, consistent with experimental validation, identifies primary mechanics underlying this unknown possib…

2022-12-04cond-mat.dis-nncond-mat.stat-mechcs.LG

更多相关论文收起更多论文

Unsupervised learning of regression mixture models with unknown number of components

Aix Marseille Université（艾克斯-马赛大学）；CNRS（法国国家科学研究中心）；ENSAM（法国国立高等工程技术学校）；Université de Toulon（土伦大学）

Regression mixture models are widely studied in statistics, machine learning and data analysis. Fitting regression mixtures is challenging and is usually performed by maximum likelihood by using the expectation-maximization (EM) algorithm. However, it is well-known that the initialization is crucial for EM. If the initialization is inappropriately performed, the EM algorithm may lead to unsatisfactory results. The EM algorithm also requires the number of clusters to be given a priori; the problem of selecting the number of mixture components requires using model selection criteria to choose on…

2014-09-24cs.LGstat.MEstat.ML

$k$-MLE: A fast algorithm for learning statistical mixture models

Sony Computer Science Laboratories, Inc（索尼计算机科学实验室）

We describe $k$-MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expected complete) likelihood. Given prescribed mixture weights, the hard clustering $k$-MLE algorithm iteratively assigns data to the most likely weighted component and update the component models using Maximum Likelihood Estimators (MLEs). Using the duality between exponential famil…

2012-03-23cs.LGstat.ML

过指定混合判别分析：指数收敛、统计保证与遥感应用

Overspecified Mixture Discriminant Analysis: Exponential Convergence, Statistical Guarantees, and Remote Sensing Applications

Machine Learning Department（机器学习系）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫比内扎伊德人工智能大学）；Remote Sensing Department（遥感系）；National Center of Space Research and Technology（国家航天研究与技术中心）；Department of Mathematical Sciences（数学科学系）；Purdue University Fort Wayne（普渡大学富尔顿分校）；Department of Mathematics and Statistics（数学与统计学系）；University of Minnesota Duluth（明尼苏达大学杜鲁斯分校）；Department of Mathematics（数学系）；Nazarbayev University（纳扎尔巴耶夫大学）

本研究分析了过指定条件下混合判别分析（MDA）的分类误差，证明EM算法在总体水平指数收敛至贝叶斯风险，有限样本下分类误差以 $n^{-1/2}$ 速率收敛，并在遥感数据集上验证了理论。

2025-10-30cs.LGstat.ML

可识别的贝叶斯深度生成Copula模型：未知层宽下任意边缘分布数据的建模

Identifiable Bayesian Deep Generative Copulas with Unknown Layer Widths for Data with Arbitrary Marginal Distributions

Department of Statistics and Data Science（统计与数据科学系）；Washington University in St. Louis（圣路易斯华盛顿大学）；Department of Statistics（统计系）；Columbia University（哥伦比亚大学）

提出Deep Discrete Encoder (DDE) Copula模型，通过二元潜变量的分层有向网络与Copula框架结合，实现任意边缘分布数据的可识别与可解释生成建模，并基于秩似然进行估计与后验推断。

2026-05-26cs.LGstat.ML

Initialization and Regularization of Factorized Neural Layers

Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research（微软研究院）

Factorized layers--operations parameterized by products of two or more matrices--occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head self-attention architectures. We study how to initialize and regularize deep nets containing such layers, examining two simple, understudied schemes, spectral initialization and Frobenius decay, for improving their performance. The guiding insight is to design optimization routines for these networks that are as close as possible to that of their well-tuned, non-decomposed coun…

2022-10-04cs.AIcs.CLcs.CV

期望最大化（EM）在一般无偏混合中收敛

Expectation Maximization (EM) Converges for General Agnostic Mixtures

Department of Computer Science

本文研究了在无偏设置下拟合k个参数函数的问题，提出并分析了梯度EM算法，证明其在适当初始化和分离条件下的指数收敛性。

2026-04-07cs.ITcs.LGmath.IT

Robust EM algorithm for model-based curve clustering

Information Sciences and Systems Laboratory (LSIS), UMR CNRS 7296（信息科学与系统实验室（LSIS），法国国家科学研究中心 UMR 7296）；University of the South Toulon-Var (USTV)（南土伦瓦尔大学（USTV））

Model-based clustering approaches concern the paradigm of exploratory data analysis relying on the finite mixture model to automatically find a latent structure governing observed data. They are one of the most popular and successful approaches in cluster analysis. The mixture density estimation is generally performed by maximizing the observed-data log-likelihood by using the expectation-maximization (EM) algorithm. However, it is well-known that the EM algorithm initialization is crucial. In addition, the standard EM algorithm requires the number of clusters to be known a priori. Some soluti…

2013-12-25cs.LGstat.MEstat.ML