Hidden Markov models (HMMs) are widely used statistical models for modeling sequential data. The parameter estimation for HMMs from time series data is an important learning problem. The predominant methods for parameter estimation are based on local search heuristics, most notably the expectation-maximization (EM) algorithm. These methods are prone to local optima and oftentimes suffer from high computational and sample complexity. Recent years saw the emergence of spectral methods for the parameter estimation of HMMs, based on a method of moments approach. Two spectral learning algorithms as proposed by Hsu, Kakade and Zhang 2012 (arXiv:0811.4413) and Anandkumar, Hsu and Kakade 2012 (arXiv:1203.0683) are assessed in this work. Using experiments with synthetic data, the algorithms are compared with each other. Furthermore, the spectral methods are compared to the Baum-Welch algorithm, a well-established method applying the EM algorithm to HMMs. The spectral algorithms are found to have a much more favorable computational and sample complexity. Even though the algorithms readily handle high dimensional observation spaces, instability issues are encountered in this regime. In view of learning from real-world experimental data, the representation of real-valued observations for the use in spectral methods is discussed, presenting possible methods to represent data for the use in the learning algorithms.
An Algorithm for Approximating Continuous Functions on Compact Subsets with a Neural Network with one Hidden Layer
George Cybenko在1989年的里程碑式论文中指出,存在一个前馈神经网络,它只有一个隐藏层(和有限数量的神经元),可以任意逼近单位超立方体上给定的连续函数$f$。这篇论文没有讨论如何找到这样一个网络的权重/参数,或者找到它们在计算上是否可行。本文给出了一个具有一个隐层的神经网络重构任意连续标量或向量值连续函数的算法。