Nonparametric Statistical Inference for Ergodic Processes
Daniil Ryabko, Boris Ryabko
备注：
Conference version in: D. Ryabko, B. Ryabko, On hypotheses testing for ergodic processes, in Proceedgings of Information Theory Workshop, 2008, Porto, Portugal, pp. 281-283
发表信息：
IEEE Transactions on Information Theory 56, 3 (2010) 1430-1435
In this work a method for statistical analysis of time series is proposed, which is used to obtain solutions to some classical problems of mathematical statistics under the only assumption that the process generating the data is stationary ergodic. Namely, three problems are considered: goodness-of-fit (or identity) testing, process classification, and the change point problem. For each of the problems a test is constructed that is asymptotically accurate for the case when the data is generated by stationary ergodic processes. The tests are based on empirical estimates of distributional distance.
In this paper we consider the nonparametric estimation of density and regression functions with non-negative support using a gamma kernel procedure introduced by Chen (2000). Strong uniform consistency and asymptotic normality of the corresponding estimators are established under a general ergodic assumption on the data generation process. Our results generalize those of Shi and Song (2016), obtained in the classic i.i.d. framework, and the works of Bouezmarni and Rombouts (2008, 2010b) and Gospodinov and Hirukawa (2007) for mixing time series.
We study the nonparametric Nadaraya-Watson estimator of the drift function for ergodic stochastic processes driven by fractional Brownian motion of Hurst parameter H > 1/2. The estimator is based on the discretely observed stochastic processes. By using the ergodic properties and stochastic integral, we obtain the consistency of the proposed estimator.
We consider the problem of estimating the context tree of a stationary ergodic process with finite alphabet without imposing additional conditions on the process. As a starting point we introduce a Hamming metric in the space of irreducible context trees and we use the properties of the weak topology in the space of ergodic stationary processes to prove that if the Hamming metric is unbounded, there exist no consistent estimators for the context tree. Even in the bounded case we show that there exist no two-sided confidence bounds. However we prove that one-sided inference is possible in this …
The present paper deals with a nonparametric M-estimation for right censored regression model with stationary ergodic data. Defined as an implicit function, a kernel type estimator of a family of robust regression is considered when the covariate take its values in R^d (d >= 1) and the data are sampled from stationary ergodic process. The strong consistency (with rate) and the asymptotic distribution of the estimator are established under mild assumptions. Moreover, a usable confidence interval is provided which does not depend on any unknown quantity. Our results hold without any mixing condi…
In this paper we find nonasymptotic exponential upper bounds for the deviation in the ergodic theorem for families of homogeneous Markov processes. We find some sufficient conditions for geometric ergodicity uniformly over a parametric family. We apply this property to the nonasymptotic nonparametric estimation problem for ergodic diffusion processes.
This paper is devoted to the nonparametric estimation of the jump rate and the cumulative rate for a general class of non-homogeneous marked renewal processes, defined on a separable metric space. In our framework, the estimation needs only one observation of the process within a long time. Our approach is based on a generalization of the multiplicative intensity model, introduced by Aalen in the seventies. We provide consistent estimators of these two functions, under some assumptions related to the ergodicity of an embedded chain and the characteristics of the process. The paper is illustrat…
Nonparametric identification and maximum likelihood estimation for finite-state hidden Markov models are investigated. We obtain identification of the parameters as well as the order of the Markov chain if the transition probability matrices have full-rank and are ergodic, and if the state-dependent distributions are all distinct, but not necessarily linearly independent. Based on this identification result, we develop nonparametric maximum likelihood estimation theory. First, we show that the asymptotic contrast, the Kullback--Leibler divergence of the hidden Markov model, identifies the true…
We review a nonparametric version of Amari's Information Geometry in which the set of positive probability densities on a given sample space is endowed with an atlas of charts to form a differentiable manifold modeled on Orlicz Banach spaces. This nonparametric setting is used to discuss the setting of typical problems in Machine Learning and Statistical Physics, such as relaxed optimization, Kullback-Leibler divergence, Boltzmann entropy, Boltzmann equation
A piecewise-deterministic Markov process is a stochastic process whose behavior is governed by an ordinary differential equation punctuated by random jumps occurring at random times. We focus on the nonparametric estimation problem of the jump rate for such a stochastic model observed within a long time interval under an ergodicity condition. We introduce an uncountable class (indexed by the deterministic flow) of recursive kernel estimates of the jump rate and we establish their strong pointwise consistency as well as their asymptotic normality. We propose to choose among this class the estim…
We consider the problem of adaptive estimation of the regression function in a framework where we replace ergodicity assumptions (such as independence or mixing) by another structural assumption on the model. Namely, we propose adaptive upper bounds for kernel estimators with data-driven bandwidth (Lepski's selection rule) in a regression model where the noise is an increment of martingale. It includes, as very particular cases, the usual i.i.d. regression and auto-regressive models. The cornerstone tool for this study is a new result for self-normalized martingales, called ``stability'', whic…
We consider the question of estimating the drift and the invariant density for a large class of scalar ergodic diffusion processes, based on continuous observations, in $\sup$-norm loss. The unknown drift $b$ is supposed to belong to a nonparametric class of smooth functions of unknown order. We suggest an adaptive approach which allows to construct drift estimators attaining minimax optimal $\sup$-norm rates of convergence. In addition, we prove a Donsker theorem for the classical kernel estimator of the invariant density and establish its semiparametric efficiency. Finally, we combine both r…
In this paper we examine the use of topological methods for multivariate statistics. Using persistent homology from computational algebraic topology, a random sample is used to construct estimators of persistent homology. This estimation procedure can then be evaluated using the bottleneck distance between the estimated persistent homology and the true persistent homology. The connection to statistics comes from the fact that when viewed as a nonparametric regression problem, the bottleneck distance is bounded by the sup-norm loss. Consequently, a sharp asymptotic minimax bound is determined u…
We analyze the statistical properties of nonparametric regression estimators using covariates which are not directly observable, but have be estimated from data in a preliminary step. These so-called generated covariates appear in numerous applications, including two-stage nonparametric regression, estimation of simultaneous equation models or censored regression models. Yet so far there seems to be no general theory for their impact on the final estimator's statistical properties. Our paper provides such results. We derive a stochastic expansion that characterizes the influence of the generat…
We consider the double functional nonparametric regression model $Y=r(X)+ε$, where the response variable $Y$ is Hilbert space-valued and the covariate $X$ takes values in a pseudometric space. The data satisfy an ergodicity criterion which dates back to Laib and Louani (2010) and are arranged in a triangular array. So our model also applies to samples obtained from spatial processes, e.g., stationary random fields indexed by the regular lattice $\mathbb{Z}^N$ for some $N\in\mathbb{N}_+$. We consider a kernel estimator of the Nadaraya--Watson type for the regression operator $r$ and study its l…
This paper deals with the estimation problem of misspecified ergodic Lévy driven stochastic differential equation models based on high-frequency samples. We utilize the widely applicable and tractable Gaussian quasi-likelihood approach which focuses on (conditional) mean and variance structure. It is shown that the corresponding Gaussian quasi-likelihood estimators of drift and scale parameters satisfy tail probability estimates and asymptotic normality at the same rate as correctly specified case. In this process, extended Poisson equation for time-homogeneous Feller Markov processes plays an…
A new negative result for nonparametric estimation of binary ergodic processes is shown. I The problem of estimation of distribution with any degree of accuracy is studied. Then it is shown that for any countable class of estimators there is a zero-entropy binary ergodic process that is inconsistent with the class of estimators. Our result is different from other negative results for universal forecasting scheme of ergodic processes.
We construct a general procedure for the Quasi Likelihood Analysis applied to a multivariate point process on the real half line in an ergodic framework. More precisely, we assume that the stochastic intensity of the underlying model belongs to a family of processes indexed by a finite dimensional parameter. When a particular family of laws of large numbers applies to those processes, we establish the consistency, the asymptotic normality and the convergence of moments of both the Quasi Maximum Likelihood estimator and the Quasi Bayesian estimator. In addition, we illustrate our main results b…
We study minimax convergence rates of nonparametric density estimation under a large class of loss functions called "adversarial losses", which, besides classical $\mathcal{L}^p$ losses, includes maximum mean discrepancy (MMD), Wasserstein distance, and total variation distance. These losses are closely related to the losses encoded by discriminator networks in generative adversarial networks (GANs). In a general framework, we study how the choice of loss and the assumed smoothness of the underlying density together determine the minimax rate. We also discuss implications for training GANs bas…
A new negative result for nonparametric distribution estimation of binary ergodic processes is shown. The problem of estimation of distribution with any degree of accuracy is studied. Then it is shown that for any countable class of estimators there is a zero-entropy binary ergodic process that is inconsistent with the class of estimators. Our result is different from other negative results for universal forecasting scheme of ergodic processes. We also introduce a related result by B. Weiss.
This paper is devoted to the nonparametric estimation of the derivative of the regression function in a nonparametric regression model. We implement a very efficient and easy to handle statistical procedure based on the derivative of the recursive Nadaraya-Watson estimator. We establish the almost sure convergence as well as the asymptotic normality for our estimates. We also illustrate our nonparametric estimation procedure on simulated and real life data associated with sea shores water quality and valvometry.
We study the problem of estimating a nonparametric probability density under a large family of losses called Besov IPMs, which include, for example, $\mathcal{L}^p$ distances, total variation distance, and generalizations of both Wasserstein and Kolmogorov-Smirnov distances. For a wide variety of settings, we provide both lower and upper bounds, identifying precisely how the choice of loss function and assumptions on the data interact to determine the minimax optimal convergence rate. We also show that linear distribution estimates, such as the empirical distribution or kernel density estimato…
We study the problem of estimating the coefficients of a diffusion (X_t,t\geq 0); the estimation is based on discrete data X_{nΔ},n=0,1,...,N. The sampling frequency Δ^{-1} is constant, and asymptotics are taken as the number N of observations tends to infinity. We prove that the problem of estimating both the diffusion coefficient (the volatility) and the drift in a nonparametric setting is ill-posed: the minimax rates of convergence for Sobolev constraints and squared-error loss coincide with that of a, respectively, first- and second-order linear inverse problem. To ensure ergodicity and li…
In applications the properties of a stochastic feature often change gradually rather than abruptly, that is: after a constant phase for some time they slowly start to vary. In this paper we discuss statistical inference for the detection and the localisation of gradual changes in the jump characteristic of a discretely observed Ito semimartingale. We propose a new measure of time variation for the jump behaviour of the process. The statistical uncertainty of a corresponding estimate is analyzed by deriving new results on the weak convergence of a sequential empirical tail integral process and …
In this paper, we discuss the use of $\varepsilon$-admissibility for estimation in high-dimensional and nonparametric statistical models. The minimax rate of convergence is widely used to compare the performance of estimators in high-dimensional and nonparametric models. However, it often works poorly as a criterion of comparison. In such cases, the addition of comparison by $\varepsilon$-admissibility provides a better outcome. We demonstrate the usefulness of $\varepsilon$-admissibility through high-dimensional Poisson model and Gaussian infinite sequence model, and present noble results.
We consider nonparametric statistical inference for Lévy processes sampled irregularly, at low frequency. The estimation of the jump dynamics as well as the estimation of the distributional density are investigated. Non-asymptotic risk bounds are derived and the corresponding rates of convergence are discussed under global as well as local regularity assumptions. Moreover, minimax optimality is proved for the estimator of the jump measure. Some numerical examples are given to illustrate the practical performance of the estimation procedure.
Consider a periodic, mean-reverting Ornstein-Uhlenbeck process $X=\{X_t,t\geq0\}$ of the form $d X_{t}=\left(L(t)+αX_{t}\right) d t+ dB^H_{t}, \quad t \geq 0$, where $L(t)=\sum_{i=1}^{p}μ_iϕ_i (t)$ is a periodic parametric function, and $\{B^H_t,t\geq0\}$ is a fractional Brownian motion of Hurst parameter $\frac12\leq H<1$. In the "ergodic" case $α<0$, the parametric estimation of $(μ_1,\ldots,μ_p,α)$ based on continuous-time observation of $X$ has been considered in Dehling et al. \cite{DFK}, and in Dehling et al. \cite{DFW} for $H=\frac12$, and $\frac12<H<1$, respectively. In this paper we c…
For nonparametric regression in the streaming setting, where data constantly flow in and require real-time analysis, a main challenge is that data are cleared from the computer system once processed due to limited computer memory and storage. We tackle the challenge by proposing a novel one-pass estimator based on penalized orthogonal basis expansions and developing a general framework to study the interplay between statistical efficiency and memory consumption of estimators. We show that, the proposed estimator is statistically optimal under memory constraint, and has asymptotically minimal m…
We investigate the statistical complexity of estimating the parameters of a discrete-state Markov chain kernel from a single long sequence of state observations. In the finite case, we characterize (modulo logarithmic factors) the minimax sample complexity of estimation with respect to the operator infinity norm, while in the countably infinite case, we analyze the problem with respect to a natural entry-wise norm derived from total variation. We show that in both cases, the sample complexity is governed by the mixing properties of the unknown chain, for which, in the finite-state case, there …
We consider a binary branching process structured by a stochastic trait that evolves according to a diffusion process that triggers the branching events, in the spirit of Kimmel's model of cell division with parasite infection. Based on the observation of the trait at birth of the first n generations of the process, we construct nonparametric estimator of the transition of the associated bifurcating chain and study the parametric estimation of the branching rate. In the limit, as n tends to infinity, we obtain asymptotic efficiency in the parametric case and minimax optimality in the nonparame…
Consider a Poisson point process with unknown support boundary curve $g$, which forms a prototype of an irregular statistical model. We address the problem of estimating non-linear functionals of the form $\int Φ(g(x))\,dx$. Following a nonparametric maximum-likelihood approach, we construct an estimator which is UMVU over Hölder balls and achieves the (local) minimax rate of convergence. These results hold under weak assumptions on $Φ$ which are satisfied for $Φ(u)=|u|^p$, $p\ge 1$. As an application, we consider the problem of estimating the $L^p$-norm and derive the minimax separation rates…
University of Pennsylvania（宾夕法尼亚大学）；Yale University（耶鲁大学）
This paper studies the problem of nonparametric estimation of a smooth function with data distributed across multiple machines. We assume an independent sample from a white noise model is collected at each machine, and an estimator of the underlying true function needs to be constructed at a central machine. We place limits on the number of bits that each machine can use to transmit information to the central machine. Our results give both asymptotic lower bounds and matching upper bounds on the statistical risk under various settings. We identify three regimes, depending on the relationship a…
Hypothesis testing and other statistical inference procedures are most efficient when a reliable low-dimensional parametric family can be specified. We propose a method that learns such a family when one exists but its form is not known a priori, by examining samples from related populations and fitting a low-dimensional exponential family that approximates all the samples as well as possible. We propose a computationally efficient spectral method that allows us to carry out hypothesis tests that are valid whether or not the fit is good, and recover asymptotically optimal power if it is. Our m…
The Bayesian nonparametric inference and Dirichlet process are popular tools in statistical methodologies. In this paper, we employ the Dirichlet process in hypothesis testing to propose a Bayesian nonparametric chi-squared goodness-of-fit test. In our Bayesian nonparametric approach, we consider the Dirichlet process as the prior for the distribution of data and carry out the test based on the Kullback-Leibler distance between the updated Dirichlet process and the hypothesized distribution F0. We prove that this distance asymptotically converges to the same chi-squared distribution as the chi…
A new model for time series with a specific oscillation pattern is proposed. The model consists of a hidden phase process controlling the speed of polling and a nonparametric curve characterizing the pattern, leading together to a generalized state space model. Identifiability of the model is proved and a method for statistical inference based on a particle smoother and a nonparametric EM algorithm is developed. In particular, the oscillation pattern and the unobserved phase process are estimated. The proposed algorithms are computationally efficient and their performance is assessed through s…
We consider nonparametric testing in a non-asymptotic framework. Our statistical guarantees are exact in the sense that Type I and II errors are controlled for any finite sample size. Meanwhile, one proposed test is shown to achieve minimax optimality in the asymptotic sense. An important consequence of this non-asymptotic theory is a new and practically useful formula for selecting the optimal smoothing parameter in nonparametric testing. The leading example in this paper is smoothing spline models under Gaussian errors. The results obtained therein can be further generalized to the kernel ri…
Stationary ergodic processes with finite alphabets are estimated by finite memory processes from a sample, an n-length realization of the process, where the memory depth of the estimator process is also estimated from the sample using penalized maximum likelihood (PML). Under some assumptions on the continuity rate and the assumption of non-nullness, a rate of convergence in $\bar{d}$-distance is obtained, with explicit constants. The result requires an analysis of the divergence of PML Markov order estimators for not necessarily finite memory processes. This divergence problem is investigated…
We formulate the notion of minimax estimation under storage or communication constraints, and prove an extension to Pinsker's theorem for nonparametric estimation over Sobolev ellipsoids. Placing limits on the number of bits used to encode any estimator, we give tight lower and upper bounds on the excess risk due to quantization in terms of the number of bits, the signal size, and the noise level. This establishes the Pareto optimal tradeoff between storage and risk under quantization constraints for Sobolev spaces. Our results and proof techniques combine elements of rate distortion theory an…
We proposed a learning algorithm for nonparametric estimation and on-line prediction for general stationary ergodic sources. We prepare histograms each of which estimates the probability as a finite distribution, and mixture them with weights to construct an estimator. The whole analysis is based on measure theory. The estimator works whether the source is discrete or continuous. If it is stationary ergodic, then the measure theoretically given Kullback-Leibler information divided by the sequence length $n$ converges to zero as $n$ goes to infinity. In particular, for continuous sources, the m…
In this paper, first, we survey the concept of diffeological Fisher metric and its naturality, using functorial language of probability morphisms, and slightly extending Lê's theory in \cite{Le2020} to include weakly $C^k$-diffeological statistical models. Then we introduce the resulting notions of the diffeological Fisher distance, the diffeological Hausdorff--Jeffrey measure and explain their role in classical and Bayesian nonparametric estimation problems in statistics.
Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we study differentially private tests of independence between a categorical and a continuous variable. We take as our starting point traditional nonparametric tests, which require no distributional assumption (e.g., normality) about the data distribution. We present private analogues of the Kruskal-Wallis, Mann-Whitney, and Wilcoxon signed-rank tests, as well as the parametric one-sample t-test. These tests use novel test statistics developed specifically for the pr…
Fiducial inference, as generalized by Hannig et al. (2016), is applied to nonparametric g-modeling (Efron, 2016) in the discrete case. We propose a computationally efficient algorithm to sample from the fiducial distribution, and use the generated samples to construct point estimates and confidence intervals. We study the theoretical properties of the fiducial distribution and perform extensive simulations in various scenarios. The proposed approach yields good statistical performance in terms of the mean squared error of point estimators and the coverage of confidence intervals. Furthermore, …
We study the nonparametric estimation of the branching rate $B(x)$ of a supercritical Bellman-Harris population: a particle with age $x$ has a random lifetime governed by $B(x)$; at its death time, it gives rise to $k \geq 2$ children with lifetimes governed by the same division rate and so on. We observe in continuous time the process over $[0,T]$. Asymptotics are taken as $T \rightarrow \infty$; the data are stochastically dependent and one has to face simultaneously censoring, bias selection and non-ancillarity of the number of observations. In this setting, under appropriate ergodicity pro…
Let $\{(X_i,Y_i)\}$ be a stationary ergodic time series with $(X,Y)$ values in the product space $\R^d\bigotimes \R .$ This study offers what is believed to be the first strongly consistent (with respect to pointwise, least-squares, and uniform distance) algorithm for inferring $m(x)=E[Y_0|X_0=x]$ under the presumption that $m(x)$ is uniformly Lipschitz continuous. Auto-regression, or forecasting, is an important special case, and as such our work extends the literature of nonparametric, nonlinear forecasting by circumventing customary mixing assumptions. The work is motivated by a time series…
